mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
2174f2dfa5ea6c4e926e8da537fd31c5a0280d05
7681
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2174f2dfa5 |
perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed. The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f2e449f8f6 |
fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person's Retry on a connected task (for example a GitHub pull request review) must start work. #21080 made a Retry whose chat replay is refused after the click continue from the recovered saved work > - After #21080 was deployed, accepted Retries on a busy GitHub review agent still did not start > - Those tasks had a saved chat request that arrived while the failed run was still running. Admission holds such a request as `message_predates_stop`, so it can never start alone > - #21080 refused to continue while any saved chat input existed, so this held input blocked the continuation. For a second group, the latest run was an admission that the recovery hold cancelled before dispatch, and the recovered saved work belongs to the earlier failed run > - This pull request lets the continuation adopt input that was held because it predates the stop, and lets the recovered run that held a cancelled admission count as the saved work > - The benefit is that an accepted Retry starts work on these tasks too ## Linked Issues or Issue Description Follow-up to #21080. No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for blocked tasks of a GitHub review agent (`paperclip_runner`), but no run started. Each Retry receipt changed to `failed` about 5 seconds later (on the 14:48 Retries, 5 seconds for one task and about 75 seconds for two others), with no run and no `issue.retry_continued_from_saved_work` activity. Every affected task had a saved chat request (a newer GitHub review request) with an `issue.execution_waiting` record of reason `message_predates_stop`. The request arrived while the failed run was running, so admission holds it permanently. The interruption projection kept pointing at that held request. In a second group of tasks, the latest run was cancelled before dispatch with `execution_reconciliation_required`, and the recovered saved work belonged to the earlier failed run that held it. Root cause: in `continueRefusedTaskChatRetry` (`server/src/services/task-retry.ts`), any saved chat input from `selectTaskRetryInput` stopped the continuation, so the receipt settled `failed`. The held input can never start by itself, so nothing ran. The recovered-work check also looked only at the latest run, so it missed the recovered run that held a cancelled admission. **Expected behavior** An accepted Retry starts work. Saved input that admission holds because it predates the stop joins the continuation, and the task continues from the recovered saved work. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Let a native run fail with `native_continuation_requires_reconciliation`, and recover its saved work. 2. Save a chat request while that run is still running, so admission holds it with `message_predates_stop`. 3. Click Retry while the chat identity lock is busy, so the source check is deferred and then refused. 4. The route returns 202. The Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** |
||
|
|
d5bd38468d |
fix(runner): stop a leftover Runner on the same state folder before a relaunch (#21079)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) in a
Daytona sandbox. The Runner keeps its saved state in one folder per
session and listens on one port in the sandbox.
> - During an automatic upgrade, the old server can launch a Runner a
few seconds before it stops. That Runner stays alive in the sandbox and
keeps the state folder and the port.
> - The new server then restarts the run on the same sandbox. Each new
Runner fails at once with "Address already in use"
(`runner_ingress_bind_conflict`). Each launch also overwrites the
process marker that pointed at the old Runner, so the next check cannot
find the old Runner to stop it. The run restarted about 12 times in 19
minutes and then stopped, and the task stayed blocked.
> - Two Runner processes must never use the same saved state at the same
time.
> - This pull request makes each Runner launch stop any live Runner that
still names the same state folder, before the new Runner starts.
> - The benefit is that a restart after an upgrade gets the port and
continues the run, instead of looping on the bind conflict.
## Linked Issues or Issue Description
**What happened?**
On staging, a Runner launched by the old server just before an upgrade
kept the Runner port. Every relaunch by the new server exited within
seconds with "runner_ingress_bind_conflict: failed to bind listener:
Address already in use (os error 98)". The run looped for about 19
minutes and then failed with `native_runner_process_exited`. The task
was blocked and needed a person.
**Expected behavior**
The relaunch stops the leftover Runner of the same session, gets the
port, and continues the run.
**Steps to reproduce**
1. Launch a Runner for a run in a sandbox.
2. Stop the server before the Runner is adopted (an upgrade). The Runner
stays alive.
3. Let the new server restart the run's bootstrap. The new Runner cannot
bind the port, and the next restart cannot see the old Runner because
the marker now names the dead new process.
## What Changed
- `native-session-executor.ts`: the detached Runner launch script now
runs a short fence before it writes the process marker.
- It finds processes whose argv[0] is `paperclip-runnerd`, that are not
a `reap-lineage` command, and whose `--state-dir` is followed by this
exact folder. Runners on another folder, a folder named by a different
flag, and other programs are not touched.
- It records the stale Runner's lineage by PID **and start time**: its
descendants, and members of a process group or session that a recorded
process leads while that leader is alive (provider tools start their own
sessions). It signals a PID only while its start time still matches, so
a reused PID is never signalled, and it never signals a whole process
group.
- TERM, up to 5 s wait, then KILL, up to 2 s wait. Then it runs
`<runnerd> reap-lineage --state-dir … --runner-pid …` for the stopped
Runner, which stops the tools its lineage journal recorded that have
already left the process tree.
- It fails closed: a survivor (`runner_stale_state_holder_alive`) or a
reap that leaves a recorded process alive or gives no report
(`runner_stale_lineage_unproven`) makes the launch exit 75 **before**
the marker is written, so the existing marker is kept. The reason goes
to the process stderr log only when that path is a plain file (same rule
as the Runner stderr log). The whole stop is bounded at about 14 s,
under the 20 s marker wait.
- Two reap refusals let the launch continue with a
`runner_stale_lineage_partial` warning: `journal_unavailable` (the
Runner kept no journal; its visible lineage is already gone) and
`unattributed_orphan` (reported only after every recorded process is
gone, for orphans the reaper can neither claim nor signal) **only when
every orphan it names is proven to have another owner**: this launch's
own detached shell, a `paperclip-runnerd` serving a different
`--state-dir`, or that Runner's stderr trimmer. Any other orphan could
be an old tool that escaped between samples, so the launch fails closed.
- When `/proc` is not readable, or `grep` is missing, the fence does
nothing (the old behavior).
- `native-session-executor.test.ts`: Linux tests (a stale Runner whose
tool runs in its own session, plus four look-alikes that must survive;
fail-closed with the marker kept, with a plain and a symlinked log; reap
refusals for orphans owned elsewhere, an unknown orphan, and no journal;
a normal launch starts no extra node process) and a cross-platform check
that the embedded script parses and has no shell-special characters.
**When a launch happens vs. a reattach.** The launch is reached only
after the server chose not to adopt a Runner. A same-run reattachment
(`reattach_remote_runner`) first verifies the marker's Runner: if it is
alive it is adopted and nothing launches; a transient verification
failure (for example the read timeout that #21053 keeps the Runner alive
for) throws and never reaches a launch; only a Runner proven not running
falls through to a fresh launch. Warm recovery likewise throws instead
of launching when revalidation fails. Every launch overwrites the
marker, so after a launch no reattachment could adopt a leftover Runner
anyway, and its port would make the new Runner fail with
`runner_ingress_bind_conflict`. Stopping it therefore never removes a
Runner that the same run could still re-adopt.
## Verification
- Linux (Debian bookworm container, node 24): the Linux tests pass on
this head. The same tests against the previous head (
|
||
|
|
4e932fd178 |
perf(attention): index the agents and review paths once for every review of a request (#21132)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task in review needs a path to its next action: a reviewer, a pending interaction or approval, a monitor, a run, a wake, an external reply or a recovery. A review without a path is stalled, and the attention feed tells a person about it. > - The server decides this for each task in review (`listIssueReviewAttentionMap` in `server/src/services/issues.ts`, with the classification in `server/src/services/recovery/issue-graph-liveness.ts`). The task list, the blocked inbox and the attention feed call it. `GET /attention/summary`, which the board sidebar reads on most pages, runs it for every task in review of the company. > - The classification did the company-wide work again for each task. Each check of an agent copied the agent list, indexed it again and walked the reporting chain. Each task scanned every run, wake, interaction, approval, external wait and recovery of the request. Each stalled review also filtered and sorted every agent again, with one such check for each agent. > - Thus the work grew with reviews × agents for covered reviews and with stalled reviews × agents² for stalled reviews. It also allocated a new list and a new index for each check, and the garbage collector of the server paid for that. > - On a hosted instance, an allocation sample of the server put 4.7% of all allocated bytes under this function, and garbage collection used 25% of the main thread. The two slowest board requests there are the attention summary and the task list, which both call it. > - This pull request builds one index for one classification: the agents by id, the eligibility of each agent, the invokable agents of each company in id order, and the paths of each task. Each task then costs its own paths. > - The benefit is that the review classification of a summary request falls from 62 ms to 5 ms for 900 reviews and 60 agents, and from 123 ms to 8 ms for 300 reviews and 180 agents. The request allocates 57% to 91% fewer bytes. Every finding, every review path and every count is the same. ## Linked Issues or Issue Description Refs #20851 and #20858 (they batched the review-wait provenance check and the task read decision of the same requests; this change is the next cost on that path). Refs #19194 and #13729 (open; they change the failed-run lookup of the attention feed, which is a different statement and a different mechanism; this pull request does not touch it). **What existing behavior does this improve?** The time and the memory that the server uses to decide the review attention of the tasks in review, for `GET /api/companies/:companyId/attention/summary`, `GET /api/companies/:companyId/attention`, `GET /api/companies/:companyId/issues` and the blocked inbox. **Subsystem affected** Server: task-graph liveness classification (`server/src/services/recovery/issue-graph-liveness.ts`) and the review-attention map in `server/src/services/issues.ts`. **Current behavior** - `classifyIssueReviewPaths(input, issue)` built `new Map(input.agents…)` for each task and scanned six path lists of the request for each task. - `isInvokableAgent` copied the agent list for each check. `getAgentOrgChainHealth` then indexed that copy again. - `ownerCandidatesForRecoveryIssue` ran for each stalled review. It filtered and sorted all agents, and it checked each agent again in two more loops. Each check did the copy and the index above. - `hasActiveExecutionPath` built a joined list of the runs and the wakes for each blocker it looked at. - `listIssueReviewAttentionMap` searched the review list for each open recovery task (`reviewIds.includes`) and for each review (`issues.find`). Evidence from a hosted instance (sanitized): - Server main thread at 93% of one core. Garbage collection: 25% of the main thread (387 pauses in 32 s, the longest 224 ms). - Allocation sample, 30 s: 77.7 MB/s in total. `listIssueReviewAttentionMap` and its callees: 4.7% of all bytes (`classifyIssueReviewPaths` 4.4%, `classifyIssueGraphLiveness` 2.3%, `reviewFinding` 2.3%). - `GET /attention/summary` took 16.3 s of server time and `GET /issues` up to 15.3 s. Most of that time is waiting behind other work on the same thread (see "Not verified"). **Proposed behavior** One classification indexes its input one time. The cost of one task is the number of its own paths. The cost of the agents is one eligibility decision for each agent. The results do not change. **Reason and benefit** The summary request runs on most board pages. Its CPU time and its allocation also delay every other request, because the server has one main thread and one garbage collector. **Breaking changes** None. `classifyIssueGraphLiveness` and `classifyIssueReviewPaths` keep their signatures and results. `packages/shared` does not change. ## What Changed - `server/src/services/recovery/issue-graph-liveness.ts` - New internal index for one input (`createIssueGraphLivenessIndex`): agents by id, a memo of `getAgentWorkEligibility` for each agent record, the invokable agents of each company in id order, and the paths of each task for the six path lists. The path lists are indexed on first use and keep the order of the input. The company check of each path stays. - `classifyIssueGraphLiveness` builds the index one time and uses it for the review paths, the waiting paths of blockers, the owner candidates and the assignee eligibility of blockers. - New export `createIssueReviewPathClassifier(input)`. It returns the review paths of many tasks from one index. `classifyIssueReviewPaths(input, issue)` stays for single tasks. - The memo uses the agent record as its key, not the agent id. Thus an input that lists one id two times gets the same answers as before. - The eligibility decision itself is still `getAgentWorkEligibility` of `@paperclipai/shared`, with the same agent list. The index calls it one time for each agent, not one time for each check. - `server/src/services/issues.ts` (`listIssueReviewAttentionMap` only): one classifier for all reviews of the request, a set for the review ids, and the review by position instead of a search. No query changes. - Tests and the reproduction (see Verification): - `server/src/__tests__/helpers/issue-graph-liveness-reference.ts`: the classification as it was before this change. It is the oracle of the equivalence tests. - `server/src/__tests__/issue-graph-liveness-equivalence.test.ts`, `server/src/__tests__/review-attention-classification-equivalence.test.ts`, `server/src/__tests__/helpers/review-attention-fixture.ts`. - `server/scripts/benchmark-review-attention.ts`: the reproduction through the real routes. ## Verification Reproduction and before/after measurement. The script starts a local PostgreSQL, seeds a synthetic company, mounts the real `attentionRoutes` and `issueRoutes` with a board user, and sends the two requests over HTTP. With `--baseline-ref` it runs the pre-fix modules in a second process against the same database. It fails unless both versions return the same summary, the same full attention feed, the same task list and the same review attention for every task in review. ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000,9000 --agents=60 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000 --agents=180 ``` Fixture: one company with tasks of every status (10% in review, 10% blocked), four of five tasks are children, a reporting tree of agents with paused, terminated and pending agents. The reviews cover every kind of path and three ways to stall (31% stalled). The blocked tasks wait behind chains, cycles, cancelled, done, backlog and unassigned blockers. Open recovery tasks (5% of the tasks) name reviews, blocked tasks and done tasks. 6% of the reviews wait for a GitHub reply. A second company with the same shapes is in the same database. Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding. Each version ran in a new process, three rounds in alternating order, 2 warm-up requests and 5 measured requests in each round. A CPU profile of 3 more requests gives the last two columns. | Tasks / agents / reviews | Request | Version | Statements | Server ms, median (min) | CPU ms, median (min) | Main thread busy ms, median (min) | Allocated MB | Review classification ms | GC ms | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 3000 / 60 / 300 (94 stalled) | attention/summary | before | 69 | 121.4 (112.7) | 131.5 (114.4) | 80.2 (73.2) | 170.4 | 16.8 | 5.0 | | 3000 / 60 / 300 (94 stalled) | attention/summary | after | 69 | 105.9 (94.7) | 113.3 (96.3) | 67.3 (58.6) | 73.3 | 2.4 | 3.4 | | 9000 / 60 / 900 (283 stalled) | attention/summary | before | 85 | 280.9 (263.9) | 281.4 (253.4) | 203.7 (186.8) | 509.8 | 62.1 | 30.0 | | 9000 / 60 / 900 (283 stalled) | attention/summary | after | 85 | 224.8 (213.4) | 220.1 (205.8) | 149.9 (142.3) | 219.1 | 4.8 | 6.6 | | 3000 / 180 / 300 (94 stalled) | attention/summary | before | 69 | 241.2 (203.9) | 248.7 (227.4) | 197.1 (171.1) | 950.2 | 123.2 | 21.2 | | 3000 / 180 / 300 (94 stalled) | attention/summary | after | 69 | 127.8 (108.5) | 128.5 (102.0) | 76.7 (64.8) | 87.3 | 7.8 | 5.1 | | 3000 / 60 / 300 | task list (500 rows, 48 reviews) | before | 59 | 102.7 (96.7) | 92.4 (83.6) | 54.7 (48.9) | 73.2 | 2.6 | 2.4 | | 3000 / 60 / 300 | task list (500 rows, 48 reviews) | after | 59 | 106.2 (91.2) | 89.9 (78.8) | 55.2 (44.8) | 63.4 | 0.9 | 1.6 | | 9000 / 60 / 900 | task list (500 rows, 50 reviews) | before | 42 | 194.3 (186.0) | 77.5 (69.1) | 43.3 (38.4) | 57.3 | 1.0 | 1.8 | | 9000 / 60 / 900 | task list (500 rows, 50 reviews) | after | 42 | 189.6 (182.3) | 72.1 (69.5) | 41.9 (39.4) | 57.8 | 1.2 | 2.5 | | 3000 / 180 / 300 | task list (500 rows, 48 reviews) | before | 59 | 124.6 (95.7) | 110.2 (99.4) | 67.8 (58.1) | 165.4 | 15.4 | 3.8 | | 3000 / 180 / 300 | task list (500 rows, 48 reviews) | after | 59 | 104.1 (91.9) | 90.7 (79.9) | 56.2 (47.1) | 78.2 | 2.9 | 5.3 | How to read the table: - "Review classification ms" is the time in `listIssueReviewAttentionMap` and everything it calls on the main thread, for one request, from the CPU profile. It is the direct measure of this change. Before the fix it was the largest block of application code in the summary handler: 22% of the busy time at 3000 tasks, 20% at 9000 tasks, and 61% with 180 agents. `getAgentOrgChainHealth` under `isInvokableAgent` under `ownerCandidatesForRecoveryIssue` had most of it. After the fix it is 4% to 9%. - The scaling: three times the tasks gave 3.7 times the classification time before and 2.0 times after. Three times the agents gave 7.3 times before and 3.3 times after. - "Allocated MB" is all bytes that the server process allocated for one request (`total_allocated_bytes`), including the HTTP client of the script. - Other agents used the same machine during the measurement (load average 8 to 16 on 18 CPUs), so the medians of server time have noise. The profile columns and the allocation column are the stable ones. - The task list changes little with 60 agents, because one page has about 50 reviews. With 180 agents its classification falls from 15 ms to 3 ms and its allocation by half. - Statement counts do not change: this change removes no statement. Red/green regression test for the mechanism (`server/src/__tests__/issue-graph-liveness-equivalence.test.ts`): - `reads each agent and each path a number of times that does not grow with the reviews`. The test counts the reads of the agent status and of the path task id for 20 and for 400 stalled reviews with 40 agents and 360 paths. On the code before this change: `expected { agentStatus: 200800, pathIssueId: 144000 } to deeply equal { agentStatus: 10040, pathIssueId: 7200 }`. After: 316 agent reads and 360 path reads for both sizes (one read of each path). The test also runs the oracle and asserts that its reads grow, so it shows what it catches. - `classifies the review paths of a page from one index`. The same count for `createIssueReviewPathClassifier`. Equivalence tests (old code and new code on the same input): - `returns the findings and review paths of the pre-fix classification for random task graphs`. 400 seeded random graphs of two companies. Each graph has reviews with and without paths, blocked chains and cycles, terminal blockers, relations and paths that name the other company or a missing task, reporting chains with cycles, missing managers and managers of the other company, and ids that the input lists two times. The test compares the findings of the graph and the review paths of every task, for both `classifyIssueReviewPaths` and the new classifier. It asserts that the graphs reach all six finding states and all nine kinds of path, each at least 20 times. - `returns the pre-fix findings for blocked chains, cycles and terminal blockers`. One named graph with the expected state of each finding, so the comparison cannot pass on two wrong answers. - `server/src/__tests__/review-attention-classification-equivalence.test.ts`, on real PostgreSQL through the real routes. The test wraps the classification module: every classification that a request runs is also run by the oracle on the same input, and a difference fails the test. - `gives every review the paths the fixture was built with, and the pre-fix classification of every request`. `GET /issues?status=in_review`, `GET /issues` and `GET /issues?attention=blocked` (the blocked inbox, whose graph has the chains, cycles and terminal blockers). Each of the 64 reviews has the path kinds and the stalled reason that the fixture states for it, including reviews that wait for an external reply and reviews that open recovery tasks cover. - `counts the reviews that need a person, leaves out the ones a person dismissed, and reads one company`. `GET /attention/summary` and `GET /attention`: the review count and the review cards are the reviews that the fixture built to need a person. The second company has its own counts and none of the first company's tasks. Three dismissed cards leave the count and stay stalled in the task list. - The benchmark above compares the complete responses of both versions through the routes at each size (summary, full feed, task list, review attention of every review). Commands run locally: - `pnpm --filter @paperclipai/server exec vitest run` for `issue-graph-liveness-equivalence`, `review-attention-classification-equivalence`, `issue-liveness`, `recovery-classifiers`, `issue-review-attention`, `issue-blocker-attention`, `blocked-inbox-review-wait-scope`, `heartbeat-issue-liveness-escalation`, `attention-retention-work` and `attention-service`: 210 tests passed. In that run the `afterAll` hook of `attention-retention-work` and of `attention-service` (the stop of the test database) timed out at 30 s while other agents loaded the machine. No assertion failed. CI runs both files. - `tsc --noEmit` for the server package: no errors. The new test files and the script are outside the server `tsconfig`; a focused `tsc` program for them reports no error in these files. - `pnpm check:module-boundaries`, `node scripts/check-no-git-push.mjs` and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: pass. - The first CI run of this pull request failed `ci / Runner support suite`. An earlier revision also let the eligibility functions of `packages/shared` accept a prebuilt index, and the source of `packages/shared` is pinned for the runner SDK source proof (`tests/runner-e2e/source-sdk-public-inputs.json`). The fix does not need that change, so the current revision leaves `packages/shared` as it is on `master`: no pin moves, and the UI, the CLI and the runner use the same eligibility code as before. `source-sdk-public-inputs.test.mjs`, `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` pass locally on this revision (58 tests), and the drift guard reports no stale pin. - `pnpm verify` returns exit 2 (broader verification required) because the change adds a file under `server/scripts/`. CI supplies the broad coverage. The repo-wide typecheck, build and test suite were not run locally (shared machine). Not verified: - The effect on the hosted instance. This pull request does not deploy. The local requests take 0.1 s to 0.4 s, and the hosted requests took 15 s. Thus most of the hosted time is not this code: the hosted main thread is saturated by other work, and each of the 69 to 85 statements of a summary request waits for its turn on that thread. This change removes CPU time and allocation from that thread. It does not remove a statement. - The number of agents and of stalled reviews of the hosted company. The hosted allocation sample has most of the bytes in `classifyIssueReviewPaths`, which is the cost for each review (reviews × agents). The local fixture has more stalled reviews, where the owner candidates cost more. - A remaining cost inside the classification: each eligibility decision of `@paperclipai/shared` indexes the agent list again. One classification now does this one time for each agent (agents² map entries for one request, 3 ms with 180 agents), not one time for each check. The helper can accept a prebuilt index in a later change; that change moves the SDK source pins, so it needs its own review. - A second, independent cost that this pull request does not change: `GET /attention/summary` returns counts only, but it builds the full feed. After this change, the largest application blocks in its profile at 9000 tasks are the full task list of the 900 blocked tasks (15 ms on the main thread, of which 9 ms decode description previews that the summary never returns), the blocker attention of those tasks (9 ms), and the text of its 85 statements. A later change can give the summary a count-only path. ## Risks - The attention feed decides what a person is told needs them. The risk is a classification that differs from the previous one. Mitigation: the rules did not move. Only the lookups changed (a map lookup instead of a scan, a stored eligibility instead of a new one). The oracle tests compare the findings and the paths for random graphs, for a named graph and for every classification of the route tests. The benchmark compares the full responses. - The index is a snapshot of the input. `createIssueReviewPathClassifier` documents that the input must not change while the classifier is in use. The two callers build the input and use it in the same synchronous block. - When the input has no `now`, one classification now reads the clock one time. Before, `classifyIssueGraphLiveness` read it one time for the blocker checks and one more time for each review. All callers in the server that classify many tasks pass `now` or run in one synchronous call, so the values differ by less than the run time of the call. - The oracle file is a copy of the previous classification (about 640 lines, test-only). A later change of a classification rule must change the oracle in the same way. The header of the file says so. - The repository rule in `CONTRIBUTING.md` says that a pull request has no code comments. This pull request has comments where the code has a constraint that the names do not show (the memo key, the snapshot rule, the purpose of the oracle, the usage of the script), as the merged changes around this code do. - No migration. No query change. No change to company scope or to actor permissions: the change is after the reads, on rows that the same queries return. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests, CPU profiles). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1112.0-canary.13 |
||
|
|
df89950eb6 |
fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result. Then a finalizer exports the run's workspace files and applies the result to the task. > - When a newer run already decided the task, the finalizer retires the older result as superseded. Since #21106, this also ends the run. > - If that retirement interrupts a failed workspace export, the run's files stay unsaved. The run also stays the writer of the task's workspace. Nothing records those files for recovery, so nothing ever releases the workspace. > - Every run queued on the task then waits behind a save that never settles. After one hour, each queued run fails setup instead. > - This pull request records those files for the existing automatic workspace recovery. Recovery saves them from the retained sandbox, or, when every bounded attempt fails, continues the task from its last save with the files attached. > - The benefit is that the task's queue moves again by itself, and the run's unsaved work is saved or kept findable. The superseded result is never applied. ## Linked Issues or Issue Description Refs #21106 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. Its workspace export failed once with `remote_workspace_archive_fs_eacces`. The finalizer then retired it as `native_finalization_superseded`, because another run had already decided the task. #21106 ended that run at 14:59:17Z (`failed`). The task still could not move. The next queued run was cancelled with `workspace_busy` ("ended, but its workspace save has not settled"). Its retry re-armed every minute. Four more runs stayed queued. The task showed no recovery action. **Expected behavior** The ended run's unsaved files are saved from its retained sandbox, and the workspace is released. If they cannot be saved within the bounded retries, the task continues from its last committed save, and the files are attached to the task. The queued runs proceed in both cases. The superseded result is never applied. **Steps to reproduce** 1. Accept a native result for a run that writes a remote workspace. 2. Let the workspace export fail once, while a newer run has already committed the task's decision. 3. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is `failed`. 4. See that the workspace still names the run as its dirty writer, and that no retained-file recovery action exists. 5. Queue another run on the task. It defers with `workspace_busy`, and after one hour it fails setup with "The workspace still has an unfinished writer or save operation." **Root cause** `recordRetryableFailure` (`native-run-finalizer.ts`) retires a superseded finalizer without preserving the export. `preserveNativeFailedWorkspace` records a retained-file hold for an accepted result only when the failure code is `native_workspace_sync_out_retry_exhausted`. A `native_finalization_superseded` coordinator never qualifies. `readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the holder until its save settles. Without a hold, no worker ever settles it. **Paperclip version or commit** Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the same behavior. ## What Changed - `native-superseded-workspace-export.ts` (new): identifies a superseded finalizer that failed in the workspace export. It selects ended runs that still hold their workspace with unsaved files and no retained-file hold. It records the hold through `preserveNativeFailedWorkspace`. It checks again only while a just-ended run's sandbox stop receipt is still arriving (at most one hour). - `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace` treats a superseded workspace export like an exhausted copy-back, with the same proofs (run ended, no live controller, accepted result, confirmed sandbox stop-and-retain, source still bound to the workspace). It marks the hold `supersededWorkspaceExport`. - `native-run-projection-work.ts`: the native run projection worker also selects these runs. It loads them at startup and wakes when a finalizer changes. Rows written by older releases are therefore settled after deploy without a person. - `workspace-recovery-fallback-discovery.ts` and `workspace-recovery-fallback.ts`: a marked hold whose bounded re-export is spent (at most 5 retries in 30 minutes) is eligible for the existing continue-from-last-save fallback. That fallback sets aside only this run's workspace epoch and keeps its location on the workspace record. It attaches the readable files to the task unapplied. The task message says Paperclip "could not save" the previous run's unsaved files. - `preserved-result-finalization.ts`: a superseded result is never resumed for application, even after its files are saved. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (new). The first two tests fail on master and pass on this branch: - "records the files for automatic recovery, saves them from the retained sandbox and frees the workspace". It uses the staging row shape. The projection worker records the hold, and a second pass changes nothing. The workspace waits for that save. The system recovery saves `saved.txt` into a committed snapshot. The workspace becomes free. The finalizer, the result and the task's decision do not change, and the result is not resumed. On master the worker selects nothing (`expected 0 to be 1`). - "continues from the last save with the files attached when every re-export fails". After an exhausted re-export budget, the hold is resolved `restored`. The files are attached (`quarantine.state = captured`). The workspace moves to epoch 2 with no writer, and the abandoned epoch keeps its lease location. The message has no request for a person to fix anything. - "leaves a superseded result alone when it never failed in the workspace export". - Related suites pass: `native-run-projection-work`, `native-finalization-recovery`, `workspace-recovery-fallback`, `native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The change acts only on runs whose finalizer is already terminal and superseded, that failed in the workspace export, and that still own a dirty workspace. All existing preservation proofs still apply. - The saved files of a superseded run become the workspace's next saved version. The run's result (the task decision) is still never applied. - When the export can never succeed, the task continues from its last save. Files that cannot be read are not attached. The workspace record keeps the set-aside epoch's location. Saving the readable part of a partly unreadable workspace is not in scope. - Runs that already failed setup before this deploys are not restarted by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
782d6bead4 |
Configure managed plugin Runner lifecycle (#21065)
## Thinking Path > - Paperclip runs agents in managed environments. > - An agent can use an explicit environment or the instance default. > - A lifecycle plugin must receive the driver that provisions the agent. > - A managed plugin environment also needs to select the Runner process lifecycle. > - This change gives lifecycle selection and provisioning the same environment lookup. ## Linked Issues or Issue Description **What existing behavior does this improve?** Managed plugin environment provisioning and lifecycle configuration. **Current behavior** Lifecycle calls lack the selected plugin driver. Managed plugin configuration cannot select warm Runner execution. **Proposed behavior** Resolve the environment once per lifecycle operation. Pass the driver key for the plugin that serves it. Parse and persist the managed Runner lifecycle mode. The portable ownership and workspace placement changes are in [#21066](https://github.com/paperclipai/paperclip/pull/21066) and [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Share agent environment resolution between required plugin selection and the lifecycle RPC. - Pass `environmentDriverKey` to the plugin that serves the selected environment. - Allow managed plugin environments to set `warm` or `per_turn` and persist the selected mode. - Advertise `pluginEnvironmentRunnerLifecycleMode` in the release capability manifest so senders can check support before including the field. ## Verification - Built shared types, the plugin SDK, and Runner. Server typecheck and SDK source-pin check passed for the initial layer. - The capability contract test failed before the manifest fix and passed after it. Independent verification passed all 160 tests across managed configuration, plugin environment persistence, lifecycle routing, and environment reconciliation. - Routing coverage includes the instance default, an explicit second driver from the same plugin, and an explicit sandbox. - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862) passed on `f07b192c21`. The first attempt timed out in an unchanged workspace-recovery fixture and then collided with a background recovery action. The exact test and all 44 tests in its suite passed locally; the isolated shard retry passed without code changes. ## Risks - Existing managed configuration can omit the lifecycle mode. Senders must check the target release for `pluginEnvironmentRunnerLifecycleMode` before sending the new field. Publishing this signal does not update external deployment tools. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1af987690d |
perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process serves every request and agent turn on one thread. The same process saves each remote workspace as one encrypted archive in object storage, and restores it before a turn > - `server/src/services/remote-workspace-snapshot-storage.ts` hashed, encrypted and decrypted every byte of those archives on that thread. `server/src/storage/s3-multipart.ts` computed the Content-MD5 of each uploaded part on that thread too > - Hashes and ciphers are synchronous in Node. A CPU profile of a busy hosted server showed this work on the main thread, which was at 93% of one core > - Each megabyte of a snapshot delays every request and turn that waits behind it > - This pull request moves those passes to a small, bounded set of worker threads. The checks, their order, their failures and the stored format do not change > - The benefit is that a 1 GiB save uses about 0.7 s of the server's thread instead of about 2.8 s, and an unrelated request during the save is answered in 1.3 ms (p99) instead of 16.7 ms ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `workspace snapshot hash encrypt`, `event loop` and `archive io`. Related merged work: #19900 gave these stream passes 1 MiB buffers, #20554 removed a read-back from migration, #20722 took runner artifact hashes off the event loop, #20709 records long event-loop blocks. No open pull request covers this change. **What happened?** A 30 second CPU profile of the main thread of a hosted server (commit |
||
|
|
fb8b61bc3e |
fix(native): restart a bootstrap whose closing server staged files but never launched its Runner (#21134)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) in a Daytona sandbox. A release swap stops the old server and a new server recovers each run. > - When the old server stops during a run's bootstrap, the new server restarts that bootstrap on the same run (`bootstrap_incomplete`). The claim is granted only when the Postgres controller checkpoint proves that no provider turn was sent. > - The old server can stop after it staged launch files (Codex home, config) into the sandbox session folder, but before it launched the Runner. Its detach fence stops it there. > - The new server then finds no Runner state in the sandbox. A fresh bootstrap must create the session folder itself, and the folder already exists, so the run failed with `runner_harness_state_mismatch: resumed_unavailable`. The failure is an ambiguous session, so the task waits for a person. > - This pull request lets host preparation prove that no Runner was ever launched in that folder, move the folder aside, and restart the bootstrap. > - The benefit is that a release swap during this part of a bootstrap no longer blocks the task. No provider work can be lost or replayed, because no Runner ever ran. ## Linked Issues or Issue Description Refs #21120 (same release-swap class: the Runner was launched and connected before the old server stopped; that pull request stops it in host preparation. This pull request covers the case where no Runner was launched.) Refs #21079 (launch-time fence for a stale Runner on the same state folder.) **What happened?** On hosted staging, during a release swap on 2026-11-12, a Codex run on Daytona failed with `native_session_interrupted` and `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_unavailable`. The old server was in `harness_state.launch_material` when its shutdown drain detached the run. It recorded no Runner process identity. The new server chose "Restarting an incomplete native runner bootstrap on the same heartbeat run". Its host preparation found the sandbox resumed and no Runner state. Then the claim of the session folder failed, because the old server had already created that folder. The run needed a person. **Expected behavior** The run restarts its bootstrap on the same sandbox and continues without a person. A provider turn is never replayed and two Runners never share the run. **Steps to reproduce** 1. Start a Codex run on a Daytona sandbox. 2. Stop the server after the bootstrap stages launch files into the sandbox session folder and before it launches the Runner. 3. Let a new server recover the run with a `bootstrap_incomplete` claim. The new server has no local controller root for the run. 4. Host preparation fails with `runner_harness_state_mismatch: resumed_unavailable`. **Paperclip version or commit** Staging release `704b3f4d63`. The code is the same on master at `ef2b53b675`. **Deployment mode** Hosted staging, remote sandbox execution on Daytona. **Agent adapter(s) involved** Paperclip Runner with Codex (`codex_app_server`). ## What Changed - `native-session-executor.ts`: under a same-run `bootstrap_incomplete` claim with no failover backup, when host preparation finds no Runner state, a new check asks the sandbox in one command whether the session folder exists and holds no Runner trace. A Runner trace is the live `runner/runner-state.json`, the `runner/runner-process.identity` launch marker (the launch script writes it before the Runner starts), an `authority-epochs` archive, or any other `runner-state.json` or launch marker anywhere under the folder. If the folder holds no Runner trace, no Runner was launched there, so host preparation records the restart proof (`absent_confirmed`). It then uses the existing path: move the folder aside (it is kept for inspection), claim a new folder, and bootstrap again. If the folder is missing, the existing fresh-claim path runs unchanged. If any Runner trace exists, the run stays fail-closed as before. An archived session is therefore never discarded. - `native-session-executor.test.ts`: new `interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes restart the bootstrap. A launch marker, an archived authority epoch, Runner state elsewhere in the folder, or a failover backup keeps the run fail-closed with `resumed_unavailable`. ## Verification - Red: with the source change reverted, `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session" --maxWorkers=2` fails on both `interrupted_bootstrap_unlaunched` restart cases with `runner_harness_state_mismatch: resumed_unavailable`, which is the staging error. - Red: without the archived-state guard, the `_archived` and `_nested_state` cases fail (the folder is moved aside). With the guard they pass. - Green: the same command passes (64 tests). - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts --maxWorkers=2`: full file, 1422 passed before the guard commit. The focused 64 pass on the final revision. - `npx tsc --noEmit -p server/tsconfig.json`: passes. - `git merge-tree` against #21120's head (`0d18f18928`) merges without conflict. The new check runs only when no Runner state exists, so #21120's new call site (it runs only when a Runner state exists) gets the same results as before. ## Risks - Low. The new path runs only in host preparation, under a `bootstrap_incomplete` claim for the same run id, with no failover backup. That claim is granted only after the previous controller's lease has ended. With no Runner state or launch marker anywhere in the folder, no Runner and no provider session existed there. The old folder is moved aside, not deleted. - A controller that is still launching at the same moment would have to hold a lease that the claim already took over. The detach fence stops such a launch, and #21079 adds a launch-time fence for a leftover Runner on the same state folder. - No added wait before a normal turn starts: the check runs only on this failure path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser for read-only run evidence, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5b698b68cf |
perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update, insert, count/max read, COMMIT). The query builder built five of them again for every event, and each event created two transaction objects. A native source event now takes the run lock and then runs one statement that finds a replayed event or reserves the sequence and inserts, and reads the retained count and maximum for the acknowledgement. The lock stays a statement of its own so the append reads a snapshot taken after the lock was granted. The control-plane port passes its controller check to the append instead of opening an outer transaction, so there is no SAVEPOINT. The hot statements are fixed SQL text. Events that wake a queue still register the wake before the write and only when the event is new. Order, sequence numbers and durability do not change: every event still commits before its append returns. Local benchmark (server/scripts/benchmark-run-event-append.ts), one native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated, 1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events. One ordinary event: 105 -> 27 KB allocated. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5c953c1c1b |
fix(ui): keep chat pins first and allow drag reordering (#20830)
Keep pinned chat agents first in saved order. Add pointer and keyboard reordering with shared navigation pin controls. Preserve focus and hidden pin positions. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
9e910ff9ae |
fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields
## Linked Issues or Issue Description
**What happened?**
Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.
**Expected behavior**
Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.
**Steps to reproduce**
Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.
**Paperclip version or commit**
`master` at
|
||
|
|
8bdd53c11f |
perf(server): start no git process for server info outside a checkout (#21122)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reports which commit it runs. `GET /api/health` and `GET
/api/liveness` return it, and the board UI and the hosting control plane
poll these routes
> - The commit comes from `getServerInfoSnapshot()` in
`server/src/server-info.ts`. The health route calls it on each request.
The snapshot keeps its Git metadata for 3 seconds, then reads it again
with `execFileSync("git", ["show", ...])`, which is a synchronous child
process
> - A container or packaged install has the `git` command but no
checkout. There `git show` can only fail, and the server then uses the
commit that the build supplied. The failed start still stops the event
loop for the length of a process start
> - A CPU profile of a busy hosted server showed this cost on the main
thread
> - This pull request looks for a checkout before it starts `git`.
Without a checkout the server takes the build-commit path directly
> - The benefit is that health polls start no child process on a server
that does not run from a checkout, and the responses stay the same
## Linked Issues or Issue Description
No public issue exists. I searched open and closed pull requests for
`server-info git health`, `readGitInfo` and `getServerInfoSnapshot`.
Related merged work: #8753 added the 3 second refresh so that the debug
view follows a checkout, #10563 and #10566 added the build commit,
#18649 took the process start time off the Git probes. No open pull
request covers this change.
**What happened?**
`server/src/app.ts` creates the health routes without a `serverInfo`
option, so each `GET /api/health` request calls
`getServerInfoSnapshot()`. When the 3 second cache is expired, the call
starts `git show -s HEAD`. On an install without a checkout the command
fails each time, so the failure is paid again on each refresh. The rate
is one `git` start for each request when requests are 3 seconds or more
apart, and one start for each 3 seconds when requests are more frequent.
It is not one start for each request in a burst.
A 30 second CPU profile of the main thread of a hosted server (commit
|
||
|
|
ef2b53b675 |
fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - When the resume call times out, the run failed after 90 seconds with `reusable_sandbox_resume_failed (cause=transport_timeout)` > - The provider keeps starting the sandbox after the host call times out, so the sandbox is usually ready a little later > - The run gave up after one call, and the task then waited behind a Retry button > - This pull request keeps the same run retrying an idle sandbox whose resume only timed out, up to the provider's start deadline > - The benefit is that recovery works automatically and a sandbox with unsaved work is never destroyed ## Linked Issues or Issue Description Refs #20626, #20828, #21020 **What happened?** Three Codex runs on Daytona failed before provider dispatch with: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` Each run had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Each run took 100 to 122 seconds: one resume call used the whole 90-second budget, so no in-run retry was possible. The preserved leases were idle. Their last runs had ended about two hours before, with a confirmed stop-and-retain. The workspaces were not safe to replace, so the run failed strictly. The automatic bootstrap retry was then refused ("The automatic retry could not revalidate the original user continuation."), and all three tasks ended `blocked` behind a Retry. **Expected behavior** A resume that only timed out, before any provider work, on a sandbox that no live run holds, continues in the same run. The run keeps the same sandbox and does not create a replacement or destroy the sandbox. **Steps to reproduce** 1. Let a task's reusable Daytona sandbox stop and stay idle for some hours (it auto-archives after 60 minutes). 2. Retry the task. The workspace has unsaved work, so replacement is not safe. 3. The `environmentResumeLease` call takes longer than 90 seconds. The host call times out (`RPC call "environmentResumeLease" timed out after 90000ms`), the 90-second budget is spent, and the run fails with `reusable_sandbox_resume_failed`. **Paperclip version or commit** Staging at `849f109f` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `codex_local` (Codex) on a Daytona sandbox environment. The change is adapter-independent. **Root cause** `server/src/services/environment-runtime.ts`, reusable resume loop: the first `environmentResumeLease` call got the whole 90-second budget as its RPC timeout. When that call timed out, the loop saw no time left and threw. The 90-second budget assumes that the bounded bootstrap retry resumes the same sandbox later. That retry is not always admitted. A timeout is different from a refusal: the provider is still starting the sandbox. ## What Changed - `reusable-sandbox-resume-replacement.ts`: add `reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no other live run holds it). `reusableSandboxReplacementIsSafe` now uses it, with the same rules as before. Add `REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start deadline plus the call margin). - `environment-runtime.ts`: when the 90-second budget ends on a `transport_timeout` and the preserved lease is idle, extend the deadline one time, up to the provider start deadline (at most 330 seconds from the first attempt). The run keeps retrying the same sandbox with backoff. Each call still has at most 90 seconds. Refusals, a live owner, and other failures keep the current behavior. The run never replaces or destroys the sandbox because of this extension. - `doc/execution-semantics.md`: document the idle-timeout extension. ## Verification - Red before the fix, with the new tests in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (staging shape: released lease, finished owner run, workspace with unsaved work, 300-second provider deadline, host RPC timeout): 2 tests failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in the same run with no create or destroy call. If every call times out, the run stops at or before 330 seconds, makes more than one call, and keeps the lease `released`. A sandbox that a live run still holds gets one call and fails at 90 seconds, as before. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/services/remote-workspace-transport.test.ts`: 378 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - `pnpm verify` exits 2 because a doc is a global CI input. Required CI supplies the broad coverage. - Not run: a live Daytona resume on staging. No later run attempted the three preserved leases, so it is inferred, not observed, that the sandboxes were ready after the host call timed out. ## Risks - A run can now wait up to 330 seconds in preparation before its first turn, but only when the resume timed out on an idle sandbox. Before this change, that run failed and the task waited for a person. - A provider that is down (not slow) now takes up to 330 seconds to fail instead of 90 seconds. The failure is the same `reusable_sandbox_resume_failed`, and the lease is still preserved. - No persisted fields change. Leases saved by older releases follow the same rule. - The refused bootstrap retry admission for task Retry continuations is a separate gap. This pull request does not change it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.12 |
||
|
|
704b3f4d63 |
fix(server): a card accepted while its run settles keeps one continuation (#21094)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run ends while its task is in review, Paperclip checks that the task still has a maintained review path. If it has none, Paperclip queues one review-path recovery run > - An approval card for a tool action sends its continuation through a delivery receipt. The receipt waits until the source run has settled and the action has finished > - A person accepted a card while the source run was still settling. The interaction was no longer pending, and the continuation wake did not exist yet, so the review-path check found no path and queued a recovery run > - The real continuation then ran too, so the task got two follow-up runs for one decision > - This pull request counts an undelivered tool review continuation as a maintained review path > - The benefit is that a card accepted during settle produces exactly one continuation run ## Linked Issues or Issue Description **What happened?** An agent posted a tool approval card and its run finished. The person accepted the card 1.3 s later, while that run was still settling. The review-path check saw no pending interaction and no queued wake, and it queued an `issue_review_path_lost` recovery run. The accept continuation was deferred behind that run and then started as a second follow-up. **Expected behavior** The accepted card's pending continuation counts as the task's review path. Exactly one continuation runs. **Steps to reproduce** Have an agent in a task that is in review post a tool approval card. Accept the card while the creating run settles and before the tool action delivers its result. Before this change, a review-path recovery run was queued in addition to the continuation. ## What Changed - `server/src/services/issues.ts` (`listIssueReviewAttentionMap`): reads tool review continuations for the review tasks in one statement. A continuation counts as a queued review path when (a) its receipt is undelivered, its card is resolved, and the requesting assignee can still receive it (not paused, terminated or pending approval), or (b) its delivered continuation wake, or that wake's run, is still active. The path list leaves out a continuation whose wake or run is already listed, so each continuation shows once. The path shows as "Queued tool action review wake". - `heartbeat-issue-liveness-escalation.test.ts`: a new embedded-Postgres test runs a heartbeat for a task in review with an accepted card whose tool action is still executing. It checks that no review-path recovery wake is queued for that run. It then completes the action, delivers it, and checks that exactly one continuation wake exists. ## Verification - The new heartbeat test failed before the change (one `issue_review_path_lost` wake) and passes after it. - A new parameterized attention test covers an undelivered receipt, a delivered continuation (listed and not listed in the wake list), a pending card, a terminated assignee and a consumed continuation. The terminated, pending-card and delivered cases were red against the earlier revisions of this change. - `pnpm vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts src/__tests__/issue-review-attention.test.ts src/services/recovery/review-path-recovery.test.ts src/__tests__/attention-service.test.ts src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts src/__tests__/tool-gateway-service.test.ts`: 332 passed. - Product evidence: a paid chat-eval run of `questions-and-approvals.approve-tool-once` on a branch that combines this change with three related fixes. The result is in a PR comment. ## Risks - Every caller of review attention (review-path recovery, its escalation, the attention feed and the approvals view) now sees a pending tool review delivery as a covered path. That is the intended state: the delivery service always sends one continuation for it, and its periodic sweep retries a failed delivery. - One more indexed query for each review-attention read. It reads only undelivered receipts for the given tasks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d2e4d8d9 |
fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result, then a finalizer exports the workspace and applies the result to the task. > - When a newer run already committed the task's decision, the finalizer retires the older result as `native_finalization_superseded`. > - That retirement set the finalizer terminal, but it did not end the run. The run stayed `running` and held the task and every run queued behind it, with no way out except a person. > - A stuck run must never hold a task and its queue forever. > - This pull request ends the run when its finalizer is retired as superseded, and it ends runs that an older release left in that state. > - The benefit is that the task continues with its queued runs by itself. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. The workspace export then failed (`remote_workspace_archive_fs_eacces`). The task's last decision came from another run that was created 42 seconds later but ran earlier. The finalizer judged this run superseded: it set the finalizer to `terminal_failure` with `native_finalization_superseded`. The run stayed `running` with "Recovery needed". Over one hour later it still held the task, and five queued runs waited behind it. A server upgrade at 12:02Z did not change it. **Expected behavior** A run whose finalizer is retired ends. The task keeps the newer run's decision, the retired result is never applied, and the queued runs proceed. **Steps to reproduce** 1. Accept a native result for a run. 2. Commit a status decision on the same task from a run that was created later. 3. Let the older run's finalization fail once, so the finalizer records the failure. 4. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is still `running`. **Root cause** `recordRetryableFailure` (`server/src/services/native-runtime/native-run-finalizer.ts`) projects a terminal run status only when `exhausted && !supersededByNewerRun`. A superseded retirement therefore never ended the run. No other path ends a run whose finalizer is already `terminal_failure`: the reconciler skips `terminal_failure` rows. **Paperclip version or commit** Staging Core `3eca7a6112`. The behavior is the same on master `482063aeca`. ## What Changed - `native-run-finalizer.ts`: a superseded retirement now ends the run with the same status mapping as an exhausted finalization (workspace failure: `failed`; otherwise the provider's terminal state). It does not change the task, does not apply the result, and does not keep the workspace export lease. The run's error says in plain words that a newer run already decided the task. - `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`. It ends a native run that is still `running` while its finalizer is `terminal_failure` / `native_finalization_superseded`. This covers rows that an older release wrote. It locks in the finalizer's order (coordinator, task, run), and re-checks the row under the lock. - `native-run-projection-work.ts`: the registered native run projection worker (it loads at startup, pages through all candidates, and wakes on finalization changes) also selects such runs and ends them with `endSupersededNativeFinalizationRuns`. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/__tests__/native-finalization-recovery.test.ts`, red on master and green on this branch: - "retires an older failed finalizer when a newer run already committed the issue": the retired run now ends (`succeeded`, `finishedAt` set). Master: the run stays `running`. - "ends a run an earlier release left running after retiring its finalizer as superseded": the shape of the staging row (workspace export failed, superseded finalizer, run `running`). The repair ends it as `failed` with the plain message. The task, its decision and the finalizer stay unchanged, and no status decision is written for the run. A second pass changes nothing. - `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at startup, a run an earlier release left running after retiring its finalizer as superseded": the registered worker ends the staging-shaped row at startup and settles its claimed wake. Red without the worker change. - Related suites pass: `native-finalization-recovery`, `native-run-projection-work`, `native-finalization-reconciler-timing`, `native-reconciliation-timing`, `native-workspace-finalizer`, `native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`. - `tsc --noEmit` for `server`. ## Risks - Low risk. The change only ends runs whose finalizer is already terminal and superseded. It never applies a result and never changes the task. - A superseded run whose workspace export failed now releases its sandbox lease through the normal settlement instead of waiting forever. Its result was not applied, because a newer run decided the task. - Not changed here: the superseded check compares run creation times. A run created earlier but executed later is still judged superseded. This PR makes that verdict end the run; it does not change the verdict. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a2f618cae |
fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for a few minutes, then the old server gets SIGTERM and hands its native runs to the next controller. > - #21046 taught the legacy reaper to leave a run with its owner while the owner's server process is proven alive. Native runs have a similar gap. > - Runtime selection writes a native run's coordinator with no controller and no lease, and stops the run lease. The executor claims the coordinator about 20 seconds later. In that window the other server's sweep sees no owner and marks the run "ownership could not be verified". > - When the old server then hands the run off before any Runner checkpoint exists, no sweep looks at it again. Only the next server boot does, 15 to 40 minutes later. The task and its queued runs wait. > - This pull request leaves such a run with its live preparing server, and lets the serving server re-adopt a handed-off run through the ordinary restart claim. > - The benefit is that a run started during an upgrade continues within one sweep, with no false "blocked" mark and no wait for another upgrade. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, an upgrade started a new server at 12:19:57Z. The old server started a native run and selected its runtime at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership could not be verified; automatic recovery is blocked" (`observed_owner_unverified`). The old server's executor claimed the run at 12:20:51Z. The old server got SIGTERM and suspended the run at 12:20:54Z (`awaiting_runner_reattach`). After that the run showed "Confirming execution" for 29 minutes, and two queued runs waited behind it. At 12:49:45Z the next upgrade's startup recovery restarted its bootstrap, and the run succeeded at 12:58Z. **Expected behavior** - A server does not mark a run unverified, or recover it, while another live server is preparing it. - A run that a graceful shutdown hands to the next controller is re-adopted by the serving server at once, not at the next boot. **Steps to reproduce** 1. Seed a native run whose coordinator is `observed` with no controller and no lease, and whose `controllerProcess` names another server with a live controller instance lease. 2. Run the recovery sweep on this server. See that the run gets `native_execution_ownership_unverified`. 3. Claim the coordinator for the other server, then write what its graceful shutdown writes (`restart_suspended`, `awaiting_runner_reattach`, lapsed lease) and stop its instance lease. Give the run a predecessor's Runner PID. 4. Run the sweep again. See that nothing claims the run. **Root cause** - `recoverActiveExecutions` (`server/src/services/heartbeat/recovery.ts`) treats an `observed` coordinator without a live controller lease as an owner it cannot verify. Between runtime selection and the claim, the only owner on record is the run's `controllerProcess`, and the sweep did not check it. - `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`) selects runs for the restart claim only when they have a controller checkpoint, or no process identity at all. A run handed off just after its claim has no checkpoint, and a follow-up carries its predecessor's Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so only a server boot looked at them again. **Paperclip version or commit** Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server). The behavior is the same on master `849f109f62`. ## What Changed - `recovery.ts`: before it lists expired native runs, the sweep checks runs in the runtime-selection window (coordinator `observed`, no controller, no lease, not tracked here). When the run's `controllerProcess` names another server process that the shared owner prover proves alive, the sweep leaves the run with that server: it does not recover it and does not mark it unverified. It re-checks within 30 seconds. After 5 minutes without a claim (`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the existing rules apply. - `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that proof. - `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities` also selects a run that a graceful shutdown handed off and whose lease lapsed: recovery state `awaiting_runner_reattach` or `awaiting_evidence`, with a `restart_suspended` history entry that no restart claim followed. It stays selected through ownership waits (the departing server can still be alive at the first look). `recovery.ts` schedules a 30-second re-check for an ownership wait that has no retry of its own. The ordinary restart claim then proves the takeover and picks the disposition, exactly as the startup pass of a new server does. A row that an older release already marked unverified is re-adopted too: the claim clears the mark. - `doc/execution-semantics.md`: document both rules. - No schema change. Nothing changes on the path that starts a run or a turn: the new proof runs only in the recovery sweep, and only for runs in the window. ## Verification - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, block "a run the old server started during an upgrade overlap". Red on master, green on this branch: - "leaves the run with its live preparing server, then re-adopts it after that server hands it off": two concurrent sweeps leave the run `running` with no error and do not touch the coordinator. After the old server claims, hands the run off and stops, one sweep of the new server re-adopts it and reaches provider dispatch. Master: the run is marked `native_execution_ownership_unverified`, and after the handoff nothing claims it. - "keeps a handed-off run eligible while the departing server is still alive, and re-adopts it once it exits": the first sweep waits (`awaiting_evidence`), the sweep after the departing process exits re-adopts the run. Red with a filter that matches only the last history entry. - "re-adopts a handed-off run that an older release already marked ownership-unverified": the stuck staging shape. Master: nothing claims it. - Related suites pass: `heartbeat-native-restart-suspension` (all 109), `services/heartbeat/recovery.test.ts` (all 103). - `tsc --noEmit` for `server`. ## Risks - A run that another live server prepares, but never claims, now stays `running` for up to 5 minutes before the existing rules apply. That is the same bound #21046 uses. - The guard trusts the shared owner prover. It reads "alive" only from a live controller instance lease or a live PID in the same PID space. - More handed-off runs reach the restart claim earlier. The claim keeps all its fencing and evidence checks; this change only makes the serving server ask the same question a new boot asks. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a65644d1e1 |
fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks from a connected chat (for example GitHub pull request reviews) show Retry when a run fails > - A person's Retry on such a task first asks the chat retry owner to repeat the exact failed chat request. #21038 made a refusal at the click continue the task from saved work instead > - The chat owner can also defer its source check (a busy identity, endpoint, or agent row lock) and refuse it a few seconds later. Then the route has already answered 202, and the task Retry receipt was marked failed with no run > - The task stayed blocked, and its execution blocker told the person to send a message, while the interruption projection still offered Retry > - This pull request makes that late refusal take the same board continuation as a refusal at the click, and makes the blocker offer Retry for recovered saved work > - The benefit is that an accepted Retry always starts work, and the two projections agree ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, about 20 blocked tasks of one busy GitHub review agent (`paperclip_runner`, native remote runs) did not restart after Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`, a `chatActionId`, and `presentation.state: retry_scheduled`. No run was created. The task Retry receipt changed to `failed` 5 to 10 seconds later, and the task stayed blocked. The activity record of each click (`chat.failed_run_retry_requested`) has `sourceAuthorizationPending: true`, so the chat owner deferred its source check at the click. No chat wake row was written, so the chat replay ended before admission. The last run failed with `native_continuation_requires_reconciliation` / `runner_remote_process_identity_unavailable`. Its saved work was recovered (`workspaceRecovery.status: recovered`, remote lease destroyed with a termination receipt). The projections disagreed: `interruption` had `canRetry: true`, but `executionBlocker` had `canRetry: false` and `nextAction: "Saved work recovered. Send a message to continue."`. Root cause: when the chat retry action settles `failed`, `releaseUnadmittedTaskRetryInput` in `server/src/services/chat-channels/task-retry-input.ts` marked the linked task Retry receipt `failed`. #21038 added a board-continuation fallback only for a refusal inside the click transaction. A refusal after a deferred source check (for example 8 lock-busy attempts on the agent row, about 7 seconds) had no fallback. Separately, `getExecutionBlocker` hid Retry for every chat-bound task (`canRetry && !chatBinding`), and `workspaceRecoveryDisplay` always said "Send a message to continue" for recovered work. **Expected behavior** A Retry that is accepted starts work. If the exact chat replay is refused after the click, the person's Retry continues the task from its recovered saved work. The execution blocker and the interruption both offer Retry, and neither tells the person to send a message to unblock a failed run. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Make its latest native run fail with `native_continuation_requires_reconciliation` and recover its saved work. 2. Hold the chat identity advisory lock while you click Retry, so the source check is deferred (`sourceAuthorizationPending: true`). 3. Hold a row lock on the agent while the chat retry processes, so the deferred check fails 8 times. 4. The route returned 202. The chat action settles `failed`, the task Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** `master` at |
||
|
|
cc6728e8de |
fix(recovery): continue upgrade-stopped Retry runs that the sweep refused or skipped (#21108)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Hosted servers are upgraded often. Each upgrade stops the old server gracefully, and a run that is still preparing ends `interrupted` (`server_shutdown_interrupted`) with bootstrap evidence that provider work did not start. > - #21078 let a run that a person started with Retry continue automatically after such a stop. After it deployed, a read-only audit of a staging server found four groups of Retry-started tasks that still did not continue: some were refused on every sweep, some were never looked at, and one Retry that a person pressed failed and was not retried. > - Product rule: an upgrade never leaves a task that needs a manual Retry, and a Retry a person presses must work. A refusal that cannot succeed later must not repeat forever. > - This pull request fixes each cause found, keeps every authorization check of the continuation gate, and ends the refusal loop with one plain escalation. > - The benefit is that Retry-started tasks continue through upgrades, including tasks earlier releases left waiting, with no manual step. ## Linked Issues or Issue Description Refs #21078, #21046, #21019, #14987, #19384 **What happened?** On a hosted staging server after #21078 deployed (read-only audit, run events and task activity): 1. A Retry-started task was refused every ~2 minutes ("The automatic retry could not revalidate the original user continuation."). While preparing, its Retry had switched to a fresh isolated workspace because the saved files could not be restored (activity `issue.fresh_workspace_selected` by the system, 40 minutes after the Retry). That rewrote the Retry request's strategy from `continue` to `fresh_isolated_workspace`, and `readTaskRetryRetryAuthority` accepts only `continue`. 2. Two tasks whose Retry replaced a native session had no recovery event for hours. The stranded-task sweep skipped them silently. 3. A task whose Retry selected a fresh workspace at admission (`explicitUserContinuation.readiness`) was refused on purpose ("fresh-workspace Retries keep their own contracts"), yet that contract had no path for a shutdown before provider work. The task stayed in progress after every upgrade. 4. A person pressed Retry. The run failed before provider work with `reusable_sandbox_resume_failed`, a transient failure that is already in the bounded bootstrap retry lane. One second later the stranded-task sweep escalated the task to `blocked`, which cleared the claim. Two seconds after that, the run's own failure path scheduled its retry and was refused because the claim no longer named the run. **Expected behavior:** each of these continues automatically once, under the person's original Retry. A refusal that cannot succeed later escalates the task once with a working Retry. **Root causes:** - Cases 1 and 3: the continuation gate refused every fresh-workspace Retry, whether the fresh workspace was chosen at admission or later in preparation. That includes one that a shutdown stopped before it did anything. - Case 2: the sweep's generic skips run before the continuation. Two of them can match only because of state that the replaced session left. An unfinished session goal (`agent_task_sessions.goal_status`) is skipped as "goal recovery owns it". A native finalization row is counted as a live execution path by `hasActiveExecutionPath`. The person's Retry had already started a fresh session, so neither owns the task. The audit could not read the database, so which of the two applied on staging is not proven; the tests cover both. - Case 4: a race. The sweep does not wait while a terminal run's own executor is still settling it, so it escalates before that executor schedules the run's bounded retry. - The refusal loop: after a refusal the sweep left the task `in_progress` and tried again on the next pass. **Paperclip version or commit:** master `482063aeca`. ## What Changed - `explicit-native-continuation.ts`: `readTaskRetryRetryAuthority` renews a fresh-workspace Retry **only** when the parent run was stopped by a graceful shutdown before provider work (`isShutdownInterruptedPreparation`). The admitted `readiness` must name the same Retry and source. A strategy that preparation switched to fresh is rebuilt exactly as preparation records it: this Retry, its source, and the source's leases. An automatic retry of the fresh attempt inherits the selection from its own receipt. The successor's receipt records the selection under `automaticRetry.readiness`. The successor's `explicitUserContinuation` keeps its plain shape. - `fresh-workspace-continuation.ts`: `readFreshWorkspaceReadiness` reads that receipt for an automatic continuation, so the successor prepares the same fresh attempt. A remote fresh workspace is keyed by the Retry, so it is reused rather than created again. `freshReadinessSchema` is exported and reused. - `recovery/service.ts`: - **Shutdown-stopped Retry preparations go first:** at the start of the stranded sweep, every `in_progress` task whose latest run is a legacy `server_shutdown_interrupted` preparation (`providerWorkStarted: false`) carrying `explicitUserContinuation` is offered to the scheduler. It is skipped when the agent is not invokable, the task has a live execution path, a board-owned recovery action or a pause hold, or the run is still settling. The unfinished-goal skip does not apply here, because the person's Retry started a fresh session. - `hasActiveExecutionPath` ignores a native finalization that a person's Retry already replaced (`failureDetail.replacementDenied = explicit_user_continuation` on a terminal run). - The sweep skips a task while its latest terminal run is still being settled: a live execution in this process, or another server's unexpired controller lease. It checks again when that lease expires. - A refused continuation (`continuation_user_authorization_missing`) escalates the task once to `blocked` with the plain notice "Retry needed … Press Retry to continue." Before, the task stayed in progress and was refused again on the next pass. - `heartbeat.ts`: `scheduleRecoveryRetry` reports a refused continuation to the sweep. - `doc/execution-semantics.md`: describes these rules. **Failure classes admitted for a Retry-started run (case 4):** no new class. The run's own bounded retry lanes are unchanged. `reusable_sandbox_resume_failed`, `remote_workspace_restore_failed`, `plugin_worker_unavailable`, `environment_provider_unavailable`, `repository_catalog_unavailable`, runner revision mismatch and the other transient bootstrap failures were already retried before provider work. The fix removes the race that took the claim away from that retry. **Refusals that remain by design:** - The task is reassigned, `done` or `cancelled`. - Another run holds the claim, or newer or live work exists. - The Retry actor, generation or run changed, or it is a chat or queued-message Retry. - The message was edited or deleted. - The receipt is missing or revoked. - The run had another interruption cause, or provider work started. - A fresh-workspace Retry whose run *failed* (not a shutdown) keeps its one-run contract. - A readiness that names another Retry or another source. Each of these now escalates the task once instead of refusing on every sweep. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` sources and green on this branch (checked by restoring the master sources): - "continues a fresh-workspace Retry that a shutdown stopped before provider work": - Runs twice: once with the fresh workspace selected at admission, once with it selected at preparation. - Two concurrent sweeps create one successor, which gets the claim, and dispatch accepts it. - `readFreshWorkspaceReadiness(successor)` returns the same fresh attempt. - A second upgrade stops the successor, and the next successor carries the same attempt. - "continues a shutdown-stopped Retry whose replaced session left state behind": two cases, an unfinished session goal and a native finalization that the Retry replaced. - "escalates a refused shutdown continuation once instead of refusing it on every sweep": two sweeps. The task ends `blocked` with one "Press Retry to continue" notice and one refusal event. - "leaves a failed Retry run to its own executor while that executor still settles it": - Production shape: `reusable_sandbox_resume_failed` and a live controller lease from the server that ran the run. - The sweep leaves the task and its claim alone. - The run's bounded retry is then scheduled and dispatch accepts it. - "keeps a fresh-workspace Retry's own contract when it does not match": three refusal cases (other Retry, other source, failed rather than stopped). Green on both. - Existing suites pass: - `task-retry`, `explicit-native-continuation`, `heartbeat-retry-scheduling`, `heartbeat-process-recovery`, `interrupted-preparation-hold-settle`, `heartbeat-issue-liveness-escalation`, `issue-recovery-actions`, `recovery-deadline-acceptance`, `legacy-continuation-authority`, `workspace-recovery-admission`, `native-workspace-export-retry`, `heartbeat-paused-assignee-new-task`, `issue-tree-pause-resume-run`, `heartbeat-stale-queue-invalidation`. 1,619 tests passed, with one failure that a change in this branch fixed; rerun of the affected suites: 560/560. - The one failure came from the first version of the settling check, which also waited for this process's own lease after a shutdown drain. The check now uses this process's live executions and other processes' leases only. - `tsc --noEmit` for `server`. - Required CI covers the full suite. **After deploy, the next audit should show:** - Tasks stopped by a shutdown while their Retry was preparing get one `scheduled_retry` successor on the first sweep. This includes tasks whose Retry selected a fresh workspace and tasks whose Retry replaced a native session. - They show no further "could not revalidate" events. - A Retry-started run that fails transiently before provider work gets its bounded retry, and the task is not escalated in between. - A task whose continuation is truly refused is `blocked` once with "Retry needed". ## Risks - **Behavior change:** the sweep now offers shutdown-stopped Retry or message preparations to the scheduler before its generic skips. All authorization checks still run at admission, under the task and run row locks. The scheduler reuses an existing successor, so two servers create one continuation. The tests run two sweeps at once. - **Settling wait:** a task whose latest terminal run still has another server's live controller lease waits until that lease expires (at most 90 s) before the sweep acts. Normal turn starts are unchanged, and nothing is added before a turn starts. - **Escalation on refusal:** a refused continuation now moves the task to `blocked` with a Retry notice, where before it stayed `in_progress` with no path. Pressing Retry is a new, separately admitted request. - **Rows from older releases:** no schema change and no backfill. A shutdown-stopped fresh-workspace Retry is continued once. An older successor receipt without `automaticRetry.readiness` behaves as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, read-only API reads, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1d153f62b6 |
feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds, probes, gates and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after that image is published. > - Each master push rebuilds the image from scratch: about 10.6 min for amd64 and 7.8 min for arm64. > - The owner approved a merge queue for Core. A merge queue tests the exact commit that master will become, before the merge. > - This pull request builds and probes the image for that exact commit in the queue, with no registry credentials. The later master push pushes those exact bytes instead of rebuilding. When no matching verified build exists, master builds from source exactly as today. > - The benefit is that the image legs on master drop from about 8–11 min to about 3–4 min, and nothing changes until the owner turns the queue on. ## Linked Issues or Issue Description Refs #21013 (it measured the image path and proposed this plan). **What existing behavior does this improve?** Every master push rebuilds the standard image, even when the same commit could already have been built while it waited to merge. **Subsystem affected** `.github/workflows/docker.yml` (`build-and-push` only), new `.github/workflows/docker-merge-group.yml`, new `scripts/merge-group-image.mjs`. **Current behavior** `build-and-push` always runs the full BuildKit build on master: 639 s (amd64) and 468 s (arm64) in run 43133035209. **Proposed behavior** On a `merge_group` event, build both architectures for the merge-group commit, probe PID 1, and upload an OCI archive plus a record of commit, tree, architecture and digest. On the master push for the same commit, each leg pushes that archive by digest instead of building. Everything after the leg (probe, exact-source gate, signed-index check, attestation) is unchanged. **Reason and benefit** The build moves before the merge, where it runs in parallel with the queue's other checks. The master leg only downloads and pushes. **Breaking changes** None. Without `merge_group` events, `find` returns no run and the leg builds as before. **Additional context** This PR does not touch the exact-source gate jobs (`source_verification`, `publication_verification`, `merge-and-push`), `scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another change owns them. ## What Changed - **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs only for this repository's master queue. Workflow permissions are `{}` and the job has `contents: read` only. There are no secrets, no registry login, no `cache-to`, no `push: true`, no environments, and checkout uses `persist-credentials: false`. It runs on GitHub-hosted runners, not the paid AWS fleets. Steps: build `target: production` with the same build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA, which becomes the master SHA). Read the public registry cache. Write `type=oci` to a tar. `docker load` it and run `scripts/assert-orphan-reaping.sh`. Only then write `record.json` and upload `merge-group-image-<arch>` (3-day retention). - **`scripts/merge-group-image.mjs` (new).** - `find` (master push only): selects a completed, successful run of this workflow, with event `merge_group`, the same `head_sha`, repository and head repository both `1170821064`, and a `gh-readonly-queue/master/` head branch. It needs exactly one unexpired artifact for the architecture. - `record` (merge group only): reads the archive's `index.json`, checks each blob's sha256 and that the image is `linux/<arch>`, and writes `{sha, tree, arch, digest}`. - `verify` (master push only): re-reads the archive the same way and requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse HEAD^{tree}` of the master checkout, same architecture, and archive digest = recorded digest. - **`docker.yml` `build-and-push`.** - The leg gets `actions: read` so it can download the artifact. - New steps run after the registry login: find, download, then promote (`verify`, then `skopeo copy --all --preserve-digests oci-archive:… docker://ghcr.io/…@<digest>`). The step sets its `digest` output only after the push succeeds. - All three new steps are `continue-on-error`. Buildx, labels, metadata and "Build and push by digest" run when `steps.promote.outputs.digest == ''`, so any failure builds from source. - "Select image digest" feeds the existing PID 1 probe and digest export. - **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):** - Run selection rejects another commit, another event, another workflow, a fork head, another repository, another queue base, and failed, cancelled or running builds. - Artifact selection. - Archive digest with and without provenance, a tampered blob, the wrong architecture, and a real tar read through the system `tar`. - Record checks for commit, tree, architecture and digest mismatch. - The CLI refuses `record` outside `merge_group` and `find`/`verify` outside a canonical master push. - Static checks of the merge-group workflow: credentials, triggers, permissions and runners. - Static checks of the `docker.yml` wiring: promotion is gated and falls back, the output is set only after the push, and every build step has the fallback condition. ### How promotion proves it publishes the verified tree 1. `find` accepts only a run whose `head_sha` equals `github.sha` of the push, meaning the queue landed the commit it tested (unverified: see "Merge method" in Risks). 2. `verify` compares the record's tree with the tree of the master checkout, and its commit with `GITHUB_SHA`. 3. The archive is content-addressed. The digest is recomputed from the archive blobs and must equal the recorded digest. `skopeo --preserve-digests` to that digest reference fails rather than produce other bytes. 4. The leg then runs the PID 1 probe on that registry digest. `merge-and-push` still signs only an index whose Linux images are the probed digests (#21013), after the unchanged exact-source gate. 5. The pipeline itself is trusted only as master already ran it. `find` and `verify` run the copy of `scripts/merge-group-image.mjs` from the push's `before` commit (`git show "$BEFORE:…"`), not the copy the push brings. `verify` refuses promotion if anything in `before..HEAD` (every commit of a multi-PR group, not only the tip) changes `docker-merge-group.yml`, `docker.yml`, the promotion script or the PID 1 probe. Such a push builds from source. So does the first push after this PR merges, because its `before` has no verifier. ## Verification - `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds a real git history in which a two-PR push changes the pipeline in its first commit only, and checks that the range check catches it. - `node --test` on the workflow and contract suites (cloud-readiness, docker-canary-promotion, post-merge-runner-routing, lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image, standard-image-contract, cloud-source-verification, merge-group-image): 492 pass. - `vitest run server/src/__tests__/docker-build-stamp.test.ts server/src/__tests__/container-init-reaping.test.ts`: 22 pass. - `actionlint` on both workflows: no new findings (the 6 remaining are pre-existing `ls` and word-splitting notes in `docker.yml`). - Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI layouts kept the index digest. It accepted a `docker://…@sha256:` destination and failed only at the network step. Cannot be exercised before the queue exists, because no `merge_group` event fires: - the merge-group run itself: the hosted-runner build time, disk use with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a ~5 GB artifact upload; - the cross-run artifact download on master; - whether `skopeo` exists on the runs-on AMI and accepts digest destinations in its installed version. Each master-side failure falls back to the full build. A merge-group failure only blocks the queue if its check is required, which is why that check is required last in the steps below. ## Risks - **Inert until enabled.** No `merge_group` events fire before the owner adds a merge queue. On master, `find` lists runs (one API call per leg, about 1 s), finds none, and the leg builds as today. The only always-on change is the leg's `actions: read` permission. - **Build cache freshness.** A promoted master push writes no `buildcache-<arch>`, and merge-group builds read but never write it. With every push promoted, the cache ages, so dependency or weekly tool-layer changes rebuild in each merge-group run until a fallback build refreshes the cache. A follow-up can refresh the cache from master without blocking publication. - **Publication time is still gate-bound.** The exact-source gate (about 13–14 min after push) still decides when the image publishes. See "Expected timing". - **Merge method (unverified).** Promotion needs the master commit to be exactly the commit the queue tested, because the image stamps the commit SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack revision). Matching only the tree would publish an image stamped with another SHA, so this PR does not relax the check. I expect the queue to fast-forward master to its tested commit. GitHub's merge-queue page does not state this, and it cannot be tested before the queue exists. If the queue lands a different SHA, promotion never matches and every push builds from source. That is safe but inert. Step 4 below checks this. - **skopeo on the AWS fleet.** The promote step checks for `skopeo` first and logs "Merge-queue promotion unavailable" if it is missing, then builds from source. If that notice appears, add skopeo to the fleet image. - **Hosted runner capacity.** amd64 merge-group builds run on 4-core hosted runners, colder and slower than the AWS fleet. That adds queue latency, not master latency. ### Settings the owner must enable, in order 1. **Prerequisite, a code change, not a setting.** The `master` ruleset (id 13619726) requires `ci / verify` and `ci / e2e`. Those come from `pr.yml`, which triggers only on `pull_request`. A merge queue waits for required checks on the `merge_group` event too, so every entry would stall until its timeout. First decide, and merge in a separate PR, how `ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on the merge-group commit, or a fast pass that relies on the PR-head result. Today the ruleset does not require up-to-date branches either. 2. Merge this PR. Nothing changes yet. 3. In Settings → Rules → Rulesets → `master` (13619726), add **Require merge queue**: - merge method Squash; - build concurrency 5, minimum group size 1, maximum group size 5, wait 0 min; - "Require all queue entries to pass required checks"; - status check timeout 60 min. Do not add the image build to the required checks yet. Promotion is then opportunistic: if the merge-group build has not finished when master is pushed, master builds from source. 4. After the first queued merge, confirm promotion can match. The master commit SHA must equal the `head_sha` of its `Docker merge-queue build` run, and the next master `Docker` run's "Promote the merge-queue image" step must succeed. If the SHAs differ, promotion stays inert under this merge method. Stop here and report it. 5. After about a week of green `Docker merge-queue build` runs, add `build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64, ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's required status checks. Then every queued commit has a verified image ready to promote. 6. The existing bypass actors (org admins, repository admins, integration 5534676) can still merge outside the queue. Those pushes have no merge-group run and build from source. ### Rollback - Remove the image checks from the required status checks (step 5), or remove **Require merge queue** (step 3). Master pushes then find no merge-group run and build from source. Nothing to migrate. - Code: revert this PR. Old merge-group artifacts expire after 3 days. Promoted images are ordinary digests under `ghcr.io/paperclipai/paperclip` and stay valid. ### Expected timing - Image leg on master: about 3–4 min (download about 5 GB, push the new layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min (arm64). - Push to published: still bounded by the exact-source gate on master, about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is the same bound, but with the legs as a second 11-min path and the serialized-run queue in front. - If the gate later accepts proof produced in the merge queue (owned by the gate change), push to published would be about 4–5 min (leg promotion plus `merge-and-push`). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only GitHub API and registry reads). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
849f109f62 |
fix(server): agent inbox reads now apply private-task visibility (#21092)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents read a person's Mine list with `GET /api/agents/me/inbox/mine`, and their own queue with `GET /api/agents/me/inbox-lite` > - The company task list routes apply private-task visibility for the reading actor > - The two agent inbox routes should use the same visibility rules, so that each route returns the same tasks for the same actor > - This pull request makes agent inbox reads apply private-task visibility > - The benefit is one consistent visibility rule across task list routes ## Linked Issues or Issue Description **What existing behavior does this improve?** Agent inbox reads now apply private-task visibility. **Subsystem affected** Server API: agent inbox routes. **Current behavior** The agent inbox routes build their lists without the issue read condition that the company task list routes use. **Proposed behavior** The agent inbox routes use the same issue read condition and per-row read decisions as the company task list routes. **Reason and benefit** All task list routes apply one visibility rule. ## What Changed - `server/src/routes/agents.ts`: `GET /api/agents/me/inbox/mine` adds `issueReadSqlCondition` to its list query and keeps only the rows that `access.decideReads` allows. - `GET /api/agents/me/inbox-lite` adds the same read condition to its list query. - `inbox-mine-routes.test.ts`: a new embedded-Postgres test for the agent Mine read with private and open tasks. - `agent-permissions-routes.test.ts`: the mocked route test expects the read condition and the read decisions. ## Verification - `pnpm vitest run src/__tests__/inbox-mine-routes.test.ts`: 7 passed. The new test was red before the route change and is green after it. - `pnpm vitest run src/__tests__/agent-permissions-routes.test.ts src/__tests__/openapi-routes.test.ts`: all passed. - A chat eval run on a branch that combines this change with three other fixes. The result is in a PR comment. ## Risks - Low risk. The routes return only tasks that the reading agent can open through the task routes. - The Mine route adds one batched read decision for each request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.10 |
||
|
|
482063aeca |
fix(daytona): reuse or repair a read-only skill bundle instead of failing the sync (#21075)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner in a Daytona sandbox. Before a turn starts, Paperclip copies each selected skill into the sandbox as a read-only bundle. > - A sandbox is often kept between turns, so the read-only bundle is already there when the next turn (or a recovery) copies it again. > - The server writes the same skill files again on its own disk, with new timestamps. The copy compared the bundles with `tar --diff`, which also compares timestamps. So the copy saw a difference, extracted over the read-only directories, and failed with "Cannot open: File exists". > - The turn then stopped and the task stayed blocked until a person sent a message. > - This pull request treats a timestamp-only difference as the same bundle, and repairs a bundle that really differs instead of failing. > - The benefit is that a kept sandbox no longer blocks a task when its skills are copied again. ## Linked Issues or Issue Description **What happened?** On staging, two tasks in 30 minutes stopped with "Daytona syncIn extract command failed (exit 43): tar: .paperclip-repository/...: Cannot open: File exists". One failure was a normal new turn on a kept sandbox. The other was a recovery of an interrupted turn. Neither one started again by itself. **Expected behavior** The turn uses the bundle that is already in the sandbox when its content and permissions match. When the content does not match, the copy repairs it. **Steps to reproduce** 1. Copy a read-only (0555/0444) skill directory into a sandbox. 2. Change only the timestamps of the copy (this is what a new server preparation does). 3. Copy the same skill again. GNU tar fails with "Cannot open: File exists". ## What Changed - `file-sync.ts` (read-only directory copy): run the `tar --diff` check with `LC_ALL=C` and ignore lines that only say "Mod time differs". Bytes, modes, file types and missing files still count as differences. - When the bundle really differs, make its read-only subdirectories writable before the extract. GNU tar then replaces the files and restores the archived directory modes (`--delay-directory-restore`). Extra files in the target stay. - If reopening the directories or the repair extract fails, re-seal only the directories that were reopened for it (their list is recorded first). Writable user directories beside the bundle keep their modes. - `file-sync.test.ts`: the existing GNU tar test now covers a timestamp-only resync (kept, no failure) and a damaged bundle (repaired to the source bytes and modes, instead of the old fail-closed result), and a repair that cannot finish (bundle directories re-sealed to 0555, a writable user directory left at 0755), and a reopen that hits an unreadable user directory (bundle re-sealed, no scratch files left). ## Verification - Red: with the old `file-sync.ts`, the updated test fails with the same error seen on staging: `tar: references/overview.md: Cannot open: File exists`. - Green: `npx vitest run src/file-sync.test.ts` in `packages/plugins/sandbox-providers/daytona` passes 33/33 with GNU tar. - I reproduced the root cause by hand with GNU tar 1.35: a re-extract over a 0555 directory fails with "Cannot open: File exists", and `tar -df` reports only "Mod time differs" for a copy that differs only in timestamps. ## Risks - Low risk. The change affects only read-only (immutable) directory copies on Daytona. - A bundle that really differs is now repaired in place instead of failing the turn. Each file is replaced by tar (unlink and create). The directory modes come back from the archive. - If a target has extra read-only subdirectories that are not in the archive, they become owner-writable after a repair. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2a7fe6c585 |
fix(server): withhold a queued assignment run once its task moves to backlog (#21095)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Assigning a ready task to an agent queues an assignment run. Paperclip does not queue one for a backlog task > - An agent created a ready task, which queued an assignment run, and then moved the task to backlog. The agent was busy, so the run stayed queued > - When the run started, the queued-run gate did not check for backlog, and the run checked out and did the backlog task > - This pull request makes the queued-run gate withhold a never-started assignment run whose task is now in backlog > - The benefit is that moving a task to backlog keeps it parked, as when it is created in backlog ## Linked Issues or Issue Description **What happened?** An agent created a task without a status, so it was created ready with a queued assignment run. The agent then moved the task to backlog. The queued assignment run started two minutes later and completed the backlog task. **Expected behavior** A queued assignment run that has not started does not execute a task that has since moved to backlog. The run is cancelled with a clear reason. **Steps to reproduce** Assign a ready task to a busy agent so that its assignment run waits in the queue. Move the task to backlog. Let the agent become free. Before this change, the assignment run started and checked out the task. ## What Changed - `run-dispatch/domain/policy.ts` (`decideQueuedRunStaleness`): a queued run whose wake reason is `issue_assigned` is stale when its task is in `backlog`. The new error code is `issue_in_backlog`, with the reason "Cancelled because the task moved to backlog before its assignment run could start". - `pre-dispatch-review-wait.ts`: lists `issue_in_backlog` as a never-started dispatch cancellation, like the other queued-run gate codes. - `run-dispatch/adapters/postgres.test.ts`: a new test moves the task to backlog after the run is queued and checks that `cancelStaleQueuedRun` cancels it before it starts. A control test checks that a comment wake on a backlog task still starts. ## Verification - The new test failed before the change (`not_stale`) and passes after it. - `pnpm vitest run src/modules/run-dispatch/ src/services/pre-dispatch-review-wait`: 185 passed. - Product evidence: a paid chat-eval run of `questions-and-approvals.accept-one-proposal` on a branch that combines this change with three related fixes. The result is in a PR comment. ## Risks - Low risk. The rule applies only to assignment wakes. Comment, interaction and recovery wakes on a backlog task keep their current behavior. - To start the task, a person or agent moves it out of backlog. That wakes the assignee with `issue_status_changed`, as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
67403baaa4 |
fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first
## Linked Issues or Issue Description
Refs #20186 (the same treatment for a governance pause Stop)
**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.
**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.
**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.
## What Changed
- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.
## Verification
- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.
## Risks
- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ca62ca9e2a |
fix(release): keep the Runner chaos evals out of the Core image publication gate (#21088)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check
## Linked Issues or Issue Description
**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.
**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.
**Why**
Every minute on the gate is a minute on every staging release.
## What Changed
- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
- `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).
Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.
How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.
Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.
## Verification
- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
- With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.
## Risks
- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b1330c98e8 |
fix(evals): grade full cohort 6 false failures on saved state and keep failed-run evidence (#21089)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat agent does real product
work, and they grade saved state, not reply wording
> - Full cohort 6 (OpenCode with DeepSeek V4.1 Flash on master
|
||
|
|
6d8f59d494 |
fix(api): name the secret binding proposal as the way to give an agent a company secret (#21090)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents find product API routes through the runner API search,
which ranks operations by their OpenAPI summary and description
> - In the chat eval `secrets.configure-company-secret`, a person asked
the CEO to let Morgan use an existing deployment API key
> - The agent searched for agent configuration and environment variable
routes, tried `PATCH /api/agents/{id}` (403) and spent 3.5 minutes
before it used `POST /api/agents/me/secret-proposals`
> - The summary "Propose a company secret or agent secret binding" did
not match how agents describe the task
> - This pull request rewrites the summary and adds a description that
names the binding fields
> - The benefit is that the first search finds the supported route, so
the turn is short
## Linked Issues or Issue Description
**What happened?**
Runner API search did not rank `POST /api/agents/me/secret-proposals`
for the queries an agent uses to give another agent a company secret
("agent configuration update env secret", "agent environment variables
set secret ref"). The agent probed other routes for 3.5 minutes.
**Expected behavior**
Those searches return the secret binding proposal route in the first
results, and the result explains `secretId`, `configPath` and
`targetAgentId`.
**Steps to reproduce**
Call `searchRunnerApi({ query: "agent configuration update env secret",
limit: 3 })` on master. The proposal route is not in the results.
## What Changed
- `server/src/routes/openapi.ts`: new summary and description for `POST
/api/agents/me/secret-proposals`. The description states the
reporting-line target rule, the `env.<KEY>` config path, and that a
board member approves the binding.
- `server/src/__tests__/secret-binding-discovery.test.ts`: the four
queries return the route in their top three results. The test fails on
master (red) and passes with this change (green).
## Verification
- `npx vitest run` on 24 server guidance, discovery, OpenAPI and runner
API test files: 2305 passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid chat eval run: see the PR comment.
## Risks
- Low. Only OpenAPI text changes. Other search results can move down by
one place for queries that share these words.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3eca7a6112 |
fix(runner): log which check refuses a saved provider-pack admission on reattach (#21085)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode turns run in a remote Runner. When a server restarts, the new server reconnects to the Runner and reuses the provider pack admission that the run saved earlier. > - On staging, a run reconnected cleanly after an upgrade but then failed with `native_provider_artifacts_admission_unverified`. Its saved admission state stayed "pending". > - The reuse step refuses a saved admission for several reasons. It returns only false and logs nothing, so we cannot tell which check refused it. > - This pull request adds diagnostics only. It names the refused check in the run log and the server log. It also logs when the admission stays unverified for a session that needs it. > - The benefit is that the next upgrade audit shows the real cause, so the next fix can target it. ## Linked Issues or Issue Description **What happened?** After an automatic staging upgrade, a run reconnected to its live Runner without a new process. The next checkpoint failed with `native_provider_artifacts_admission_unverified`, which used one of the run's 3 attempts. The run log had no line that explained why the saved admission was not reused. **Expected behavior** When a saved admission is not reused, the run log and the server log name the check that refused it. **Steps to reproduce** 1. Start an OpenCode run on a remote sandbox. 2. Restart the server, so the run reconnects to its Runner. 3. If the saved admission is refused, the run fails later with `native_provider_artifacts_admission_unverified` and no reason in the logs. ## What Changed - `native-provider-pack-admission.ts`: `rememberRecoveredNativeProviderPack` accepts an optional `onRefused` callback. It is called with one reason: `source_state_pending`, `source_state_unknown`, `source_state_missing`, `receipt_missing`, `runner_instance_differs`, `environment_lease_differs`, `receipt_invalid`, or a session identity reason. The result of every check stays the same. An error in the callback is ignored. - `native-session-executor.ts`: the reattach path logs the reason to the run log and the server log. When `recordNativeProviderPackAdmission` returns false for a session that requires admission, it logs that too. - `native-provider-pack-admission.test.ts`: each refusal names its reason, and a reused admission reports nothing. ## Verification - `npx vitest run src/services/native-runtime/native-provider-pack-admission.test.ts` passes 46/46. The new reason tests fail on the old code because `onRefused` is never called. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1412/1412. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. This change adds diagnostics only, and admission decisions do not change. The new log lines contain only a fixed reason code and run ids. They do not contain secrets or file paths. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b853cc28ae |
perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server process starts, recovers saved work, and then reports ready on `/api/ready`. The deploy health gate waits for that signal. > - On a busy staging workspace, one boot took 85 s to bind the listener and 4.4 minutes to report ready. Small, idle stacks take 20–90 s. > - Two parts of that time do not need to block readiness. The server published several hundred UI assets one at a time before it bound the listener. After the listener bound, it waited for the first pass of each periodic recovery queue, and those passes grow with the backlog. > - The volume-free data gate, which can refuse a start, ran last. By then the server had already dispatched queued runs that a refusal would abandon. > - This pull request moves the refusal gate to the start, publishes the assets 8 at a time with replayable bodies, and runs the backlog-sized first passes after readiness, in their original order. > - The benefit is a faster time from container start to ready, and no work admitted before a gate that can refuse to start. ## Linked Issues or Issue Description No public issue exists. This is the problem description (enhancement template). **Is your enhancement related to a problem? Please describe.** Server boot time grows with the saved backlog. These timestamps come from the log of one staging boot (release |
||
|
|
2e593bf3cd |
fix(native): drop a repeated identity-record check and prove an unsettled predecessor stays an ordinary refusal (#21086)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21001 sends a re-wake to a fresh sandbox when live adoption of a crashed predecessor's warm Runner is refused, and lets an inherited Runner identity pass the "workspace unchanged" proof > - The reviewer found two gaps on #21001 that were not answered before it merged > - The unchanged proof checked for a `native.process_identity_recorded` event twice, and no test proved that an unsettled predecessor stays an ordinary refusal > - This pull request removes the repeated check and adds that proof > - The benefit is simpler proof code and a test that stops the fresh-sandbox fallback from widening to unsettled predecessors ## Linked Issues or Issue Description Refs #21001, #20863 **What happened?** Two review warnings on #21001: 1. `carriesOnlyInheritedRunnerIdentity` in `remote-workspace-unchanged-proof.ts` looked for the run's own `native.process_identity_recorded` event. `executionStartedEventPredicate()`, later in the same proof, already refuses a run with that event (`execution_recorded`). 2. The existing denial cases in `native-controller-recovery.integration.test.ts` use `toThrow(denied)`. That also matches the wrapped `NativePredecessorRetirementUnprovenError`, so a regression that wraps an unsettled predecessor's refusal would pass. **Expected behavior** The proof has one check per fact, and the tests fail when an unsettled predecessor goes to the fresh-sandbox fallback. **Steps to reproduce** 1. Read `proveRemoteWorkspaceUnchanged` and `executionStartedEventPredicate`. 2. Change `remote-warm-session-recovery.ts` to wrap every refusal; the old denial cases still pass. ## What Changed - `server/src/services/remote-workspace-unchanged-proof.ts`: delete `carriesOnlyInheritedRunnerIdentity` and its `PROCESS_IDENTITY_RECORDED` import; call `carriesInheritedRunnerIdentity(tx, run)` directly. A run with its own identity record is still refused, now by `execution_recorded`. - `server/src/services/native-runtime/native-controller-recovery.integration.test.ts`: - The denial cases for an unsettled predecessor (other company, other execution, finalization lease held, run still `running`) now assert the error is not a `NativePredecessorRetirementUnprovenError`. - The pending-command case after the predecessor is `succeeded` and `committed` now asserts it is one (the #21001 behavior). - The crash-restart case adds a predecessor run that is `failed` or `running`: an unsettled journal is refused with `warm_recovery_refused=owner_unsettled`, not wrapped. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/remote-workspace-unchanged-release.test.ts`: 50 passed. The three inherited-identity cases from #21001 return the same results (released / refused / refused). - `cd server && npx tsc --noEmit -p tsconfig.json`: passes. ## Risks - Low risk. No behavior change: the removed check is covered by `executionStartedEventPredicate()`. The rest is tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bed5f70278 |
perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Core image is published only after the release verification of the same commit passes (`cloud-readiness.yml` calls `release-verify.yml`) > - The image is ready about 3 minutes before that gate opens, so the slowest verification job sets the staging release time > - The slowest job is a general-server test shard: 11.8-12.7 min, while other server shards take 8-9 min > - The shards are balanced by a duration manifest, but the manifest has not changed since 2026-10-31. The weekly refresh that updates it failed on 2026-11-02 and 2026-11-09 > - About 40 newer suites per shard use the 1 s median fallback weight, but several of them run for 90-200 s, so they pile up on one shard > - This pull request repairs the weekly refresh and refreshes the server weights from three current master verifications > - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8 min of test time) on every master verification and every PR run ## Linked Issues or Issue Description **What happened?** The Core image publication waits about 13.5 min for source verification. The critical path is one general-server shard. In three master verifications (runs 43133035381, 43134251278, 43145252393) the slowest server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and 8.4 min. **Root cause** `scripts/general-server-shard-durations.json` came from one run on 2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice: - 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A server shard log is 30-65 MB. One download took 59 s locally. Three parallel downloads overran the 120 s `execFile` timeout. - 2026-11-09 (run 42580964686): the refresh's own check failed. Merged-PR sources for the Runner lane have a leading `pull` key, and the test accepted only `run, attempt, sha, url`. **Expected behavior** The weekly refresh finishes and keeps the shards balanced as suites are added. ## What Changed - `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a 512 MB buffer. JSON calls keep 120 s. A failed call now gives a short error and does not print the partial 60 MB log. - `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`. It accepts the optional leading `pull` key that `collectMergedPrSamples` writes. The checked-in Runner manifest test and the collector test both use it, so the collector format stays covered. - `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 -> 45 min (the server lanes alone fetch about 30 large logs). - `scripts/general-server-shard-durations.json` and `scripts/serialized-shard-durations.json`: refreshed with `node scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381 43134251278 43145252393` (median of three successful protected-master Cloud readiness runs; 1757/1815 general files and 173/173 serialized files measured). - No test is added, removed, skipped or reassigned to a different lane. No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check are not touched. ## Verification Measured job timings (minutes from the first job start; three master verifications): | Job | 43133035381 | 43134251278 | 43145252393 | | --- | --- | --- | --- | | Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7 | | Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 | | Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 | | Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 | | Chat (slowest of 3) | 8.8 | 10.5 | 9.4 | | Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 | | Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 | | Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 | - Queue to runner start is 0.5-1.1 min for every job. In a server shard, setup (checkout, pnpm, Node, install) is about 50 s and the preflight plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8 min). Setup is not worth caching. - Shard prediction, using the refreshed medians as the cost of each file and the 3 fork workers per shard: - general-server, 10 shards: old weights max 10.3 / min 8.1 min; new weights 8.8 / 8.8 min. - serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights 6.6 / 6.6 min. - The old-weight prediction matches what CI saw: predicted 10.3 and 9.9 min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in run 43133035381. - Red/green: - `node --test --test-name-pattern="gh api" scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix (`log timeout 120000`) and passes after it. - With a `pull` key in the Runner manifest sources, the old "package lane manifest" test failed with the same assertion as run 42580964686. The new test passes. - The refresh script, which died before the fix, completed locally in 8 min after it. - `node --test scripts/ci/test-selection-timings.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the complete, non-overlapping shard cover checks. - `actionlint .github/workflows/ci-test-durations.yml`: clean. - Not run: the release-verify workflow itself, the scheduled refresh workflow, and the Runner/e2e lane refresh. The shard times above are predictions from logged per-file timings, not a measured CI run on this branch. ## Risks - Low risk. Weights only move files between shards of the same lane. The shard tests prove every file still runs exactly once. - The predicted gain is about 1.5 min on the slowest server shard (12-12.7 -> about 10.5 min job time). The gate then waits for "Pre-release Runner chaos evals / Restart, replay, trace, and recovery faults" (10.2-11.9 min). So the expected gate saving is about 1 min (13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a shard is about ±1 min. - A larger log timeout can make a stuck download wait up to 15 min. The 45 min job timeout still bounds the run. Candidates for the owner (not changed here): - The chaos-eval job is now the bound. Its "server finalization" step takes 7-9 min. Decide if it must gate image publication, or split it. - Server shard logs are up to 811k lines, mostly repeated stack traces from `server/src/services/execution-preparation-timing.ts`. That slows log upload and this refresh. - 9 test files run over 180 s (largest: `chat-channels.integration.test.ts` 429 s, `heartbeat-process-recovery.test.ts` 344 s). One file cannot be split across shards. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, GitHub CLI read-only). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Related: #19650 (earlier repair of the weekly refresh), #21013 (Core image publish time; separate files). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f8e1fb0278 |
perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha |
||
|
|
5820d43197 |
fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.
## Linked Issues or Issue Description
**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.
**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.
**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.
## What Changed
- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.
## Verification
- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
db8848f559 |
fix(heartbeat): bound the I7 queued-run check with a resumable cursor (#21022)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20986 added a reconciler check that reports a queued run whose dispatch signal was lost (I7 `queued_run_not_dispatched`), and a worker that re-drives such runs. > - The Ponytail review of #20986 (4/5 on `fbab23cce6`) found three issues. #20986 merged at that head before the fix commit was pushed. > - The new I7 check read every queued run with no cap, while the report still said that each I7 check reads at most 2,000 records. > - This pull request adds that fix commit to master. The check now reads a bounded slice per pass and continues from a cursor. The page cursor no longer depends on the session date style. An agent with a scheduled retry now counts as busy. > - The benefit is that the reconciler pass stays bounded when many runs are held on purpose, and the coverage statement is true again. ## Linked Issues or Issue Description Refs #20986 Follow-up to the Ponytail review of #20986: https://github.com/paperclipai/paperclip/pull/20986#pullrequestreview-5868879754 (review of `fbab23cce6`, score 4/5). **What is wrong?** 1. `checkQueuedRunsDispatched` (`server/src/services/recovery/ownership-invariant-report.ts`) paged through every undispatched queued run with no cap. `COVERAGE_GAPS.record_cap` still said each I7 check reads at most 2,000 records per pass. 2. The page cursor cast `created_at::text` back with `::timestamptz`. That works only with the default `DateStyle`. 3. A queued run on an agent with a `scheduled_retry` run was reported and re-driven, but `checkVisibleNextAction` already treats `scheduled_retry` as an active run. **Expected behavior** The I7 queued-run check reads a bounded slice per pass and reaches every run over later passes. The cursor format does not depend on session settings. A scheduled retry counts as the agent's next action. ## What Changed - `server/src/services/queued-run-dispatch-liveness.ts`: new `readUndispatchedQueuedRuns`. It reads from an optional cursor until the set ends or `maxRuns` rows were read, and returns the next cursor. `listUndispatchedQueuedRuns` uses it with no limit. The cursor text is `to_char(created_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`, which keeps microsecond precision and does not depend on `DateStyle`. The live-run exclusion now also covers `status = 'scheduled_retry'`. - `server/src/services/recovery/ownership-invariant-report.ts`: the I7 queued-run check reads at most `MAX_RECORDS_PER_KIND` (2,000) per pass. It keeps a keyset cursor in `OwnershipReportState.queuedRunCursor`, and the next pass continues from that cursor. A pass that stops at the cap keeps a follow-up pass due (`retryNeeded`), like the I3 and I4 backlogs. `COVERAGE_GAPS.record_cap` now describes this check. - `server/src/__tests__/ownership-reconciliation-report.test.ts`: new test. Bounded slices resume from the cursor and end with `next: null`. A `scheduled_retry` run on the agent removes its queued runs from the set. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/heartbeat-held-comment-wake.test.ts`: 75 tests pass on this branch (fresh master + this commit). - Server `tsc --noEmit` passes. - The restart integration test from #20986 ("starts a continuation that queued behind its predecessor's sandbox release after a graceful restart") passed with this change on the #20986 branch, in both cases (signalled and unsignalled). - Restart-chaos CR1 360 s, Codex W2, `--daytona-warm --ephemeral-server`, seed 9512 (`w-20261112T040314-s9512`, on the #20986 change set before this commit): the fault crossed the task and every rule passed (R1–R6, T, U1–U5). ## Risks - Low risk. The report reads less per pass. A very large held queue is now covered over several passes instead of one. - A queued run behind a `scheduled_retry` is no longer re-driven by the backstop. The scheduled retry's own promotion signals dispatch. - No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.9 |
||
|
|
559551e977 |
fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a reused warm sandbox first retires the previous run's warm session, and that retirement collects the old session's agent files > - On a server without a persistent volume, agent-file collection takes a per-agent Postgres lock, so runs of one agent collect one at a time > - #20863 sends a run whose predecessor cannot be retired to a fresh sandbox. It classified every retirement error as unprovable, including a busy lock held by another run of the same agent > - So ordinary lock contention failed the run, dropped a healthy sandbox and rested the task > - This pull request makes retirement wait for a busy directory lock in the same run, and keeps the fresh-sandbox fallback for retirements that stay unprovable after the lock is obtained > - The benefit is that busy agents no longer fail follow-up runs on their warm sandboxes ## Linked Issues or Issue Description **What happened?** A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent on Daytona, with a volume-free server, failed one monitor-due follow-up after 6 seconds. The error was `adapter_failed`, cause `native_predecessor_retirement_unproven`: "The previous run's session on this sandbox could not be retired (Agent directory operation is already active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The stack shows the error came from `claimWarmNativeInstructionCopy` during instruction preparation. The previous session had closed; only its agent-file collection met the per-agent lock that another run held. **Expected behavior** A busy agent-directory lock is transient. The run waits for it and then retires the predecessor on the same sandbox. Only a retirement that stays unprovable after the lock is obtained takes the fresh-sandbox fallback. **Steps to reproduce** 1. Run several warm native tasks of one agent at once on Daytona with a volume-free server. 2. Re-wake one task while another run of the agent collects its files. 3. The follow-up fails with `native_predecessor_retirement_unproven` and its sandbox is dropped. Refs #20863, #20829, #20957. ## What Changed - `retireWarmPredecessor` (native executor): a busy agent-directory lock (`AgentDirectoryOperationBusyError`) is waited out in the same run, polling every second for up to five minutes (`warmRetirementDirectoryWait`). The usual holder is another run's short collection, so retirement then finishes on the same sandbox with the copy this run adopted. The run is not deferred: deferring after adoption would release the adopted copy and could skip saving the predecessor's edits. - Past that budget the run fails with its own error, `native_predecessor_retirement_directory_busy`, not as an unprovable retirement. Before failing it files the adopted copy's retirement receipt, so the copy is kept as `pending_collection` and run cleanup does not mark it `unavailable`. The owner is not marked unretired and stays registered for its sandbox, so a later run retires it there. - An idle-timeout close that meets the lock retries after the agent-directory retry delay instead of leaving the session registered with no retry. - Closing idle sessions for environment delete (#20957) reports such an owner as `busy`, not `failed`. ## Verification - RED on master, then GREEN: - `native-session-executor.test.ts -t "directory lock"`: two runs of one agent contend on the lock while a follow-up retires the predecessor. On master the claim rejects with `NativePredecessorRetirementUnprovenError` (`Agent directory operation is already active`). Now a short-lived lock is waited out and the predecessor is retired and collected on the same sandbox; a lock that outlasts the budget fails with `native_predecessor_retirement_directory_busy`, environment delete reports `busy: 1`, a fresh-sandbox attempt does not forget the owner, and once the lock is free a later run retires it. - `native-session-executor.test.ts -t "idle close whose file collection"`: on master the idle close never retries; now it retries and closes. - Existing suites pass: native executor, agent-directory busy heartbeat, native pre-dispatch retry, native controller recovery and plugin runner eviction. Server `tsc --noEmit` passes. ## Risks - Under contention a follow-up can wait up to five minutes before its first turn. The normal path adds no wait. - A lock held for more than five minutes fails the run visibly instead of dropping the sandbox. - An idle owner whose collection stays busy keeps retrying at the 5 to 15 second agent-directory delay until the lock frees, a controller restart, or the next admission. - No persisted state changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
533f135780 |
fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a warm remote sandbox leaves its Runner `ready` in the sandbox. The next run of the task adopts that Runner live, after it proves that the previous owner finished > - When the server is killed after a run parked on a question, the warm owner lived only in the killed server's memory. The Runner in the sandbox was never suspended > - A re-wake on the replacement server resumed that sandbox and tried live adoption. A proof failed, and the new run failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry, although it did no provider work > - #20863 already sends such a run to a fresh sandbox when the predecessor's Runner cannot be retired, but the live-adoption path of a resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to it, and the failed run's inherited Runner identity blocked the "workspace unchanged" proof that the fallback needs > - This pull request connects the two: a refusal of live adoption after the predecessor is durably terminal with an accepted result goes to the #20863 fresh-sandbox path, and an inherited Runner identity no longer counts as a process of the follow-up > - The benefit is that a re-wake seconds after a crash restart continues without a Retry ## Linked Issues or Issue Description Refs #20863, #20844, #20928, #20979, #20953, #20972 **What happened?** Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`, volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh home and overlap), task t08 (W2Q): - Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm owner stayed in the server's memory. - K1 killed the server at 01:30:06. The replacement was ready at 01:30:15. - A board comment woke the task at 01:30:17. Run `8f812df6` started at 01:30:19 and resumed the same sandbox (`run.startup.reuse` `verified_reuse`). - The first live-adoption inspection passed. The second check (`revalidate`, in backend setup, before the Runner attach) refused at 01:30:21 (stack: `remote-warm-session-recovery.ts:217` from `revalidate`). The run failed `native_session_interrupted` with `INSTRUCTION_COLLECTION_UNAVAILABLE`. - The failed run kept its full save because the unchanged-files proof refused with `process_recorded`: the run carried its predecessor's Runner process identity. Retry (`6968dc0f`) worked. **Expected behavior** The predecessor finished with an accepted result before the kill, and the new run did no provider work. The task must continue without a Retry. **Steps to reproduce** 1. Finish a warm remote run that parks on a question. 2. Kill the server (SIGKILL) and start a replacement. 3. Wake the task within seconds. The new run resumes the warm sandbox. 4. Any live-adoption proof fails, and the run fails terminally with `prior_owner_terminal_state_indeterminate`. **Which proof failed** The old refusal had no reason. The wave evidence proves that it was `revalidate`, after a passed inspection, and not the SIGTERM of the next K15 fault (that server logged "shutdown signal received" only at 01:31:09). `revalidate` checks the predecessor's coordinator row, the current run's coordinator row and lease, and the task. The evidence cannot tell which one changed in the first seconds after the restart. This pull request names the failed proof (`warm_recovery_refused=<reason>`), so the next occurrence identifies it. The fix covers the class: every refusal that concerns the predecessor or the sandbox. ## What Changed - `server/src/services/native-runtime/remote-warm-session-recovery.ts`: - Each refusal names its proof from a closed list: `identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`, `execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`, `controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`, `current_lease_changed`, `current_run_not_admitted`, `current_owner_changed`. - After the predecessor's run is `succeeded` and its coordinator is `committed` with a result and no lease, a refusal about the predecessor, its journal or the sandbox throws `NativePredecessorRetirementUnprovenError` (predecessor run, sandbox). The existing #20863 path then marks the sandbox `replaceBeforeReuse` and re-queues the run in the uncharged `interrupted_preparation` lane. The fresh sandbox is used only when the workspace save is clean and the harness backup is verified (unchanged #20863 safety rules); otherwise the sandbox is resumed as before. - A refusal because the new run is no longer admitted (stopped, reassigned, paused, company inactive) or no longer owned by this controller stays an ordinary failure, so a fallback cannot become a path around Stop or reassignment. - `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose recorded Runner process identity is exactly a settled predecessor's identity of the same session (`carriesInheritedRunnerIdentity`, from #20979), with no `native.process_identity_recorded` event of its own, is no longer refused with `process_recorded`. A provider session of its own (`sessionIdAfter`) and every other check still refuse. - `doc/execution-semantics.md`: document both rules. ## Verification - Red on `origin/master` (`8818402e7a`): - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "crash restart"`: `AssertionError: expected Error: runner_state_identity_mismatch: pr… to be an instance of NativePredecessorRetirementUnprovenError`. - `npx vitest run src/services/remote-workspace-unchanged-release.test.ts -t "follow-up that carries"`: `expected false to be true` for the inherited-identity case; the own-identity and own-record cases passed (still refused). - Green with the fix: `native-controller-recovery.integration`, `remote-workspace-unchanged-release`, `native-recovered-owner-followup` and `heartbeat-native-predispatch-retry` pass (149 tests). Also run: `native-session-executor.test`, `explicit-native-continuation.test`, `task-workspace-binding.test`, `native-runner-restart-recovery.integration.test` (see the PR comment for the result). `tsc --noEmit` for the server passes. - The new integration case is the staging shape: a predecessor that parked on a question (unanswered `semantic_tool.input` before its terminal), a replacement controller, a resumed sandbox. It checks the journal refusal, a predecessor row taken by another worker between inspection and revalidation, a changed current lease, and that a cancelled current run is not sent to the fallback. - Chaos: the before/after rerun of seed 7000 is in a PR comment. ## Risks - The fallback starts the next attempt on a fresh sandbox instead of reusing the warm Runner. That costs one cold start, only after live adoption is refused. - The unchanged-files proof accepts an inherited Runner identity. The identity must match a settled predecessor of the same session exactly (PID, group and start time), and the run must have no own identity record, no execution event, no runner checkpoint, no workspace operation and no result. These are the same facts #20979 uses for restart recovery. - Error text of this refusal gains `; warm_recovery_refused=<reason>`. Callers match the `runner_state_identity_mismatch: ` prefix only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0cfab73520 |
test(chaos): W2C changes the agent configuration the way the settings form does (#21011)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos re-wake cell W2C changes the agent's configuration
after a run succeeds and before the task is woken again, so the next run
cannot resume the saved session
> - The harness sent a partial `adapterConfig` patch (`{ idleTimeoutMs
}`). On a volume-free server that patch is refused for the Codex agent
with 422 "Volume-free servers require managed agent instructions", so
the configuration never changed and W2C cells only tested W2
> - The agent settings form does not send a partial patch:
`buildAgentUpdatePatch` (`ui/src/lib/agent-config-patch.ts`) sends the
whole merged `adapterConfig` with `replaceAdapterConfig: true`
> - This pull request makes W2C change the configuration the same way
the settings form does
> - The benefit is that the W2C cell exercises the configuration change
on every provider while the partial-patch refusal is handled separately
## Linked Issues or Issue Description
**What is the problem?**
W2C's configuration change is refused with 422 for a Codex agent on a
volume-free server, so the cell does not test what it is named for.
**What is the expected behavior?**
W2C changes `idleTimeoutMs` the way a person would in the agent settings
form, and the change succeeds.
**Steps to reproduce**
Run a Daytona wave with `--ephemeral-server --adapters codex --workloads
W2C --faults K0`. The timeline records `rewake_config_changed` with
`httpStatus: 422` and the check `config_changed_before_wake` fails.
## What Changed
- `tests/restart-chaos/rewake.ts`: W2C reads the agent and PATCHes `{
adapterConfig: { ...existing, idleTimeoutMs }, replaceAdapterConfig:
true }`.
- `tests/restart-chaos/shape-cells.test.ts`: the W2C step test expects
the full-config patch.
- `tests/restart-chaos/README.md`: W2C describes the patch it sends.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/shape-cells.test.ts` passes.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- A volume-free Codex W2C wave is queued to confirm the change returns
200 live.
## Risks
- Low risk. Test harness only. The partial-patch 422 itself is a
separate product question and is not changed here.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ad2d123dad |
fix(recovery): count a current response-wake wait as a visible next action in I7 (#20994)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The ownership reconciler checks invariant I7: an agent-assigned `in_progress` issue must have a visible next action. A live run, a pending wake, an open recovery action or a saved monitor deadline counts. > - #20935 added a parked wait. When a native agent yields `response_wake` and no response source is admitted, the arbiter commits `response_wake_waiting_for_input`. The task stays `in_progress`, and the run shows "Waiting for a reply". > - I7 did not know this wait. A task that really waits for the board was reported as `no_owner_no_wake`. > - This pull request makes I7 count that wait as a visible next action while it is still current. I7 still reports the issue when the wait is stale. > - The benefit is that a genuinely parked task is not reported as a silent stall, and a stale wait is still reported. ## Linked Issues or Issue Description **What happened?** The ownership reconciler reported `I7_visible_next_action` / `no_owner_no_wake` for an issue whose latest applied decision was `response_wake_waiting_for_input`. That is the visible "Waiting for a reply" state from #20935. **Expected behavior** A current, committed response-wake wait is a visible next action. The issue is reported again when a user comment arrives after the waiting run started, or when that decision is no longer the issue's latest decision. **Steps to reproduce** 1. Commit a native run whose arbiter decision is `response_wake_waiting_for_input`, and make it the issue's latest decision. 2. Wait past the 5-minute next-action grace window. 3. Run an ownership reconciliation pass. Before this change, I7 reports the issue. **Additional context** Follow-up to #20989 and #20935. The conditions match `hasCommittedNativeResponseWakeWait`, which recovery already uses for the same wait. Refs #20935 Refs #20989 ## What Changed - `server/src/services/recovery/ownership-invariant-report.ts`: `checkVisibleNextAction` excludes an issue with a parked response-wake wait. The wait must meet all of these conditions: - The issue's `lastStatusDecisionId` is an applied `response_wake_waiting_for_input` decision. - Its native finalization is `committed`. - Its run succeeded and belongs to the current agent assignee. - The issue has no user assignee and is not hidden. - No user comment arrived after that run started. - New test in `server/src/__tests__/ownership-reconciliation-report.test.ts`: - A current wait is not reported. - A later user comment makes I7 report the issue. - A decision that is no longer current also makes I7 report it. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts -t "response-wake wait"`. I7 reported the waiting issue. - Green: the full `ownership-reconciliation-report.test.ts` and `environment-pending-cleanup-queue-owner.test.ts` pass (43 tests). `npx tsc --noEmit -p server` passes. - Restart-chaos harness, for reviewers: the harness U1/R1 grading should treat a sample as waiting, not ownerless, when the issue's latest run shows the execution projection phase `waiting_for_answer` ("Waiting for a reply"). It should still fail when a user comment is newer than that run. The harness reads I7 through `tests/restart-chaos/invariants.ts`, so I7 changes with this PR. The U1 sampler does not read the projection yet, and this PR does not change it. ## Risks - Low risk. The change is in a report-only check. It writes nothing and changes no admission, status or recovery path. - If the conditions are too wide, a stalled task is hidden. The conditions are the same as recovery's durable-wait check, and the test proves that a stale wait is still reported. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c4927e1ec2 |
fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade stops the old server gracefully. A legacy run that is still preparing ends `interrupted` with `server_shutdown_interrupted`. The new server must continue it automatically (#21019, #21046). > - A run that a person started with Retry (or a message) carries `explicitUserContinuation`. Its automatic retries need their own receipt from `admitExplicitContinuationRetry` (#14987, #19384). > - That admission refused every shutdown-interrupted Retry run. The task then stayed in progress with no run, and the stranded-task sweep refused it again about every 2.5 minutes, forever. A person had to press Retry. > - An upgrade is routine. It must never leave a task that needs a manual Retry. > - This pull request lets the person's original authorization cover a continuation after a shutdown that stopped the run before provider work started. All other checks of the gate stay. > - The benefit is that Retry-started tasks continue through an upgrade, and tasks that earlier releases left waiting continue on their own after deploy. ## Linked Issues or Issue Description Refs #21019, #21046, #14987, #19384 **What happened?** On a hosted staging server, each automatic upgrade stopped legacy runs that were still preparing (`executionRecovery {kind: bootstrap, providerWorkStarted: false}`). When the run came from a person's Retry (wake reason `task_retry`), the continuation was refused with "The automatic retry could not revalidate the original user continuation." (`continuation_user_authorization_missing`). The task stayed `in_progress` with no run and no next retry. Runs from older upgrades were refused again by each sweep. A run from the same upgrade that did not come from a Retry got its continuation in the `interrupted_preparation` lane. **Expected behavior** A Retry-started run that a graceful shutdown stopped before provider work continues automatically, once, under the person's original Retry. Tasks that an earlier release already refused continue after deploy with no manual step. **Root cause** Three conditions refused it (`server/src/services/heartbeat/retries.ts`, `server/src/services/explicit-native-continuation.ts`): 1. The scheduler admits an explicit continuation only for the `transient_failure`, `server_restart`, preparation-wait and agent-directory lanes. A shutdown-interrupted preparation with unspent failure budget uses `interrupted_preparation`, so it was refused before admission ran. 2. `admitExplicitContinuationRetry` accepted only a `failed` or `timed_out` parent. These parents are `interrupted`. 3. It required `task.executionRunId` to be the parent. The refusal released the claim (`releaseIssueExecutionAndPromote`), so every later sweep was refused by this check too. **Paperclip version or commit** master `f30005440b`. ## What Changed - `retries.ts`: an explicit continuation may also use the `interrupted_preparation` lane. Lane choice is unchanged (see below). - `explicit-native-continuation.ts`: `admitExplicitContinuationRetry` also accepts a parent that is `interrupted` by a graceful shutdown (`isServerReplacementInterruption`, so `server_shutdown_interrupted`) with bootstrap evidence `providerWorkStarted: false` (`isShutdownInterruptedPreparation`). - For such a parent only, a released claim (`executionRunId` null) is accepted. Admission restores the claim to the parent under the task lock, as the shutdown path leaves it, so the scheduler passes the claim to the successor and dispatch accepts it. If the parent validation then refuses, admission releases the claim again in the same transaction. A claim that another run holds still refuses. - `doc/execution-semantics.md`: describe the rule and the conditions that still refuse. Also apply the sentence that #19384 left pending (a plain task Retry is renewable). - No schema change, no backfill. Nothing changes on the path that starts a turn. **Which lane a graceful-shutdown legacy preparation uses (after #21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation` for an `interrupted` run with the bootstrap evidence. `usesServerRestartLane(run, true)` returns false while the failure budget is unspent, so the run uses `interrupted_preparation`: it does not spend the failure budget, the first continuation is due at once, and it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a spent failure budget it uses `server_restart`. This is the intended design (#21046 comment on `usesServerRestartLane`), so no lane change. The `server_restart` lane is for `process_lost` replaced preparations and for runs past dispatch. **Conditions that still refuse** (each has a test or an existing test): - The task has another assignee, or is `done` or `cancelled`. - Another run holds the task's claim. - Newer work on the task (a run created after the parent, a retry child, or a live run). - The Retry request changed: other actor, not admitted, other generation, bound to another run, a chat Retry, a fresh-workspace Retry. A message continuation with an edited or deleted message. - A missing or revoked receipt (dispatch validation of the parent fails). - An execution blocker on the task, or a live adapter controller for the parent. - An `interrupted` parent with another cause (for example an operator interrupt), or with no bootstrap evidence, or with `providerWorkStarted: true`. A parent that failed after provider work keeps the existing rules (`failed`/`timed_out` only). - Native runtime parents (unchanged, see below). **Native runtime.** A native `paperclip_runner` run is not interrupted by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and the next server reattaches it (`restart_suspended`). It does not reach this gate. Native replacement paths (`native-safe-replacement`, `native-provider-capacity-retry`) remove `explicitUserContinuation` from the successor and use their own contract. So the native path does not have this refusal, and this pull request does not change it. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` (no successor), green on this branch: - "continues a Retry-admitted run that a graceful shutdown stopped while it was preparing": production shape. A legacy run admitted by a task Retry, `preparing`, owned by this server, with bootstrap evidence. `drainRunningRunsForShutdown("SIGTERM")` interrupts it with `server_shutdown_interrupted` and schedules one `interrupted_preparation` successor (`failureRetries: 0`, `interruptedPreparations: 1`) with its own receipt. The claim passes to the successor and dispatch accepts it. A concurrent stranded sweep and a second schedule call reuse the same successor. - "continues a task an earlier release left waiting after refusing its shutdown continuation" (a direct Retry run, and an automatic retry of a Retry run): the row and task as an older release left them (claim released). Two concurrent stranded sweeps schedule exactly one successor; the claim passes to it; dispatch accepts it. - "does not continue a shutdown-interrupted Retry run without its original authority" for: provider work started, no bootstrap proof, another interruption cause, claim held by another run, reassigned, superseded, other Retry actor, cancelled task. No successor, no receipt, and the claim is left as it was. - I confirmed that each source change is needed: without the lane change both main tests fail; without the released-claim rule the older-release test fails. - Suites: `task-retry`, `explicit-native-continuation`, `legacy-explicit-continuation-recovery`, `quarantined-task-retry` (587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling` (520/520). - `tsc --noEmit` for `server`. - `pnpm verify` selects a broad run (exit 2) because `retries.ts` is central; required CI covers that. **After deploy, the next audit should show:** a Retry-started run that a shutdown stops while preparing gets a `scheduled_retry` successor with `scheduledRetryReason: interrupted_preparation` (or `server_restart` with a spent failure budget) and no `continuation_user_authorization_missing` event. The tasks that earlier upgrades left waiting get one such successor on the first stranded sweep of the new server, with the task claim on the successor. ## Risks - Behavior change: a shutdown-interrupted preparation of a Retry or message turn now continues automatically, within the existing uncharged and bounded lanes. The run did no provider work, so there is nothing to reconcile. - Rows saved by older releases: a refused row whose claim was released is continued once. If another run took the claim, or newer work exists, it stays refused. - Two servers at once: admission runs in the scheduler transaction under the task and run row locks, and the scheduler reuses an existing successor (`retryOfRunId`). The tests run two sweeps concurrently and get one successor. An old server that still refuses cannot clear the claim after the new server moved it to the successor (its release matches only the stopped run). - The claim restore writes `executionRunId` for a terminal run inside the scheduling transaction. If admission refuses after the restore, it releases the claim again before the transaction commits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
814fe5d20f |
fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox start `runnerd` in the sandbox, and Paperclip connects to it through provider ingress and authenticates it with a one-use bootstrap ticket > - The ticket lives for 60 seconds, but Paperclip issued it before the remote launcher staged its files in the sandbox > - Under load, staging took 60 to 81 seconds, so `runnerd` presented an expired ticket and every connection was closed. The run failed with `runner_ingress_auth_failed`, and the task stopped behind a manual Retry > - This pull request issues a fresh ticket right before dispatch, retries these failures automatically when the controller checkpoint proves no runner ever authenticated, and adds spans that attribute the slow startup window > - The benefit is that a slow but healthy start authenticates, a failed start recovers on its own, and the remaining startup delay becomes measurable ## Linked Issues or Issue Description **What happened?** On a hosted instance, 9 runs of one Codex agent on Daytona failed in about 70 minutes with `native_session_interrupted` and the message `runner_ingress_auth_failed: Runner ingress did not complete PRP authentication before the deadline.` In the same window only 6 runs succeeded company-wide. The run log showed Paperclip reconnecting for 60 seconds while `runnerd` reported that its peer closed each WebSocket. In 294 remote runs with timing spans, the time between the bootstrap ticket issue (end of `runner.transport.connect`) and the process dispatch (start of `runner.process.launch`) decides the outcome without exception: - 60 s or more: 23 of 23 runs failed authentication. - Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s. The tasks did not recover by themselves, because the failure was classified as ambiguous state. **Expected behavior** A runner that starts slowly but correctly authenticates. If ingress authentication still fails before any runner authenticated, Paperclip retries the start automatically. The time between the start request and the runner connection is visible in run spans. **Steps to reproduce** 1. Run a Codex agent on a Daytona sandbox environment that uses provider ingress. 2. Make remote staging (`runner.runtime.stage`, inside the remote launcher) take 60 seconds or more. Many concurrent runs of the same agent do this on staging. 3. See that `runner.prp.authenticate` fails after 60 seconds and the run ends with `runner_ingress_auth_failed`. 4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts` reproduces this locally with a 75-second staging delay. **Paperclip version or commit** `4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`. **Deployment mode** Hosted (cloud), controller on Railway, sandboxes on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`), remote sandbox target, provider-ingress transport. ## What Changed - `RunnerProcessLaunchSpec` and `spawnRunner` accept `refreshBootstrapTicket`. A restart does not inherit it. - The runnerd Codex transport passes a refresh that issues a new bootstrap ticket and flushes the controller checkpoint. It does not do this for a warm-transition launch. - The remote sandbox launcher calls the refresh after `ensureArtifact` and immediately before the launch RPC, and puts the new ticket in the launch environment. - `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller checkpoint that no runner authenticated: no connections, no leases, no used tickets, no events, and only pending commands. - `nativeSessionFailureSourceCode` classifies the outbound ingress startup failures as `runner_ingress_startup_failed`. `nativeProviderRecoveryEvidence` treats it as pre-session (`bootstrap_retry`) only when that proof holds for the exact run identity and there is no provider session or provider event. Restart recovery and failed-run retry accept the new code with the same rules as similar startup codes. - Remote startup now records `runner.control_plane.open`, `runner.launch.prepare`, `runner.checkpoint.register` and `stage.asset.codex_skills.verify`. The 6 to 11 minute gap between `native.process_start_requested` and `runner.transport.connect` had no spans before. ## Verification - Red/green: `packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts` starts a real `runnerd` with a 75-second staging delay. Without the fix it fails with `runner never connected`. With the fix it opens the session. - Red/green: `native-session-executor.test.ts` "launches runnerd with a bootstrap ticket issued after remote staging". Without the fix the launch RPC gets the stale ticket. - Red/green: `native-controller-recovery.integration.test.ts` "retries a fresh remote start whose launched runner never authenticated through provider ingress" (embedded Postgres). Without the fix the failure is `native_session_interrupted` and `ambiguous_state`. - `durable-prp-control-plane.test.ts`: the new checkpoint proof, its negative cases, and the restart rule for the refresh. - `native-run-trace.test.ts`: remote startup spans are recorded without duplicate launcher spans. - Focused suites pass: runner package 5 files and 235 tests; server 9 files and 1,607 tests, including `native-session-executor`, `native-runner-execution-driver`, `remote-launch-assets`, `native-restart-recovery`, `runner-prp-outbound` and the recovery integration tests. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - A launch now issues two bootstrap tickets. The first one is never sent to `runnerd` and expires in 60 seconds. Bootstrap counters already allow more than one issued credential. - The new retry path needs a checkpoint proof that no runner authenticated, so a run that did provider work cannot be retried this way. The existing attempt budget still bounds the retries. - This change does not remove the 6 to 11 minute controller-side delay before `runner.transport.connect`. The new spans make it attributable for a follow-up change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f30005440b |
fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.
## Linked Issues or Issue Description
**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.
**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.
**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.
**Paperclip version or commit**
master at
canary/v2026.1112.0-canary.8
|
||
|
|
b7693e2076 |
fix(runner): decode native inputs saved under ACPX 0.13.1 (#21062)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner defines the native execution input that every native run saves and every later run reads back > - The ACPX upgrade to 0.22.0 (#20770) changed the qualified `acpxVersion` in each ACPX profile > - The input parser accepts only the current `acpxVersion`, so it refuses every input that was saved under ACPX 0.13.1 > - Run preparation reads the previous run's saved input for the same task, so each ACPX agent with an older task session fails before its provider starts > - This pull request lets the parser decode inputs saved under ACPX 0.13.1 and normalizes them to the current qualified profile > - The benefit is that saved ACPX agents and tasks work again after the upgrade, with no manual edit ## Linked Issues or Issue Description Refs #20770 **What happened?** Every run of an ACPX agent (Claude through ACPX) failed about 20 seconds after it started. The run error was `input.provider.profile does not match the qualified ACPX v1 profile` with the failure cause `native_execution_input_invalid`. The run log showed no runner output, only the preparation messages. Five runs on five different tasks failed this way in 40 minutes. The previous successful run on one of those tasks saved this provider profile: `{"agent":"claude","acpxVersion":"0.13.1","agentProfileVersion":1,...}`. New runs saved `acpxVersion` `0.22.0`. Run preparation reads the previous run's saved input (for example, the instruction working copy step parses the input of the task session's last run). The parser refused `0.13.1` and the error stopped the run. **Expected behavior** A run on a task whose earlier run was saved under ACPX 0.13.1 prepares and starts normally. The ACPX upgrade carries saved sessions forward. The session identity already pins `0.13.1` for this reason (`recovery-identity.ts`). The saved input must decode too. **Steps to reproduce** 1. Run an ACPX agent on a task with Paperclip before #20770, so the run saves `provider.profile.acpxVersion: "0.13.1"`. 2. Deploy a build that includes #20770. 3. Wake the agent on the same task. 4. See the run fail in preparation with `input.provider.profile does not match the qualified ACPX v1 profile`. 5. The new unit test reproduces this with the exact saved profile shape. **Paperclip version or commit** master at |
||
|
|
a858016082 |
fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master
|
||
|
|
4228bcf89c |
fix(opencode): never save the model's reasoning as its chat reply (#21051)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents that use OpenCode models reply in chat through the Runner's OpenCode driver. The driver settles one final response, and Paperclip saves it as the reply comment > - In chat-eval cohort 5 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`), four saved replies were the agent's reasoning, for example "Confirmed down. Now report honestly. ... I'll finish with paperclip_finish." > - The run events show the cause. The model wrote a reasoning part, then wrote the same text again as an ordinary text part, and ended the step without `paperclip_finish`. The driver took that copy as the final answer. This is not the #20999 path: no completion call happened > - A second, related problem is in the #20999/#21025 path. When the model writes an answer, calls `paperclip_finish`, and then writes its own new answer, the driver can save both texts, or a status note together with the answer > - This pull request drops the repeated reasoning, lets the existing empty-response correction ask for the real answer, and saves the post-completion message alone unless it is a short line that refers back > - The benefit is that people never see the agent's planning text or tool names as its reply, and the eval catalog now fails when they do ## Linked Issues or Issue Description Refs #20999 Refs #21025 **What happened?** The saved chat reply held the agent's reasoning, not an answer. Cohort 5 cases: `connection-setup.unreachable-mcp`, `plugins.configure-plugin`, `tasks.website-task`, `teams.install-team`. In each case the reply text is byte-identical to the reasoning part before it, and the run result evidence is `provider:opencode:agent-message` (no `paperclip_finish` call). Also, `recovery.explain-stall` saved its answer twice, and `status-cards.refresh-card` saved a "STATUS: ..." note above its answer. **Expected behavior** The saved reply is the model's answer to the person. It never holds reasoning, completion tool names or report field names. A short closing line such as "Answered above" still keeps the earlier answer. **Steps to reproduce** 1. Use an OpenCode agent with `openrouter/deepseek/deepseek-v4.1-flash-1017`. 2. Ask a chat request that the model finishes without a completion call after a long reasoning step. 3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds the reasoning text. **Paperclip version or commit** master at |
||
|
|
91e26ff237 |
fix(ui): refetch an open chat's comments when its conversation settles (#21054)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to agents in chat, and the open chat updates live when
an agent replies
> - Run finalization can save a turn's reply after the run's terminal
status is published, and that save logs no `issue.comment_added`
activity
> - Finalization then settles the conversation, which logs
`issue.updated`, but the UI did not refetch comments on that update
> - So the reply stayed hidden in the open chat until the person
reloaded the page
> - This pull request makes the open chat refetch its saved comments
when its conversation is updated
> - The benefit is that a person sees the agent's reply without a reload
## Linked Issues or Issue Description
**What happened?**
In the chat eval cohort on master
|
||
|
|
fd9985b571 |
fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes talk to a Runner through durable PRP commands. The controller waits for each command's result > - After a server replacement, the new controller reattached to live Runners. For runs it reached later than about 115 s after detach, `session.snapshot` got no answer within 30 s > - The failure said only `provider_transport_failed: PRP command session.snapshot timed out`. The run log could not tell whether the Runner was disconnected, whether the command waited behind older commands, or when the Runner last sent an event > - This pull request adds that evidence to the timeout message. It names types, counts and durations only, never payloads > - The benefit is that the next upgrade's failures explain the stall, so the runner-side cause can be fixed ## Linked Issues or Issue Description Refs #21053, #21052, #21039 **What happened?** At the 07:10Z staging swap, 9 remote native runs that the new controller reached 114–152 s after detach failed. Each recovery read ended with `PRP command session.snapshot timed out`. The transport had authenticated, and in some runs the same Runner process was verified. The Runner journal recorded no diagnostic other than "WebSocket peer closed the connection". Runs reached within 39–116 s succeeded. **What did you expect to happen?** A command timeout should say what the command waited behind, so the cause can be found from the run log. **Steps to reproduce** 1. Start a remote native run and replace the server while a provider turn is active. 2. Let the new controller reach the Runner more than about two minutes after detach. 3. Read the run log. The failure says only "timed out". ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `waitForRunnerCommand` accepts an optional `describeStall`. When a command times out, the message ends with a bounded description, for example `(runner connected; pending ahead 1 [semantic_tool.result] oldest 188s; last runner event seq 152 181s ago)`. The description covers whether a Runner is connected or how long it has been disconnected, the pending commands ahead of this one (types and count, up to 6 types, and the age of the oldest), and the last committed Runner event's sequence and age. A failing or empty description keeps the plain message. - The transport passes this description for every command it waits on. The last event's age comes from the PRP payload's `emittedAt`. - `server/src/services/native-runtime/native-session-executor.ts`: the startup `session.open` timeout classifier accepts the optional description, so a startup timeout stays `provider_session_open_timeout` and keeps its bounded same-run retry. A regression case covers each startup code with the suffix. - Tests: the exact message for the staging shape, the plain message when there is nothing to describe or the description throws, and a disconnected Runner with no events. ## Verification - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets no answer"` passes 3 of 3. These tests fail on `origin/master`, which has no `describeRunnerCommandStall` export and no description in the message. - The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The one failure (`stops a lost runnerd's provider and escaped tool job from its lineage journal`) is a process-timing test that passes alone and does not touch this code. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - Message matchers: the only matcher anchored on the end of the timeout text was the startup `session.open` classifier. It now accepts the suffix. `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "classifies a startup session.open timeout"` passes. Without the classifier change it fails, because the suffix turns the timeout into `native_session_interrupted`. ## Risks - Low. Only the text of a timeout message changes, by appending text. No control flow, persisted state or Runner behavior changes. - The description holds command types, counts and durations only. It holds no payloads, identifiers or provider text. - No wait is added before a turn start. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
8e956da1cf |
fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. A server replacement detaches the controller and leaves the Runner alive. The new server reattaches to it and continues the same turn > - On staging, about half of the in-flight runs at each upgrade lost their turn. The new controller reattached to the live Runner (same process verified), its recovery read timed out (`PRP command session.snapshot timed out`), and the failure cleanup killed the Runner > - The next same-run reattachment then found `process_not_running`. #21039 continues these runs in a fresh session, but the in-flight turn is lost and its outcome is unknown > - The cause of the kill is in the Codex driver's `recoverSession`: its cleanup closed the transport without retention. For a reattached Runner that means SIGKILL after the close deadline. #21023 already keeps the Runner for a failed open session, but not for a failed recovery > - This pull request keeps the live Runner when a recovery fails or is aborted on its way to it. Identity refusals and protocol integrity faults still stop it > - The benefit is that a slow reattachment no longer destroys a turn that was still running ## Linked Issues or Issue Description Refs #21023, #20923, #21039, #21052 **What happened?** Staging replaced its server at about 07:10Z with 17 remote native runs in flight. For each, the old server logged "Server shutdown suspended native controller ownership without cancelling provider work". The new server reattached. Example run: the Runner process (same pid and start time) was verified at 07:12:37 and the transport authenticated at 07:12:32. At 07:13:28 the recovery failed with `native_session_recovery_failed: ... PRP command session.snapshot timed out`, and a same-run resume was saved. At 07:14:12 the same-run reattachment found `process_verification` / `process_not_running`, with the Runner journal still `ready` and no exit record from the Runner itself. The run failed with `runner_remote_process_identity_unavailable` / `native_continuation_requires_reconciliation`. 90 of 93 such failures in two hours of upgrades had this sequence. **What did you expect to happen?** A recovery that times out against a live Runner is a transient failure. The same-run reattachment should find the Runner alive and adopt it again. Killing the Runner is only for a deliberate stop, an identity refusal, or an integrity fault. **Root cause** `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts` `recoverSession`: the `catch` block and the abort handler call `bootstrapCancellation(...).close()`, which calls `transport.close()` without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts` `#closeOnce`, an unresponsive adopted Runner that is not retained gets `SIGKILL` after the close deadline. The Codex driver serves remote Codex, OpenCode and ACPX runs, which matches the failures seen on staging for all three driver kinds. ## What Changed - `codex-app-server-driver-impl.ts`: `recoverSession` classifies a thrown recovery failure. A transient one (`provider_transport_failed`, a `PRP command ... timed out`, or the transport closing while waiting) now detaches this controller with the transport's restart detach (`detachControllerForRestart`). That path queues no `turn.stop` and no `runner.suspend` and sends no signal, the same as a server restart. The Runner and its in-flight turn stay for the same run's next reattachment. An abort (a Stop or the recovery deadline), any message that names a mismatch, a different session, identity, authority, integrity or incompatibility, any unclassified failure, any returned identity refusal, and a protocol integrity fault still close and stop the Runner. `openSession` is unchanged. - `codex-app-server-driver.recovery.test.ts`: the staging message (`codex_history_read_failed: ... provider_transport_failed: PRP command session.snapshot timed out`), and a plain snapshot timeout now detach and never close. An abort of a hung read still closes and stops the Runner. A thrown different-session refusal, `native_adopted_provider_identity_mismatch`, an identity refusal that also mentions a timeout, and an unclassified failure still close without retention. The existing integrity-fault cases check that the close does not retain the Runner. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of 69 (the two detach cases; master calls `close()` and stops the Runner). - Green with the fix: 69 of 69. - `npx vitest run src/drivers/codex src/native-session-runtime.test.ts src/backends src/live/runnerd-codex-transport-remote-retention.test.ts --maxWorkers=4` passed except two timing tests (Codex command resolution, background cleanup containment) that pass alone and do not touch this code. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - `detachControllerForRestart` is the existing restart path. It stops the pump, disconnects the Runner, and drains the controller journal. It queues no command and never signals the process. - Not run: a live Daytona restart. The next post-upgrade audit should show fewer `runner_remote_process_identity_unavailable` failures after a `session.snapshot timed out` resume. Instead, the run log should show "controller authority detached for restart; durable runner left alive" and then a successful same-run reattachment. ## Risks - Medium. A Runner that the recovery could not reach now stays alive, with its turn, for the same-run reattachment. This is the state a server restart already leaves. The same-run reattachment adopts it, or a Stop cleanup or saved-work recovery stops it. - If the Runner stays unresponsive across every same-run attempt, the run no longer gets the `process_not_running` proof that #21039 uses for its automatic fresh-session continuation. The run then keeps its visible Retry. The staging data shows that reattachments reaching the Runner sooner succeed, so a later attempt is expected to succeed. This is not proven against a live sandbox. - An abort of the recovery (a Stop or the recovery deadline) is unchanged: it closes and stops the Runner. `plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a Stop during an adopted resume. - No turn is replayed and no second Runner is launched. Adopting an existing Runner never launches a new one. - Previous-release story: no persisted state changes. The decision is made in the process that runs the recovery, so a deploy during a run changes nothing. - No wait is added before a normal turn start. Only the recovery failure path changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1112.0-canary.7 |
||
|
|
424e2c6488 |
fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for several minutes. A legacy run proves that its owner is alive by renewing a 60-second lease on its `heartbeat_runs` row. > - #21019 added an uncharged `server_restart` retry lane for runs that a restart or replacement ended. On the first upgrade that shipped it, no run used that lane. The new server also failed many runs as `process_lost` while the old server that owned them was still serving. > - Each of these runs then spent one failure retry. A run that was already the last automatic retry was held for manual reconciliation, so the task needed a person. > - An upgrade is routine. It must not cost a task its work or a retry, and it must not leave a task waiting for a manual Retry. > - This pull request sends a run that a replacement stopped during preparation to the `server_restart` lane. It also stops the reaper from failing a run while the server process that owns it is proven alive. > - The benefit is that a task continues by itself through an upgrade, also when the database is slow during the swap. ## Linked Issues or Issue Description Refs #21019, #21042 **What happened?** A hosted staging server was replaced by an automatic upgrade. The new server started at 06:18:46Z, became ready at 06:26:17Z, and the old server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new server failed 44 runs as `process_lost` ("Process lost -- server may have restarted; retrying once", `processLossDiagnostic.controllerReplaced: true`). The old server still owned these runs and was still serving. Every reaped run was a Runner run in `executionStage: preparing` with no process recorded. Each successor was scheduled with `scheduledRetryReason: transient_failure` and `executionRetryAccounting.failureRetries: 1`. No run on the server has `retryReason: server_restart` or `executionRetryAccounting.serverRestarts`. The run evidence also shows why the leases lapsed. The old server's lease renewals waited 19 to 50 seconds on the database (`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time (`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This pull request fixes how the reaper and the retry scheduler treat these runs. **Expected behavior** - A server never fails a run as lost while the server process that owns it is alive and serving. - A run that a replacement really ended continues once, automatically, without spending the failure budget. **Steps to reproduce** 1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`, owned by another server boot, with a lapsed run lease. 2. Run `reapOrphanedRuns()` on the new server. 3. See that the successor is `transient_failure` with `failureRetries: 1`. When the chain already spent its failure budget, see that no successor is created and the run is held for reconciliation. 4. Give the owner a live controller instance lease (the old server is still serving) and run `reapOrphanedRuns()` again: the run is failed as `process_lost` anyway. **Root cause** - `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`) left every run with the bootstrap evidence `providerWorkStarted: false` to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun` gives that lane only to an `interrupted` run (or a native preparation). The reaper writes the same evidence on a `failed`/`process_lost` run that it stops during preparation. That run fell through to the charged `transient_failure` lane. The #21019 tests used runs in `dispatching`, so they did not catch it. - `legacyExecutionNeedsReconciliation` held every run whose failure budget was spent before it checked the bootstrap evidence. A run that a replacement stopped before dispatch did no provider work, but it was held for reconciliation, and the task waited for a person. - The orphan reaper (`recoverActiveExecutions`, `server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease as proof that the owner was gone. It checked the shared owner prover only for "gone", never for "alive". During the swap the old server's renewals were slow, so its leases lapsed while it was alive. **Paperclip version or commit** Staging Core `51d77648bc` (first build with #21019). The behavior is the same on master `148d2b629c`. ## What Changed - `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)` keeps the `interrupted_preparation` exception only when that lane really takes the run. `scheduleBoundedRetryForRun` passes its own verdict. Without one, only an `interrupted` run qualifies. A `process_lost` run that a replacement stopped during preparation now continues in the `server_restart` lane. - `legacy-execution-recovery.ts`: a replaced preparation (`isReplacedPreparation`) does not need reconciliation, also when its failure budget is spent. The `server_restart` bound limits it. - `recovery.ts` (orphan reaper): - When a run's lease lapsed but the shared owner prover proves its owner process alive (for example, the owner's controller instance lease is live), the reaper leaves the run with that owner. The owner stops and settles the run itself (`controller_lease_lost`, uncharged `interrupted_preparation` lane). The reaper recovers the run once the owner is gone or stopped. It also recovers the run when the owner has not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes) after the lease lapsed. - A setup that a replacement stopped before dispatch uses the `server_restart` bound for its retry decision, whatever the adapter. - `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which returns the shared prover's verdict. `legacyControllerProvenGone` uses it, and the reaper computes the proof once per run. - `server-replacement-interruption.ts`: add `isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted: false`, a recorded controller process stamp, and a server-replacement interruption. A row saved by an older release without the stamp keeps the charged lane (the existing documented rule). - `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a reconciliation hold that an older release saved on a replaced preparation is settled once its owner is proven gone, even when the failure budget is spent. The run continues in the `server_restart` lane. - `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the `legacy_interrupted_preparation_hold` kind judges a finished run's hold by its recorded controller process (`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper swaps in a random boot id, so before this change a reaped preparation's hold had owner "none" and was never settled. - `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports exhaustion from the scheduler's own `retry_exhausted` result, so a spent `interrupted_preparation` lane blocks the task even when the synchronous budget check reads an unspent restart bound. - `doc/execution-semantics.md`: document these rules. - No schema change. Nothing changes on the path that starts a run or a turn (no new waits and no file work). ## Verification - Red on `origin/master`, green on this branch (`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs ended by a server restart or replacement"): - "continues a first attempt / a run that was already the last automatic retry that a replaced server left preparing, in the restart lane". This is the production shape: a `paperclip_runner` run in `preparing`, two servers sweep at the same time. Master: `transient_failure` with `failureRetries: 1`, and no successor for the spent chain. Now: one `server_restart` continuation, due at once, with the failure count unchanged. - "leaves a preparing/dispatching run whose lease lapsed with its owner, then recovers it once the owner stops". The owner's controller instance lease is live and two servers sweep. Master: `process_lost` at once. Now: the run stays `running` with its owner. After the owner stops, it becomes `process_lost` and a `server_restart` continuation is scheduled. - "recovers the run when its live owner has not settled it within the grace". - `interrupted-preparation-hold-settle.test.ts`: "releases an older release's hold on a preparation a server replacement stopped with its failure budget spent" for an unrevoked and a reaper-revoked lease (red without the settlement and owner-stamp changes) and "blocks the task when a replaced preparation's lane is spent even though the restart bound is not" (red without the sweep change). - Related suites: heartbeat process recovery, retry scheduling, orphaned active lease sweep, recovery deadline (bootstrap and ordinary bootstrap), legacy continuation authority, undispatched execution wait, agent status recovery, legacy workspace restore recovery, legacy execution recovery, legacy controller lease, server replacement interruption, heartbeat recovery, provider failure classification, reconciliation run read, and wake-queue policy. - `tsc --noEmit` for `server`. ## Risks - A run whose owner is alive but no longer drives it now stays `running` for up to 5 minutes after its lease lapsed, and then the reaper recovers it. The owner's own sweep still recovers a run that the owner itself lost the handle for (this boot is never deferred). - The guard trusts the shared owner prover. That prover reads "alive" only from a live or grace controller instance lease, or from a live PID in the same PID space. A server whose instance lease cannot renew for 60 seconds stops itself, so a stuck owner cannot hold runs forever. - More runs continue in the uncharged lanes. The `server_restart` bound (6 in 60 minutes) still stops a loop, and the stop is shown on the task. - Rows saved by older releases: a `process_lost` row with the bootstrap evidence and an old-boot owner now continues in the `server_restart` lane instead of being held. These runs did no provider work. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only log and API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fd5ee7e385 |
fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. When the server is replaced during an upgrade, the new server reattaches to that Runner > - On staging, upgrades every 15–40 minutes lose the in-flight Runner. The same-run reattachment then finds `process_verification` / `process_not_running`, the run fails with `runner_remote_process_identity_unavailable`, and the task is blocked > - #21039 made these runs continue once in a fresh session (the turn is recorded as having an unknown outcome and is not replayed). It covers user wakes and automatic retries (`retryOfRunId`) only > - A routine's system assignment (`issue_assigned`, requested by the system) and an external chat message wake have no `retryOfRunId`. They got `not_user_request` and stayed blocked for a person > - This pull request treats those two wakes as owed task work, so they continue the same way once the Runner is proven gone > - The benefit is that an upgrade no longer leaves these tasks waiting for a Retry click ## Linked Issues or Issue Description Refs #21039, #21023, #20923 **What happened?** Staging replaced its server at about 07:10Z. 17 remote native runs were in flight. Each new controller reached its Runner 114–152 s after detach. Recovery failed with "PRP command session.snapshot timed out". The same-run reattachment then found the Runner process gone, and each run failed with `native_continuation_requires_reconciliation`. #21039 continued 14 of the 17 automatically. The other 3 were a routine's assignment (two runs) and an external chat message. They have no `retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned `not_user_request`. Those tasks are still blocked with no retry. **What did you expect to happen?** The Runner is proven gone, so nothing can still run the turn. The run should continue once in a fresh session, the same as an automatic retry of owed work. **Steps to reproduce** 1. Start a remote native run from a routine assignment, or from an external chat message. 2. Replace the server while the provider turn is active, so that the reattachment finds `process_not_running`. 3. The run fails and the task stays blocked. No automatic Retry is requested. ## What Changed - `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the Runner is proven gone, a system wake also continues when it is a system assignment (`issue_assigned`), or when it is a comment or external chat message wake whose message is still on the task (not deleted, not written by a run). It runs under the run's responsible user, at most once per chain of runs, like an automatic retry already does. A linked person's external chat message continues under that person when the message is still on the task. Monitors and other system wakes are unchanged. - The same file and `server/src/app.ts`: the hook now passes the chat service's Retry callback (`prepareFailedChatRunRetry`, registered at app startup next to the budget-resume callback). A task in a connected conversation whose endpoint restricts external execution then continues through the chat Retry path. Before, `requestTaskRetry` refused it (`retry_refused`) and the task stayed blocked. - `server/src/services/runner-lost-mid-turn-continuation.test.ts`: production-shape cases for a system assignment, an unlinked external chat sender, and a linked person's chat message. Negative cases: a deleted message, a message written by a run, a deleted linked message, and a system assignment whose Runner was not proven gone. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd server && npx vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 16. The three new continuation cases return `{ requested: false }`. The deleted linked-message case returns `not_user_request` instead of `message_unavailable`. - Green with the fix: 16 of 16. Two more cases bind the task to a real restricted Slack endpoint and conversation. With the chat callback registered, the run continues and the Retry carries the chat action (18 of 18). Without the callback, the result is `retry_refused`, which was the behavior before for every connected conversation. - Related suites pass: `runner-lost-mid-turn-continuation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `governed-wait-runner-loss.test.ts` (116 tests). - `cd server && npx tsc --noEmit` passes. - Production evidence was read from staging run events through the read-only API: 3 of 17 losses at the 07:10Z swap took this path. After this change the next post-upgrade audit should show an automatic `task_retry` for every `native_continuation_requires_reconciliation` run whose refusal is `process_not_running`, including routine assignments and chat messages. ## Risks - Low. The change applies only after the Runner process is proven gone (`process_verification` / `process_not_running`), so no Runner can still run the turn. The turn is not replayed. The fresh session continues over the saved work, as in #21039. - The task can contain partial tool work from the lost turn. This is the same accepted risk as #21039. - Previous-release story: no persisted shape changes. The rule reads the wake's reason and comment id, which earlier releases already write. A run an older release already settled keeps its visible Retry. - This does not stop the Runner loss itself. That is a separate fix. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
4e519bd428 |
fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending
## Linked Issues or Issue Description
Refs #21035, #21002
**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).
**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.
**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.
## What Changed
- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.
## Verification
- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on
|
||
|
|
3c058c37e6 |
fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks that come from a connected chat (for example GitHub pull request reviews) show a Retry button when a run fails > - The task projection offers Retry, but the Retry route sends a connected task to the chat retry owner, which only replays the exact failed chat request > - When that exact request cannot be replayed (the attempt already saved a result, its output was published, the latest run did not start from a chat message, or the requester's access changed), the route returns 409 and tells the person to check the connection by hand > - A Retry button that is shown must work, and recovery must not ask the person to inspect anything > - This pull request makes a person's Retry continue the task from its saved work when the exact chat replay is refused, the same way a new message on the task would > - The benefit is that the projection and the route agree, and blocked connected tasks become retryable after deploy with no migration ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, five blocked tasks that came from a connected GitHub conversation showed Retry (`interruption.canRetry: true`). Each click on Retry returned 409 `chat_failed_run_retry_not_authorized`. The message was "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state." or "This attempt already recorded a result. Send a new message on the task to continue from the saved work." The latest runs were legacy sandbox setup failures, native runs that needed reconciliation, a run that started from an issue monitor instead of a chat message, and a native run that already saved its result. Root cause: `admitTaskRetry` in `server/src/services/task-retry.ts` called the chat retry owner for every connected task and passed its exact-replay refusal straight to the person. The projection cannot predict that refusal. **Expected behavior** A shown Retry works. If the exact chat request cannot be repeated, Retry continues the task from its saved work, the same way a new message on the task would. Users do not get operator instructions. **Steps to reproduce** 1. Bind a task to a restricted chat conversation. 2. Make its latest run fail so that the chat retry owner refuses the exact replay. For example, use a run whose source is not a chat message, or a native run with an accepted result. 3. Click Retry (`POST /api/issues/:id/retry`). 4. The route returns 409 `chat_failed_run_retry_not_authorized`. **Paperclip version or commit** `master` atcanary/v2026.1112.0-canary.6 nightly/v2026.1112.0-nightly.0 |