mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
52cdf76bf3e0cdf29037b63bf0f5e99aa464eab7
7595
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Fable 5.2
|
52cdf76bf3 |
fix(ephemeral): start a volume-free server over harness backups a volume-free server left unsaved (#20990)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted Paperclip server can run volume-free: its durable state is in the database and object storage, and its local disk is replaced with every deployment > - At startup, a volume-free server runs a readiness gate that refuses to start when it finds data that still needs migration off a volume > - One of those findings is a harness backup that never reached durable storage. A volume-free server that stops in the middle of a run always leaves some > - The gate then refuses every later start, the process exits, and the host restarts it into the same refusal. The workspace stays down until a person repairs it > - This pull request lets a server start over such backups when its filesystem carries the volume-free owner marker > - The benefit is that a volume-free server recovers from an interrupted run by itself ## Linked Issues or Issue Description Refs #20952 (the same crash loop for Runner controller files; this is the next check in the same gate). **What happened?** A hosted volume-free workspace stopped while runs were in progress. Every later start logged `Paperclip server failed to start` with `Volume-free server data is not eligible: native_harness_backups_require_migration`, and exited. The host restarted the container and the server exited again. The workspace did not serve any request for more than two hours. **Expected behavior** A volume-free server starts after an interrupted run. Restart recovery handles the runs whose local harness backup was lost with the old container. **Steps to reproduce** 1. Run a volume-free server (`PAPERCLIP_EPHEMERAL_SERVER=true`) with native Runner tasks in progress. 2. Stop the server process before the harness backup of an active session reaches durable storage. 3. Start the server again. It refuses to start with `native_harness_backups_require_migration`. **Paperclip version or commit** `87fe4d979355232570428deba480b47e1812a26c` (contains #20952). ## What Changed - `assertEphemeralServerData`: when the only reason is `native_harness_backups_require_migration` and the filesystem carries the volume-free owner marker (`volumeFreeControllerStateOwned`), the gate removes that blocker, logs a warning, and lets the server start. - A filesystem without the marker is refused as before. All other blockers are unchanged. - `readEphemeralServerDataEligibility` is unchanged, so the migration and enrollment callers still see the blocker. - Added `ephemeral-server-startup-run-state.test.ts`. ## Verification - Red: with the change to `ephemeral-server-contract.ts` reverted, `starts a volume-free server over harness backups a volume-free server left unsaved` fails (1 failed, 2 passed). - Green: `npx vitest run src/__tests__/ephemeral-server-startup-run-state.test.ts src/__tests__/ephemeral-server-data.test.ts` in `server/`: 2 files, 8 tests passed. - Not run locally: the full suite. CI supplies that coverage. - Not verified: a live volume-free start. The hosted workspace that showed the failure will receive this change through the normal release. ## Risks - A server on a marked volume-free filesystem now starts with unsaved harness backups. Those backups were on a disk that no longer exists, so the refusal protected nothing. The affected runs depend on restart recovery. - The marker is the only evidence that there is no volume. A filesystem that still is a volume and does not carry the marker keeps the refusal. - The gate does not repair or delete the stale lease stamps. The warning repeats at each start until restart recovery settles those runs. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5d1bb5192f |
fix(retry): carry the original wake to every re-queued run (#20989)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A run serves one wake: a comment, an answer, an assignment, or a one-shot task monitor check (`issue_monitor_due`). > - When a run fails, Paperclip often starts a successor run by itself: the bounded transient retry, the uncharged `interrupted_preparation` re-queue, workspace or connection waits, missing-comment repair, capacity retries and recovery successors. Each successor links to the failed run through `retry_of_run_id`. > - Most of these re-queues replace the wake reason with their own (for example `transient_failure_retry`). The agent then does not know which wake it must handle. > - #20970 fixed this for the board Retry only. A no-fault chaos wave showed the same loss on an automatic retry: the monitor run failed before provider work, the automatic retry succeeded, and the agent did not treat it as the monitor check. The monitor was already consumed, so the task rested `in_progress` with nothing pending. > - This pull request covers all re-queue paths at one shared point: the continuation builder that every issue dispatch uses (`buildExecutionContinuation`, called from `heartbeat/task-context.ts`). It walks the re-queue chain to the first run that was not a re-queue and records that run's wake reason as `trigger.retriedReason`. The wake prompt already shows that field (#20970). > - The benefit is that every successor of a failed run handles the same wake as the run it replaces. ## Linked Issues or Issue Description **What happened?** No-fault restart-chaos wave on master `87fe4d9793`, volume-free mode (`w-20261112T010036-s6801`, Claude, task t01, workload W2). The monitor wake started run `78e27ef9`. It failed before provider work (`native_predecessor_retirement_unproven`). The `interrupted_preparation` re-queue (attempt 1/12) started run `0cd387a0` on a fresh sandbox, and that run succeeded. The agent reported that no board comment arrived and that the monitor wake was not visible. It yielded `response_wake` again. The monitor was `triggered` with no `nextCheckAt`. The issue rested `in_progress` with `canRetry=false`, no live run, no wake and no monitor. The harness graded `R1 rested_not_done` and `R3 missing_retry`. **Expected behavior** A successor of a failed run tells the agent which wake it re-delivers. A re-queued monitor check is handled as the monitor check. **Steps to reproduce** 1. Wake a native agent with `issue_monitor_due` on a task. 2. Make the run fail before provider work, for example when the previous run's sandbox session cannot be retired. 3. Let the `interrupted_preparation` re-queue run. 4. Before this change, the successor's continuation trigger has the re-queue reason only, and its prompt does not name the monitor check. **Additional context** Follow-up to #20970. Refs #20970 Refs #20935 Refs #20863 ## What Changed - `server/src/services/execution-continuation.ts`: new `requeuedWakeReason`. - It starts at the current run's `retry_of_run_id` column. If that is empty, it uses the context `retryOfRunId`, or the source run of a board Retry. - It follows the chain within the same company and issue, at most 32 links, until it finds a run with no re-queue link. It returns that run's wake reason. - A `trigger.retriedReason` already saved on a run in the chain is reused. - `trigger.retriedReason` is set only when the original reason differs from the current reason and is not itself a Retry reason. A re-queue that keeps the original wake reason (`preserveWakeContext`) is unchanged. - This one point covers every successor that links with `retry_of_run_id`: bounded transient retries, `interrupted_preparation` (#20863), workspace and AI-connection waits, missing-comment repair (`run-lifecycle.ts`), native capacity retries, safe replacement, and the board Retry. - Tests: - `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: the assembled heartbeat service runs a monitor-due wake, a predecessor-retirement failure before provider work, and the automatic `interrupted_preparation` re-queue. It asserts the successor's saved continuation and wake prompt. - `server/src/services/execution-continuation.test.ts`: a two-link chain of automatic re-queues, and a re-queue that keeps the original reason. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/heartbeat-native-predispatch-retry.test.ts -t "delivers the monitor check"`. The successor's trigger had `sourceRunId` = the failed monitor run, but no `retriedReason`. - Green: `heartbeat-native-predispatch-retry.test.ts`, `execution-continuation.test.ts` and `heartbeat/retries.test.ts` pass (78 tests). - `npx tsc --noEmit -p server` passes. - Chaos: the existing no-fault W2 cell in volume-free mode reproduces this when the monitor run fails before provider work. I did not rerun a wave. ## Risks - Low risk. The change adds one optional trigger field to more wakes. It changes no admission, status or authority path. - The walk costs at most one indexed primary-key read per chain link, only for runs that are re-queues. - The walk follows only same-company, same-issue runs. Old saved runs work, because they already store `wakeReason` and the `retry_of_run_id` column. - The resting state "agent yielded `response_wake` after the monitor was consumed" is still possible when an agent really waits. #20935 shows it on the run projection as "Waiting for a reply". The ownership reconciler's I7 check still counts it as `no_owner_no_wake`. This PR does not change I7. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8818402e7a |
feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects requests that need a human decision. > - Teams need to know who owns each request and who may act on it. > - A shared close operation must differ from hiding a card for one person. > - This pull request adds personal routing and durable outcomes to the existing attention service. > - People can claim or transfer work and inspect what another person did for them. > - Domain permissions still decide who may resolve each request. ## Linked Issues or Issue Description **Subsystem affected** Server, shared contracts, and database. This is the routing foundation for a stacked inbox UI change. **Problem or motivation** A company-wide attention feed does not give teams a clear personal queue. Archive, withdrawal, transfer, and decisions by another person need distinct scopes and records. **Proposed solution** Resolve a request audience from its addressee, task participants, and configured owners. Provide personal, unassigned, company, all-assignee, and archived views. Add claim, transfer, personal archive and snooze, shared withdrawal, receipts, and personal history. **Alternatives considered** A single shared inbox leaves ownership and personal hiding ambiguous. The service keeps one canonical request and projects its audience instead of copying requests for each person. **Roadmap alignment** This extends the existing Multiple Human Users and Agent Reviews and Approvals capabilities in ROADMAP.md. **Additional context** Refs #19363 and #19520 for the current inbox navigation and confirmation replacement rules. Refs #20268 for related actionable-request work. That PR filters actionability; this stack adds audience routing and personal outcomes. This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this branch. ## What Changed - Add nullable company and project decision owners and a company budget owner. - Add audience views and personal badge counts to the canonical attention projections. - Add claim, transfer, withdrawal, personal archive, snooze, and restore operations with permission checks and activity records. Lock addressed transfers to the current addressee or routing administrators, and validate the recipient against canonical resolution policies. - Add receipts and scoped history for work handled by another person. - Keep closed-task requests and replaced confirmations out of current cards. - Add an idempotent migration and a concurrent receipt-history index. - Update typed fixtures so the existing UI remains compatible with the new contracts. ## Verification - Server layered typechecking passes, including all three test-tree partitions. - Seven routing and interaction-resolution suites pass all 249 tests against a dedicated external PostgreSQL test database on `4caecc21d8`. Six new regressions failed before the repair; they cover takeover followed by approval, legitimate delegation, excluded creators/review requesters, and unauthorized claims. - Prior focused verification covers private approval history and receipts, transfer visibility, audience escalation, withdrawal governance, hidden-card counts, per-recipient confirmation replacement, and SDK source contracts. - A synthetic three-person company passed 37 routing and lifecycle acceptance checks. The UI stack adds browser acceptance for the shared controls. - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331) passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with no open findings. The fresh security scan passes after explicitly rechecking the addressee-transfer repair. - `pnpm verify` reports exit 2 because a CI-global script changed; full CI supplies that broader coverage. Embedded PostgreSQL could not start on this Mac because shared-memory slots were exhausted, so local database checks use a separate external test database. Linux CI covers the embedded setup. ## Risks - Personal badge counts change to the viewer's assigned requests. Accessible feed totals and task membership keep their existing scope. - The migration adds nullable owner fields and a dismissal fingerprint. Its index runs outside a transaction. The migration is classified and rerunnable. - Routing changes the addressee used by resolution authorization. The current addressee or a routing administrator must authorize an addressed transfer; the recipient must satisfy writer access, canonical resolver exclusions, and task review policy under row locks. - The shared validator export barrel crosses 1,000 lines because it exports the new attention schemas. Schema implementation stays in the dedicated attention validator module. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (OpenAPI contracts and docs/api/attention.md) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
TonioandPaperclip
|
9dfdeaad86 |
Refine agent workspaces and organization charts (#20973)
Refine agent details, identity editing, responsive controls, and organization charts with three card tiers, clearer save feedback, and stable filtering. Guard instruction-file renames and root configuration updates in one transaction through the agent lifecycle integration, preserving conflict checks, rollback, and configuration history. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
45170b0cac | Reuse Runner plugin capability checks (#20987) | ||
|
DottaandClaude Opus 5.5
|
42fd790e80 |
fix(evals): grade approval delivery on the saved answer, not on which queued wake is listed (#20981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades the saved product state > - The `steering.approval-delivery` case checks that a person's answer to an agent's question stays separate from editable queued messages and reaches the agent once > - The task composer lists one pending wake at a time. It shows editable ordinary messages before a saved card response, by design since #19331 > - When the chat lead relays the person's answer as its own task comment, the composer lists that relay. The saved answer waits on its own wake and is not listed > - The oracle required the answer and the relay in one listing. The product never produces that shape, so the case timed out every time the lead relayed > - This pull request grades the case on the saved answer and on its delivery, and accepts either valid listing > - The benefit is that the case measures the promise in its expected outcome and still fails on a lost, replayed, or editable answer ## Linked Issues or Issue Description **What happened?** The paid chat eval `steering.approval-delivery` (OpenCode, DeepSeek flash) did the answer, edit, reorder, and discard steps. It then timed out while it waited for the answer to show in the queue. **Steps to reproduce** Run the case. Let the chat lead relay the answer as its own task comment. After the person's two messages are discarded, the queue lists only the relay. The answer is saved as `answered` and keeps its own pending wake. **Expected behavior** The case passes when the answer stays saved and unchanged, cannot be edited through the queue, and reaches exactly one later agent turn. Refs #19331, #20899 ## What Changed - `runtime-approval-queue.ts`: add `approvalResponseListing`. After the discards, accept the answer listed alone and locked, or only the chat lead's relay listed. Reject any person message or an editable answer. - Keep the edit attempt through the queue route for both listings. Require it to be refused and the saved answer to stay unchanged. Check the locked card in the UI when the answer is listed. - Add `classifyApprovalSuccessors`. Grade delivery on saved run state: exactly one later worker turn bound to the answer, with the question ID and Acorn in its native input. Allow at most one other turn that carries only the lead's relay. Reject any other turn. - Accept the Acorn document from the bound turn or the relay turn. Still require the earlier work to be kept, all runs succeeded, the queue empty, and the task done. - Oracle `approval-delivery.v6`. Run bound 5 with a minimum of 4, to allow the one relay turn. - Calibration tests: the retained relay shape passes, the original listed shape passes, and these fail: no bound turn, two bound turns, a person message left behind the relay, an unexpected worker turn, two relay turns, a recorded listing that disagrees with the queue, and a listed answer whose controls were not seen as locked. No YAML changed. The expected outcome already promises retention and once-only delivery, not a queue listing. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` (46 passed) - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` (clean) - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check` (no change) - One paid single-case run of this branch. The result is added below in a comment. ## Risks - Low risk. The change is in eval code only. No product behavior changes. - The grader now allows one extra worker turn for the lead's relay. A turn that carries any other input still fails the case. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4b24352712 |
test(restart-chaos): stall the plugin during reattachment (CR4) and at launch (CR5) (#20966)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) injects faults
into a real Paperclip server and Daytona sandboxes, and grades each
task.
> - Class R cell CR3 stalls the Daytona plugin worker for 45 seconds in
the middle of a tool call. It found the failure that #20940 fixed.
> - #20945 fixed the same stall during reattachment and at launch, but
only unit tests cover those paths.
> - This pull request adds two cells: CR4 stalls the plugin right after
a K1 restart, while the run reattaches, and CR5 stalls it while the run
launches.
> - The benefit is that a wave now exercises the recovery and launch
paths, not only unit tests.
## Linked Issues or Issue Description
Refs #20909, #20940, #20945
**What happened?**
CR3 stalls the plugin only in the middle of a tool call. The
reattachment reads (saved state, identity marker, process check) and the
launch reads (runnerd launch call, identity read) had no chaos coverage
for a plugin stall.
**Expected behavior**
The harness can stall the plugin during reattachment and during launch,
and it grades the task as it grades CR3.
**Steps to reproduce**
1. Run a wave with `--faults CR4,CR5`. Before this change, the harness
rejects these fault kinds.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
5cd5adfccc |
fix(native): link a person-owned agent blocker to the agent's open question whatever owner kind the model reports (#20969)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #20931 lets a person's answer continue a task that its agent blocked
while it waited for that answer
> - That link is recorded only when the blocker's unblock descriptor is
bound to the board
> - Codex reports such a wait as `paperclip_block` with `owner: {"kind":
"user", "name": "Board"}` and scope `task_wide`. The evidence classifier
treats only `kind: "board"` (not a PRP owner kind) as board-owned, so
the arbiter binds this person-owned blocker to the reporting agent, and
no link is recorded
> - So every Codex W2Q task still loses the answer
(`issue_not_in_progress`) and needs Retry
> - This pull request classifies a `user` owner as a person
(board-bound) and records the link from the reported owner kind, not
from the spelling of the owner
> - The benefit is that the person's answer reaches the agent whatever
owner shape the model uses
## Linked Issues or Issue Description
Refs #20931
**What happened?**
Restart-chaos wave w-20261111T230717-s5201 on master
|
||
|
DottaandClaude Opus 5.5
|
dfd2a6efca |
fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs on a remote sandbox keep a warm Runner between turns. A follow-up run takes over its settled predecessor's provider session and Runner. > - When a deploy stops the server, a follow-up that is still preparing fails with `local_runtime_api_listener_unavailable`. The executor saves this failure as a same-run bootstrap retry (#20786). The saved failure proves that the run did no provider work of its own. > - After the restart, restart recovery saw a checkpoint and a provider session, and chose `reattach_remote_runner`. But the checkpoint and the Runner identity belong to the predecessor. The Runner identity check then failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id`. The run ended terminal and the task needed Retry. > - This pull request recognizes that durable shape in restart recovery and restarts the run as a fresh bootstrap on its warm session. It does not try to reattach to a Runner that belongs to a different run. > - The pull request also stops the queue from claiming a queued run after shutdown closed run admission during the queue scan. > - The benefit is that a deploy during a follow-up's preparation no longer ends the run terminal with a Retry. ## Linked Issues or Issue Description Refs #20786, #20928, #20852, #20904 **What happened?** In three Codex runs across two Daytona restart-chaos waves (warm sandboxes, CR1 and K15 with overlap), a warm follow-up run (`automation`, `retryOfRunId` = its predecessor) started 0–2 s before its server got SIGTERM. Its bootstrap failed with `local_runtime_api_listener_unavailable` and saved a `bootstrap_retry` with `providerSessionEstablished: false`, `providerEventsExist: false`. After the restart, `native.recovery.transition` chose `reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw `identity_run_id`, because the warm Runner's identity still named the predecessor. The run ended with `native_continuation_requires_reconciliation`. In every sandbox capture the Runner was cleanly suspended, with no stderr. **Expected behavior** The run did no provider work and never wrote its own Runner identity. The next server must restart it without a Retry. **Steps to reproduce** 1. Finish a remote warm native run, so that its provider session and Runner identity are saved. 2. Start a follow-up run. Admission carries the predecessor's provider session and Runner process identity into the follow-up. 3. Send SIGTERM while the follow-up prepares, so that it fails with `local_runtime_api_listener_unavailable`. 4. Restart the server. Restart recovery chooses `reattach_remote_runner`, and the run fails with `identity_run_id`. ## What Changed - `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the `remoteFollowUpNeverAttached` proof. It is true only when all of these are true: - the run is a failed, settled `bootstrap_retry` whose original failure is `local_runtime_api_listener_unavailable`, with `providerSessionEstablished: false`, `providerEventsExist: false`, no successor and no lease owner; - the run is on a prepared remote sandbox workspace with the current preparation fencing version, it is not a plugin task, and it has no Runner controller checkpoint and no staged attachment of its own; - its session checkpoint is only the carried admission checkpoint (no cursor, turn, result, terminal or pending request), and a settled earlier run of the same session saved that exact provider session; - its Runner process identity is absent or is exactly the identity of a settled predecessor; - it has no provider event, no Runner-originated event, and no `native.process_identity_recorded` event of its own. With that proof, the run gets a `bootstrap_incomplete` claim with the reason `remote_follow_up_never_attached`, and it passes the same current-admission check as the other settled bootstrap retries. - `verifyRemoteRunnerReattachment` is unchanged. It still refuses a Runner that names a different run. - `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run admission again before each claim. A run that a scan reaches after shutdown closed admission stays queued for the next server. - Tests: - `native-runner-restart-recovery.integration.test.ts`: eight cases for the staging shape. Three restart (inherited Runner identity; inherited identity plus a Runner request; a Runner request with no identity). Five stay held (own recorded Runner identity; a Runner-originated event; a carried active turn; a provider session that no settled run saved; a saved failure that established a provider session). - `queue.test.ts`: a queued run stays queued and is not executed when admission closes during the scan. ### How this relates to #20852 and #20904 - #20852 hands a preparing run to the next controller and persists launch intents, so that preparation itself survives a deploy. It does not change restart recovery classification for a settled `bootstrap_retry`, and it does not touch the run-admission gate. This PR covers runs that already failed and saved a `bootstrap_retry` (including state that older releases saved), so the two are complementary. - #20904 changes startup readiness for native-controller preservation obligations. It does not touch either code path in this PR. ### Finding about the "draining server" admission In both waves the follow-up was claimed before the process observed shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and started at 21:54:04.854Z; the server logged "shutdown signal received" at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load average 10). In the K15 overlap (seed 9513) the runs started at 23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at 23:31:13Z. During the overlap the old server is fully live and has no signal that a newer deploy serves traffic. So the existing admission gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the gate was the check-then-claim window inside `startNextQueuedRunForAgent`, which this PR closes. Holding work on a superseded but not yet signaled server needs a new "superseded deploy" signal; this PR does not add one. ## Verification - Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs packages/db/src/test-postgres-command.ts -- node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "closed the runtime listener before it attached"`: the three restart cases failed with `"kind": "reattach_remote_runner"` (the staging disposition); the five held cases passed. - Red on master: `queue.test.ts -t "closes admission during the queue scan"` failed (`expected [ { …(81) } ] to deeply equal []`). - Green with the fix: both filters pass. - `native-runner-restart-recovery`, `native-runner-restart-preparation`, `native-startup-retry` and `native-controller-recovery` integration files: 267 passed, 14 skipped. - `queue.test.ts`: 89 passed. - `cd server && pnpm typecheck`: passed. - Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1) and 9513 (K15 overlap). ## Risks - The new proof applies only to the closed-listener `bootstrap_retry` that #20786 writes, on a remote sandbox, with a carried session that a settled predecessor saved. Every other shape keeps its earlier classification. - A `bootstrap_incomplete` claim sends the run through the existing bootstrap path, which verifies the prior owner of the session root before it starts. The s9205 follow-ups already use this path with an inherited Runner identity. - Upgrade safety: no persisted shape changes. The proof reads only fields that the current and recent releases write. A row saved without these facts keeps the earlier verdict. - No wait or file work is added before `turn.start`. The new queries run only in restart recovery. - The queue change only stops claiming after admission closed. Runs stay queued for the next server, the same as the existing entry check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.1 |
||
|
DottaandClaude Opus 5.5
|
66344b76ae |
fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. A warm Daytona sandbox is reused by later runs. > - When a Runner exits before `session.open`, the run does a bounded same-run bootstrap retry. Durable evidence proves that no provider session and no provider event exist. > - The retry fails when the Runner wrote `runner-state.json` but exited before it wrote its provider state. Host preparation then calls the sandbox state "unavailable" and refuses with `runner_harness_state_mismatch: resumed_unavailable`. The task needs Retry. > - This pull request reads the Runner state on its own when the full check fails, but only for a proven bootstrap restart of the same run. A non-suspended Runner state of this run is then the run's own interrupted bootstrap, and the existing restart path moves it aside and claims a new session root. > - The benefit is that a Runner that exits before `session.open` on a reused sandbox recovers without a person. Saved state, a verified backup and every other mismatch keep their refusal. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the same signature after each restart kind. The first attempt of each run reused the warm sandbox. Its Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use` and exited with `lifecycle=connecting`. Recovery logged "Restarting an incomplete native runner bootstrap on the same heartbeat run". The second attempt reused the environment and failed with `runner_harness_state_mismatch: resumed_unavailable`. The earlier wave `w-20261111T212429-s6402` had one run with the same error code. **Root cause** `inspectRemoteHarnessState` in `server/src/services/native-runtime/native-session-executor.ts` checks the persistence directories, `runner-state.json` and the provider state file in one shell test. A Runner that exits at bind time wrote `runner-state.json` but no provider state, so the one test fails and the result is `incompleteReason: "unavailable"` before the Runner state is read. The restart path (`restartableBootstrapState`) accepts only `not_suspended` and `provider_unopened`, so it is skipped. `claimUntouchedRemoteSession` then cannot create the session root that the first attempt created, and the resumed branch throws `resumed_unavailable`. On a created lease the same state fell through without a restart. **Expected behavior** A proven, interrupted bootstrap of the same run restarts on the same sandbox. The run finishes without Retry. **Steps to reproduce** Run the s6402 cell above, or add `--runner-bind-fault` (this pull request) to force the Runner exit on every run's first launch. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the full harness check fails during host preparation, `inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It returns `not_suspended` for this run only when all of these are true: this run has a proven bootstrap restart (`bootstrap_incomplete` recovery plus the stop proof), no failover backup exists, no archived rotation snapshot is in use, and the Runner state names this run, this Runner instance and this session with a lifecycle other than `suspended`. Otherwise the result stays `unavailable`, as before. The existing restart path then moves the session root aside (it is kept, not deleted) and claims a new one. - Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives every server of the wave `PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test` only, the remote Runner launcher starts a short-lived listener on the Runner's listen port before a Runner's first launch on that server, so the Runner exits with a real `runner_ingress_bind_conflict`. Documented in `tests/restart-chaos/README.md`. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"` failed 2 of the new cells. Resumed lease: `Error: runner_harness_state_mismatch: resumed_unavailable` (the staging error). Created lease: no restart happened (`expected [] to have a length of 1 but got +0`). - Green: the same cells pass. The negative cells keep `resumed_unavailable`: no restart proof, a Runner state of another run, a `suspended` Runner state with missing provider state, and a failover backup present. - `native-session-executor.test.ts`: 1406 of 1406 pass. A new test proves the bind fault holds the listen port (a second bind gets `EADDRINUSE`) and fires once per Runner instance. - `tests/restart-chaos`: `runner-bind-fault.test.ts`, `question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and harness `tsc --noEmit` pass. - Chaos wave: see the comment below for the before/after counts. ## Risks - The new read runs only for a proven bootstrap restart of the same run with a failed full check, so the normal path has no extra command and no wait before `turn.start`. - A partly saved session (`suspended` Runner state without provider state), a Runner state of another run and any failover backup keep today's refusal. The moved-aside root is kept for inspection. - A Runner that started under an older server and is restarted by this version takes the same path, because it reads only files that older Runners already write. - The bind fault is test instrumentation. It is inert unless `NODE_ENV=test` and the variable are both set. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20928 (restart a Runner that exited before session.open), #20874, #20863, #20953. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
afcf34c9da |
fix(evals): form repair-cost-recording's saved-usage debt and grade it by saved state (#20895)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature evals check that the chat agent helps a person with real product state > - The `budgets-costs.repair-cost-recording` case checks that the agent sends the person to the saved-cost retry control and then reports the true accounting result > - The grader checked the agent's replies with keyword regular expressions, and the scripted person gave the agent the harness's own accounting readback > - Keyword checks reject valid wording and accept wrong answers, so the case did not measure the promise > - The setup also never formed: the prerequisite run always settled, so no "Ready to record" saved usage existed > - This pull request forms that state with a temporary trigger on the case-owned database (no product change), and replaces the two keyword checks with checks of saved state, the agent's own product calls, and rendered links > - The benefit is a case that can run, and a grader that does not depend on wording and that still fails the relevant broken behavior ## Linked Issues or Issue Description **What happened** The case `budgets-costs.repair-cost-recording` graded the chat replies with keyword regular expressions (`honest-post-retry-accounting-result` and `guides-accounting-without-rerun`). The scripted follow-up message also told the agent the harness's accounting result. Separately, the setup never formed in paid runs: the prerequisite native run settles as an unpriced cost event (`missing_price`), not as a `retryable` saved-usage row. Accounting retries every 5 seconds, so a retryable row only exists while accounting keeps failing. **Expected behavior** The grader accepts any wording. It requires that the guidance before the retry links the Costs page, that the post-retry run reads saved cost data before it answers, and that the agent does not write cost, budget or accounting records or start work. **Steps to reproduce** 1. Read `gradeAccounting` in `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on master. 2. Reword a valid reply (for example "Head to Costs and press the button there; the work stays as it is."). 3. See that the keyword checks reject it. Refs #20600 ## What Changed - New `tests/runner-e2e/chat-accounting-fault.ts`, following `chat-archive-storage-fault.ts`: a temporary trigger on the case-owned database refuses cost-event writes for the one prerequisite task. The product's own accounting transaction rolls back, so the finished run keeps its saved usage pending and the product's automatic retries keep failing. Refused writes are counted in a rollback-proof sequence. It is allowed only when this case is the single selected execution, and it never writes product rows. - Setup: the case creates the prerequisite task unassigned, arms the trigger, assigns the task, and waits until Costs shows the genuine `retryable` row. The trigger is closed on a setup failure and at cleanup. - Act: the trigger is dropped right before the person clicks "Retry cost recording". Automatic recovery can record it first; the saved accounting read afterwards classifies either result (an unpriced event still shows in data quality, so it counts as unresolved). - `chat-archive-storage-fault.ts`: export its owned-database resolver for reuse (no behavior change). - `identity-accounting.ts`: remove the reply keyword checks for the repair case. - Add `initial-guidance-links-saved-cost-retry`: a reply from before the retry has a rendered link to the Costs page (`/<prefix>/activity/costs` or the `/<prefix>/costs` redirect). - Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run made a successful read of a company cost report, or of the cost summary of the prerequisite task (bound by its redacted input digest). - Add `no-agent-accounting-fabrication-or-rerun`: no successful chat-agent call writes cost, finance, budget or accounting records or starts run or task execution. A denied attempt is not a write. - `observe` collects the chat agent's redacted `call_api` uses and results (`leadApiCalls`). - The scripted follow-up now reports only the toast that the page showed. That toast says "Costs recorded." even when the usage still has no price. - Own `oracleVersion` (`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and source files in code and in `chat-catalog-descriptors.json`. - Calibration tests: reworded valid replies pass; a missing or late Costs link, a run link only, a foreign link, no fresh read, a read in the wrong run, a denied read, an unrelated read, another task's cost summary, and agent cost or execution writes fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-accounting-fault.test.ts tests/runner-e2e/chat-archive-storage-fault.test.ts tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The embedded-Postgres test drives the product's `accountRunCost`, `reconcileRunCosts`, data-quality report and board retry against the trigger: only the faulted task's run stays `retryable`, other debt in the company settles, and the retry records it once the trigger is dropped. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`: unchanged. - Paid run still needed: `chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`. ## Risks - Low risk. The change affects only this eval case. - The new read check needs the chat agent to have company cost-read access. Agents cannot read `costs/data-quality` (board only), so the agent uses the cost reports or the task cost summary. - The trigger runs DDL on the case-owned database only; the same isolation checks as the archive storage fault apply, and it is dropped before the retry and removed at cleanup. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a75d7df9c7 |
fix(evals): gate steering approval-delivery and steer-active on real product state (#20899)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat feature catalog runs real native agents to prove the chat and task-steering promises. > - Two steering cases failed in setup before they could measure the product. > - `steering.approval-delivery` waited for a worker run to stay running while its question was pending. The product parks a native turn as soon as a durable question is saved, so that state never occurs. > - `steering.steer-active` waited for the queue to say steering is "available". The OpenCode harness reports that it cannot steer, so the queue says "unsupported". > - This pull request gates each case on the real product state and grades the behavior that state promises. > - The benefit is that both cases now measure their user promise on the evaluated profile, without faked runs or faked steering. ## Linked Issues or Issue Description **What happened** Both cases failed during setup on `runner-opencode`. `steering.approval-delivery` stopped with "The actual question fixture did not reach its running pending-input boundary": the worker's `request_human_input` call parked its turn (run succeeded, `reportedWorkDisposition: yielded`, continuation `response_wake` for the question). `steering.steer-active` timed out waiting for `steeringDisposition: "available"`; the OpenCode driver declares `steering: false`, so the queue reports `"unsupported"` and the page shows Interrupt instead of Steer. **Expected behavior** `steering.approval-delivery` should treat the saved yield and the pending question as the pending-input state, queue ordinary comments and the bound answer behind a real active worker turn, and prove one continuation receives the answer. `steering.steer-active` should grade same-turn steering when the harness can steer, and grade the honest fallback when it cannot: the message stays visibly queued, nothing is recorded as steered, and one normal turn consumes it. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.steering.approval-delivery` on master. 2. See the setup stop after the worker run succeeds with a yielded result. 3. Dispatch `chat-features.runner-opencode.local.steering.steer-active` on master. 4. See setup time out while `GET /api/issues/:id/queued-comments` reports `steeringDisposition: "unsupported"`. ## What Changed - `runtime-approval-queue.ts`: gate the question on the parked turn (`pendingInputYield`: succeeded, yielded, `response_wake` keyed to the question). An ordinary human message then wakes a real second Morgan turn that waits at the task gate. Ordinary comments and the bound answer queue behind that turn. The run count goes from 3 to 4. - `runtime-approval-queue.ts`: remove the reply keyword checks (`acornNamingDirection` and the "I have answered" regex). Grade exactly one saved answer path instead: the human response control, or an attributed canonical response. - `runtime-active-steering.ts`: accept either definitive capability. Unsupported branch: no enabled Steer control, a visible Interrupt control, a steer attempt with the person's session refused with `409 steering_unsupported`, no `issue.queued_comment_steered` activity, the exact message still queued, and one continuation that consumes it and registers a rendered blue `website.html`. - `runtime-active-steering.ts`: remove the reply keyword checks and the `/blue/` match on the delivery document. The rendered page proves the color. - Update calibration tests in both directions, the workspace setup test, and the descriptors (`oracleVersion` bumps, `expectedRunCount` 4 for approval-delivery). The scenario YAML is unchanged. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-active-steering.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-workspace.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 87 passed. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - The live proof is a paid run of `chat-features.runner-opencode.local.steering.approval-delivery` and `chat-features.runner-opencode.local.steering.steer-active`. ## Risks - approval-delivery now depends on a human message waking the worker while its question is pending. The product does this (`supersedeOnUserComment: false` keeps the card open, and comments wake the assignee unless the task is done or cancelled). If the worker does not run the gate on that turn, setup fails with a named label. - steer-active on OpenCode now measures the honest unsupported fallback, not same-turn delivery. A steering-capable profile still runs the same-turn branch. - Low risk to the product: only eval code changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
38501b1170 |
Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents open input images with a file reader. > - Plugin Runner tasks use an authenticated Runner connection. > - Runner needs a way to fetch an authorized file from the tenant app. > - The tenant grants access to the exact attachment for the active run. > - Runner downloads and verifies it before it returns a local path. ## Linked Issues or Issue Description Refs #19740. Builds on the attachment authorization and command transport added in #20939. ## What Changed - Add the PRP v3 `runner.attachment.download` command for native workspace providers. - Issue short-lived grants for exact authorized wake and historical attachments. Recheck the active run, source, size, and SHA-256 at the tenant endpoint. - Download current-wake inputs before provider dispatch. Download historical files when the authorized chat reader asks for them. Preserve SSH and sandbox command staging as a fallback. - Run HTTPS transfers off the Runner control loop. Report completion with one PRP event per transfer. Drop pending results on cancellation and clear files at turn settlement, authority rotation, and restart. - Grant restricted Codex sessions read-only access to the attachment directory. ## Verification - 69 targeted server attachment tests passed. They cover wake and historical downloads, authorization failures, and command fallback. - 22 focused historical attachment tests passed, including grant redemption under a held database lock. - Six focused Rust attachment tests passed, including a source-pinned Codex sandbox check. The sandbox could read the staged file but could not modify it or read adjacent Runner state. - The real Runner transport test confirmed one durable command and one completion event for a transfer. Restart-continuation tests confirmed fresh attachment paths appear before dispatch. - Server and Runner TypeScript checks, Rust formatting, and PRP generated schema checks passed. - Required GitHub CI passed on `7a33752bd7`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43122967837) - Greptile scored the same revision 5/5 with no unresolved review threads. [Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134) ## Risks - Saved provider and Runner identities keep their existing formats. A previous Runner falls back to command staging where the target supports it. A restart continuation refreshes temporary attachment paths before dispatch. - Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS access to the configured tenant origin. - Live agent-computer qualification remains a release check. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell tools, and local test execution. Exact variant and context window are unavailable in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #123` / `Closes #123` / `Refs #123` or (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
ba5ba54439 |
fix(evals): name unmet completion-gate readiness fields and grade the gate read, not reply words (#20887)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `tasks.completion-with-gate` checks that the CEO does not force Done while an independent human review is still required > - Its setup waits for a real Morgan implementation that awaits the configured review stage > - The setup timed out and did not say which state field was wrong > - The retained evidence shows a product defect: the native completion path never started the configured review stage (fixed in #20882) > - This pull request makes the readiness timeout name each unmet field, accepts valid product continuation runs, and replaces reply keyword grading with a structural check > - The benefit is that the eval measures the gate, and its setup failures are self-explaining ## Linked Issues or Issue Description **What happened** The eval failed with "Timed out waiting for Real prior Morgan implementation awaiting independent review". Morgan uploaded `website.html` and reported needs_review. The task became in_review through a generic review card, but `executionState` stayed null, so the poll (pending stage for Riley) never matched. The run finished at 229 s of the 240 s setup window. The grader also matched reply words with regular expressions. **Expected behavior** Setup reaches the configured review stage through supported product paths. A timeout names each unmet field. The grader proves the outcome from saved state and run receipts, not reply wording. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.tasks.completion-with-gate` on master. 2. Read `server.log` and `snapshots/chat-final-run-ledger.json`: `authoritativeDecision: in_review`, `finalizationReasonCode: actionable_attention_pending`, no executionState. ## What Changed - `internal-task-completion.ts`: `unmetPriorDeliveryReview` lists each unmet readiness field. The poll uses it for `accept` and `timeoutDetail`. - Setup and grader accept one or more succeeded Morgan runs (a product continuation is valid). Any failed, cancelled or timed-out worker run is an unmet readiness condition, so setup stops before the paid CEO turn. The delivery must come from one of these runs. - The `truthful-outstanding-review` reply regex check is replaced by `checked-actual-completion-gate`: the CEO's chat run read the task (`GET /api/issues/{id}` or `get_task`, matched by record-bound receipt) and replied. The saved pending stage, unchanged policy and absence of a forced Done still prove the outcome. - Run bound 2 → 3, setup window 240 s → 330 s, attempt 540 s → 660 s. `oracleVersion` is `paperclip.chat-catalog.completion-with-gate.v2`. The descriptor is updated. - Cherry-picks both #20882 commits so the paid run exercises the product fix. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `internal.test.ts`, `internal-comment-reference.test.ts`, `internal-edit-ownership.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts` (184 tests) and 12 other suites that reference the case (133 tests). - Calibration: two reworded valid replies pass. A missing gate read, an empty reply, a forced Done and a failed worker run fail. The retained product state (in_review, no executionState) produces named unmet fields. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no new errors in the changed test files. The local worktree has unbuilt packages, so CI supplies the full typecheck. ## Risks - The reply's meaning is no longer machine-checked. A person must read the reply in the paid run. - The larger run bound and windows accept slower valid runs. Readiness still fails on any failed worker run. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
87fe4d9793 |
fix(workspaces): retry a workspace_finalize record write that loses a deadlock (#20971)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a run ends, Paperclip records a `workspace_finalize` workspace operation. Dependent work reads that record to know if the workspace is safe to use. > - The record insert has foreign keys to `execution_workspaces`, `companies`, `heartbeat_runs` and `issues`. Postgres takes a key-share lock on each parent row, in trigger order: workspace, company, run, issue. > - The native finalizer and other writers lock the issue row and then the run row (the order that #20819 documents). When such a transaction holds the issue and then asks for the run, while the finalize insert holds the run and waits for the issue, Postgres detects a deadlock and aborts one side. > - In restart-chaos wave `w-20261111T205259-s6301` (task t01, run `e7129ec7`) the insert lost. The log says "failed to record workspace_finalize=failed operation; dependents may remain gated". The failed-finalize record was lost. > - The insert's lock order comes from the foreign keys, so the code cannot reorder it without a new explicit lock order that other writers (for example the remote-workspace release path, workspace then run) could invert again. A deadlock victim is safe to run again, because its transaction rolled back completely. > - This pull request retries the finalize record write on lock timeout, deadlock and serialization failure (55P03, 40P01, 40001), with the same bounded policy that #20850 added for the provider-pack admission write. > - The benefit is that a short lock conflict can no longer delete the record that releases dependent work. ## Linked Issues or Issue Description **What happened?** Run `e7129ec7` failed its native session recovery (`runner_harness_state_mismatch: backup_without_reusable_lease`). The failure path called `recordWorkspaceFinalize("failed")`. The insert into `workspace_operations` failed with `PostgresError: deadlock detected` (40P01). The server logged that dependents may remain gated. The run then settled as failed with `native_session_interrupted`; the next `finalizeNativeRun` call logged `native_finalization_missing` because that run had no proposed result. **Expected behavior** A transient deadlock does not lose the finalize record. The write runs again and the record is saved. **Steps to reproduce** 1. Transaction A: lock the issue row `FOR UPDATE`. 2. Start a `workspace_finalize` record for a run on that issue. Its insert takes key-share on the workspace, company and run rows, then waits for the issue. 3. Transaction A: lock the run row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the insert is the victim and the record is lost. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. It gives transaction A a longer `deadlock_timeout`, so the insert is always the victim, as on the chaos run. Refs #20850 Refs #20819 ## What Changed - New `server/src/services/transient-lock-contention.ts`: `postgresErrorCode`, `isTransientLockContention` and `retryTransientLockContention` (5 attempts, 100 ms × attempt backoff, only 55P03, 40P01 and 40001). - `workspaceOperationService.createRecorder().recordOperation` wraps its `workspace_finalize` state writes (the running-row insert and the terminal update) in that retry. Each attempt is a new transaction, or a new savepoint when the service runs inside a caller's transaction. - `native-provider-pack-admission.ts` uses the shared helper. Its behavior and its exported `nativeProviderPackAdmissionFailureCode` name do not change. - New test in `server/src/__tests__/workspace-operations-reconciliation.test.ts`: a real two-transaction deadlock between an issue-then-run locker and the finalize insert. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/workspace-operations-reconciliation.test.ts -t "loses a deadlock"` failed with `deadlock detected`, code `40P01`, on the same `insert into "workspace_operations"` query as the chaos log. - Green: the same test passes; I ran it 3 times. The full file and the provider-pack admission tests pass (49 tests). `native-result-commit-work.test.ts` passes. - `npx tsc --noEmit -p server` passes. - I stopped `pnpm verify` after it selected 2533 test files; required CI supplies broad coverage. - Chaos: no new cell. The existing W2 × K1 cell (Codex, Daytona warm) reached this path when the restart fault broke native session recovery. I did not rerun a wave; both wave slots were busy. ## Risks - Low risk. Only the `workspace_finalize` state writes retry, and only on the three contention codes. Other errors throw as before. - A deadlock victim's transaction is fully rolled back, so the retried insert (same operation id) and the retried update are idempotent. - The worst added delay is about 1 s of backoff across 5 attempts, on the failure path only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1880526603 |
fix(runner): never start an OpenCode session without its Paperclip tools (#20964)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode agents use Paperclip tools through a local MCP bridge in the runner > - OpenCode 1.18 connects each MCP server one time per process. It does not retry > - One failed `initialize` or `tools/list` at startup marks the Paperclip server "failed" for the full session > - The agent then works with only built-in tools. In one chat it told the person "Switched the working design to blue", but no saved state supported this claim > - This pull request makes the runner confirm the Paperclip tools while the session loads, reconnect them, restart OpenCode if necessary, and fail the session by name if the tools stay unavailable > - The benefit is that an OpenCode turn never starts without its Paperclip tools, so the agent cannot silently claim actions that it could not do ## Linked Issues or Issue Description Refs #20886, #20921. **What happened** In chat eval `steering.steer-active` (private paid run 43086418488, on a commit with #20886 and without #20921), two of three OpenCode sessions started without Paperclip tools. One was the CEO chat (fresh session). One was the worker task after its saved session resumed in a new OpenCode process. The CEO agent ran `opencode mcp list` and saw `paperclip ✗ failed — Failed to get tools`. It then sent a request to the bridge without the bearer token and got HTTP 401. That 401 came from its own probe, not from OpenCode. The run still succeeded with the reply "Switched the working design to blue …", because a plain chat reply counts as the result. **Expected behavior** Every native OpenCode session starts with the Paperclip MCP tools. If the tools cannot be loaded, the session fails with a named reason, and the turn does not start without tools. **Steps to reproduce** 1. Start OpenCode 1.18.34 with a remote MCP server whose first `tools/list` request fails (for example, the server resets the socket one time). 2. `GET /mcp` returns `{"paperclip":{"status":"failed","error":"Failed to get tools"}}`. OpenCode keeps this status for the life of the process. 3. `POST /mcp/paperclip/connect` returns `true`, and then `GET /mcp` returns `connected` (about 8 ms). We measured this with the real 1.18.34 binary. This is the same class as #20886: `Connection: close` made the one-shot startup connect fail more often (about 9 of 25 sessions), and #20921 removed the trigger. Any other transient failure at startup still produced a toolless session. This change removes the class. ## What Changed - `opencode-server-driver.ts`: while a session opens (fresh, resumed or recovered), the driver reads OpenCode's `GET /mcp`. If the `paperclip` server is not `connected`, it calls `POST /mcp/paperclip/connect` up to two times. If the server is still not connected, it throws `paperclip_tools_unavailable`. The existing startup retry restarts OpenCode for this error (three attempts in total). After that, the session fails visibly. - The check runs in parallel with session creation or resume. The added work on a healthy start is one local HTTP request. - `fake-opencode-server.mjs`: the fixture now serves `GET /mcp` and `POST /mcp/paperclip/connect`. It connects to the real bridge with the configured token, and it can simulate failed connects. - New `paperclip-tools-startup.test.ts`: reconnect after one failed connect, restart after a process stays failed, reconnect on a saved session that resumes in a new process, and a named failure when the tools never load. All four tests fail on master (master opens a toolless session). ## Verification - `pnpm exec vitest run src/drivers/opencode/paperclip-tools-startup.test.ts` (in `packages/paperclip-runner`): 4 passed. Red on master: 4 failed. - All OpenCode driver, proxy, live runnerd OpenCode and native-runtime server suites that use the fixture: 29 files, 567 tests passed. - `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11 passed. `tsc --noEmit -p packages/paperclip-runner`: clean. - Real OpenCode 1.18.34 probe (failure, then `connect`, then `connected`) as in the steps above. - Paid chat eval run 43117909094 on this change (steering.steer-active and three other OpenCode chat cases). Result is in a comment on this PR. ## Risks - Low. A healthy start adds one local `GET /mcp` request, in parallel with session creation. - If OpenCode cannot load the Paperclip tools after reconnects and restarts, the session now fails with `paperclip_tools_unavailable`. Before, it ran without tools. This is intended. - No persisted run, session or pack state changes. A session saved by the previous release resumes through the same open path and gets the same check. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
23909077ad |
fix(retry): re-deliver the failed run's wake when the board retries it (#20970)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails, the board shows a Retry button. The Retry starts a
new run with the wake reason `task_retry`.
> - A failed run can serve a wake that happens once, for example a
scheduled task monitor check (`issue_monitor_due`). The monitor is
consumed when it fires.
> - The Retry run did not tell the agent which wake it retries. The
agent saw only `task_retry` and the earlier interruption.
> - In the restart-chaos harness, the agent followed its brief ("act
only on a board comment or a scheduled monitor check"), decided that the
Retry was not the monitor check, and changed nothing. The run succeeded,
but the task stayed `in_progress` with no run, no wake and no monitor.
The person clicked a Retry that worked mechanically but lost the work it
was for.
> - This pull request records the retried run's own wake reason on the
Retry's continuation trigger (`trigger.retriedReason`) and shows it in
the wake summary.
> - The benefit is that a Retry re-delivers the same wake that failed.
Dotta's rule: if a Retry button shows up, it still needs to work.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude, Daytona warm,
volume-free server mode, restart fault K1), task t03 (workload W2). The
phase-2 monitor wake started run `60d821af`. That run failed with
`native_session_interrupted`. The board Retry was accepted (HTTP 202).
Retry run `a5a3958e` started 2 s later and succeeded. Its context had
`wakeReason: task_retry` and `executionContinuation.trigger = { reason:
"task_retry", sourceRunId: "60d821af…" }`, with no reference to
`issue_monitor_due`. The agent's summary: "This wake was a task_retry
resuming the previously interrupted run, not a new board comment or
scheduled monitor check, so per instructions I changed nothing." The
issue stayed `in_progress` with `executionRunId = null`, monitor status
`triggered` and no `nextCheckAt`. The harness graded
`R3:retry_did_not_complete` and U1 `no_owner_no_action 182s`.
**Expected behavior**
A Retry of a failed run re-delivers that run's wake. A Retry of a failed
monitor check is handled as that monitor check.
**Steps to reproduce**
1. Give an agent a task that acts only on a scheduled monitor check.
2. Let the monitor fire, and make the monitor run fail (restart fault K1
during the run).
3. Click Retry on the task.
4. Before this change, the Retry run's prompt says only `reason:
task_retry`, and the agent does not do the monitor work.
**Additional context**
The same signature (`R3:retry_did_not_complete:W2C:K0`, Codex, monitor
rewake run `bac90594` failed `native_session_interrupted`, Retry run
`cfb24146` succeeded without the phase-2 marker) is in wave
`w-20261111T192658-s5201` task t06. That case was attributed to the
response-wake loop that #20935 fixed. Its shape matches this cause: the
Retry run served the monitor wake but did not do the monitor work. That
wave kept no database, so I could not read its run context directly.
Refs #20935
## What Changed
- `buildExecutionContinuation` adds `trigger.retriedReason` when the
current wake is a Retry (`task_retry` or `retry_failed_run`). The value
is the source run's own wake reason. A Retry of a Retry inherits the
first non-retry reason from the source run's saved continuation.
- `ExecutionContinuationEnvelope.trigger` gets the optional
`retriedReason` field.
- `renderPaperclipWakePrompt` adds `- retried wake: <reason> (this Retry
re-delivers the wake the failed run did not finish; handle it as that
wake)` to the wake summary for fresh and resumed sessions.
- New test in `server/src/services/execution-continuation.test.ts` on
embedded Postgres: a failed `issue_monitor_due` run, a Retry of it, a
Retry of that Retry, and a non-retry wake.
## Verification
- Red on master: `cd server && npx vitest run
src/services/execution-continuation.test.ts -t "carries the failed run"`
failed with `trigger` = `{ reason: "task_retry", sourceRunId }` and no
`retriedReason`.
- Green: the same file passes (45 tests).
- `tsc --noEmit` passes for `packages/shared`, `packages/adapter-utils`
and `server`.
- `pnpm verify` exits 2 (a shared type changed), so required CI supplies
broad coverage.
- Chaos: the existing cell W2 × K1 (Claude, Daytona warm) reproduces
this when the fault lands in the monitor rewake run. W2C (Codex) shows
the same shape. I did not rerun a wave for this PR.
## Risks
- Low risk. The change adds one optional field and one prompt line. It
changes no admission, lock, status or authority path.
- The field is computed only for Retry wakes. Other wakes are unchanged.
- Runs saved by an older version have no `retriedReason`. A Retry built
from them reads the source run's `wakeReason`, which older runs already
saved.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
2539004201 |
fix(ephemeral): start a restarted volume-free server over the Runner controller files it wrote itself (#20952)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud servers run volume-free (`PAPERCLIP_EPHEMERAL_SERVER=true`). Postgres and object storage hold the durable state. A startup gate refuses legacy volume data that was not migrated. > - The gate reads one family of data from disk: Runner controller files under `runtime/paperclip-runner/durable-sessions/`. It treats a controller file that the database does not prove as legacy data. > - A volume-free server also writes these files during normal work. A warm follow-up restores the previous run's checkpoint as a full `durable.control-plane-state.v2` file. The file stays full until the new controller registers its database checkpoint writer. > - When the server crashes in that window and restarts on the same filesystem, the gate finds its own file and refuses to start. Every later start refuses the same way. This is a full outage that no Retry can fix. > - This pull request gives the gate a durable marker. A volume-free server claims an empty state directory before any controller can write. Files in a claimed directory belong to a volume-free server and do not block startup. > - The benefit is that a crash restart of a volume-free server starts again. Genuine legacy data keeps its protection, and the refusal now names the run, task and file. ## Linked Issues or Issue Description **What happened?** The first restart-chaos waves in volume-free server mode (harness flag `--ephemeral-server`, #20930) found this outage. Wave `w-20261111T212429-s6402` (Claude through ACPX, Daytona warm sandboxes) did a fresh-home deploy and then a `kill -9` of that server. The replacement server exited with `Volume-free server data is not eligible: native_controllers_require_migration`. A second start at +608 s exited the same way. The server stayed down. **Root cause** `hasLocalNativeControllerObligations` in `server/src/services/native-runtime/native-controller-migration.ts` scans every Runner session in the state directory. The kept instance shows the cause. The session's newest run `4a4d7ca1` was `running` (admitted 3 s before the kill). Its controller root held the full `paperclip.runner.durable.control-plane-state.v2` file of its warm predecessor `9c13b7f8`. A controller writes that full file when it opens and replaces it with the `external-control-plane-state.v1` marker only in `setCheckpointPersistence`. `preservedActiveCheckpoint` accepts only the marker, so the gate returned `true`. A Codex wave with the same fault sequence passed because no warm follow-up was in this window at the kill. Claude W2 cells start a new warm follow-up every 15 to 35 s, so the window is open much more often. **Can staging hit it?** Yes, with a restart that keeps the container. Railway documents that a manual Restart "restarts the process within the deployment's container", so the filesystem stays. Railway does not document whether an automatic crash restart reuses the container. A redeploy makes a new container, so it does not hit this. **Expected behavior** A volume-free server starts again after any restart. State that it wrote itself is disposable, because the database checkpoint is the durable record. Data that a volume-backed server wrote keeps its current protection. ## What Changed - Add `server/src/services/native-runtime/volume-free-controller-state.ts`. `claimVolumeFreeControllerState(true)` writes `.volume-free-owner.json` atomically (temporary file, fsync, rename, directory fsync) when the state directory is absent or empty. It returns `owned` for a directory that has the marker and `unclaimed` for a directory with other files. `claimVolumeFreeControllerState(false)` removes the marker. - `server/src/index.ts` calls the claim right after the volume-free configuration check. This is before any controller or recovery can write. A server started without volume-free mode releases the claim, because it may write files that exist only on its disk. An `unclaimed` result logs a warning. - `hasLocalNativeControllerObligations` ignores the marker entries. It returns no obligation for a claimed directory. It now delegates to `findLocalNativeControllerObligation`, which returns the run, task and file (or `database_checkpoint`). - `assertEphemeralServerData` adds that detail to the `native_controllers_require_migration` refusal. The error says which run and file the database does not prove and why startup refuses. - Document the marker in `doc/DEVELOPING.md` ("Runner controller state on a restarted volume-free server"). ## Verification - Red, before the gate reads the marker: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-controller-migration.integration.test.ts -t volume_free_owner` failed with `AssertionError: expected true to be false` at the first gate check. The fixture is a running warm run whose root holds a full `durable.control-plane-state.v2` file, the same shape as the incident. - Green: the same test passes. It checks: claimed directory gives no obligation; release gives the obligation back; a non-empty unclaimed directory stays blocked; the refusal names the run; the controller file is never changed or removed. - `ephemeral-server-data.test.ts`, `ephemeral-server-contract.test.ts` and all 57 cells of `native-controller-migration.integration.test.ts` pass. Server `tsc --noEmit` passes. - Probe on a copy of the kept instance database and the killed server's state directory: the unmodified gate returned an obligation for run `4a4d7ca1` (source file `.../7dbf2d5c.../control-plane/control-plane-state.json`). - Chaos: the s6402 shape (`--ephemeral-server --adapters claude --workloads W2 --faults K1,K2,K15 --k15-fresh-home --seed 6402`) runs on this branch merged with #20930. The result is added in a comment. ## Risks - A filesystem that a volume-free server claimed and that a pre-marker, volume-backed server then wrote is not detected. This needs a downgrade and a mode change on one disk. A current server in volume mode removes the marker, which closes the normal path. - A volume-free server on a directory written before this change (no marker) keeps today's refusal. A new deploy makes a new container, so staging claims the empty directory on first start after this deploy. - Other scanners of the state directory filter on the 64-hex session pattern, so the marker file does not change them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d1e4109df1 |
fix(runner): keep a connected adopted Runner through a failed liveness probe (#20965)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, the controller adopts the remote Runner
that kept running in the sandbox. A monitor then checks the Runner's
liveness every 250 ms through the environment plugin.
> - #20940 and #20945 made a stalled plugin probe "unknown" for the
launched Runner's monitor, reattachment and launch. The adopted Runner's
monitor still failed the run on the first probe that failed without an
answer.
> - A short plugin stall after adoption therefore failed a healthy run
with `native_adopted_runner_identity_unverifiable`.
> - This pull request uses the evidence that is available: while the
adopted Runner holds its authenticated connection to the controller, a
liveness check that fails without an answer does not fail the run, for
at most the reconnect grace.
> - The benefit is that a healthy adopted Runner survives a short plugin
stall. A check that reports the process gone still fails the run at
once.
## Linked Issues or Issue Description
Refs #20940, #20945
**What happened?**
`#watchAdoptedRunner` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts` called
`failTransport("native_adopted_runner_identity_unverifiable: runner
liveness could not be revalidated")` on the first `isAlive()` rejection.
For a remote adopted Runner, `isAlive()` is a plugin
`environmentExecute` RPC. A plugin stall makes it reject, even while the
Runner holds its authenticated connection and works normally.
**Expected behavior**
A rejection is not an answer. While the Runner holds its authenticated
connection, the monitor keeps checking. The bound is the reconnect
grace. A check that returns `false` (the process is gone) still fails
the run at once.
**Steps to reproduce**
1. Adopt a remote Runner after a controller restart.
2. Stall the plugin worker, so that the liveness probe rejects.
3. Within 250 ms the run fails with
`native_adopted_runner_identity_unverifiable`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
aaf76a3ca6 |
fix(native): start the configured review stage when a native worker reports done (#20882)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tasks can have an execution policy with a review stage that names an
independent human reviewer
> - The issue route starts that stage when the assignee moves the task
to done or in_review
> - Native runs do not use that route. The native finalizer writes the
task status through the issue service, which does not run the stage
transition
> - So a native worker that reports verified work as done closes the
task without the required review. A worker that reports needs_review
moves the task to in_review without the configured stage or its reviewer
> - This pull request makes the native completion path start the
configured review stage, as the route does
> - The benefit is that a configured independent review gates Done for
native workers too
## Linked Issues or Issue Description
**What happened**
A task had an execution policy with one review stage for a human
reviewer. A native worker (OpenCode on the Paperclip runner) finished
the work and reported it. When the worker reported done with verified
evidence, the task became done and the review stage never started. When
the worker reported needs_review, the task became in_review with a
generic review card, `executionState` stayed null, and the configured
reviewer was not assigned. The chat eval `tasks.completion-with-gate`
timed out in setup because the task never reached the pending review
stage.
**Expected behavior**
Verified work on a task with a configured human review stage goes to
in_review with that stage pending, the stage reviewer assigned and the
worker kept as the return assignee. The task becomes done only after the
reviewer approves.
**Steps to reproduce**
1. Create a task assigned to a native agent with `executionPolicy.stages
= [{ type: "review", participants: [{ type: "user", userId: <reviewer>
}] }]`.
2. Let the agent finish the work and report done with verified evidence.
3. Read the task. Before this fix it is `done` and `executionState` is
null.
## What Changed
- `server/src/services/issue-execution-policy.ts`: add
`completionReviewStageEntry`. It applies the full policy transition that
the route applies for `status: "done"` and returns `starts` (with the
stage patch), `none` or `rejected`.
- `native-run-finalizer.ts`: a done decision on a task whose policy
starts a stage becomes in_review with reason
`execution_stage_review_required`. When the policy refuses the
transition, the decision is in_progress with reason
`execution_stage_transition_rejected` and a recovery note. It is never
done. Deliverable checks still run first.
- `status-decision-committer.ts`: for a stage-entry decision, apply the
stale-owner check, start the stage from the locked task in the same
status update (this also clears a monitor that the new assignee cannot
use), and wake an agent participant with `execution_review_requested` or
`execution_approval_requested`, as the route does. If the policy
changed, the saved decision is stale and finalization retries.
- `native-completion-feedback.ts`: on such a task, tell the worker to
report done so the configured stage requests the review. Reject
needs_review with no request or with any review request. Approval
requests keep their own path, because they can be governed action cards.
- New regression test `native-execution-policy-review-stage.test.ts`
(embedded Postgres, 6 tests).
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-execution-policy-review-stage.test.ts`
(6 tests): no stage (done), human stage (in_review, reviewer assigned,
monitor cleared), agent stage (in_review, reviewer woken), refused live
review cycle (in_progress, not done), and feedback including mixed and
approval requests. The stage tests fail without the finalizer change
(the task becomes done).
- Related suites pass: `historical-task-questions`,
`native-completion-feedback`, `native-run-finalizer-telemetry`,
`native-terminal-status-projection`, `status-arbiter`,
`task-monitor.integration`, `__tests__/issue-execution-policy` (247
tests).
- Server `tsc --noEmit` shows no errors in changed files. The local
worktree has unbuilt packages, so CI supplies the full typecheck.
## Risks
- The agent-reviewer wake is a system workflow wake, like the route's.
Unit tests prove the wake row. An end-to-end native agent review after
this wake is not yet exercised in a paid eval.
- A decision saved with a new reason and resumed by an older release
projects in_review or in_progress without the stage. The task does not
close, so the older release fails safe.
- Agents on tasks with a configured review stage now get a rejection
when they report needs_review with no request or with a review request.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
827bd21017 |
fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy
## Linked Issues or Issue Description
Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).
**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.
**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.
**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.
**Paperclip version or commit**
master at
canary/v2026.1112.0-canary.0
|
||
|
DottaandClaude Opus 5.5
|
ba905c7254 |
fix(evals): name each team member's readiness and grade configure-team structurally (#20879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `teams.configure-team` checks that chat gets an installed team ready member by member > - Its setup installs Product Engineering, removes the QA member's account, and requires a real probe split: QA missing an account, the senior coder ready > - The QA probe returned a generic `opencode_hello_probe_failed` error, so the setup stopped with "probes do not establish the required missing-versus-ready account split" > - The cause is a product classification gap: OpenRouter answers an empty key with HTTP 401 "Missing Authentication header", which the OpenCode probe did not treat as authentication (fixed in #20878, cherry-picked here) > - This pull request makes the setup blocker name each member's observed readiness, and replaces reply keyword checks with structural checks > - The benefit is that the eval can reach the chat journey and grades it without matching words ## Linked Issues or Issue Description Refs #20878 **What happened** `teams.configure-team` failed in setup. The QA member's probe was `fail` with `opencode_hello_probe_failed`. The senior coder passed. The setup error did not say which member was in which state. **Expected behavior** The QA member's probe reports the missing-authentication check, so the split forms and the chat journey runs. When the split does not form, the setup error names each member's observed status and checks. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.teams.configure-team` on master. 2. Read `snapshots/chat-catalog-team-initial-probes.json`. 3. QA shows `fail` with `opencode_hello_probe_failed:error`; the setup stops. ## What Changed - Cherry-pick of #20878: the OpenCode probe classifies a provider 401 as `opencode_hello_probe_auth_required`. - The setup blocker now lists each member as `name (expected missing account | expected ready | other member): status [non-info checks]`. - Grader: remove the member-name substring check and the "team is ready" regex. Identification is the rendered link to the missing member's own page, which the case already required. The followup is graded by a non-empty reply from its own succeeded run, not by the words "ready/configured/verified/account". Fresh per-member probes, preserved working configuration and revisions, and the attributed repair stay unchanged. - Calibration: reworded and non-English valid followups pass; missing handoff, empty followup, and failed followup run fail. A setup test with the recorded failure shape checks the new error text. - `oracleVersion` bumped to `paperclip.chat-catalog.team-configure.v3`; `parse.ts` added to `sourceFiles`. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/external-team-configure.test.ts tests/runner-e2e/chat-catalog-domains/external-team-configure-prerequisite.test.ts tests/runner-e2e/chat-runtime-profile.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs` passes. - The paid run `chat-features.runner-opencode.local.teams.configure-team` is still to run. ## Risks - The removed "team is ready" regex failed open. A false whole-team claim in the reply now needs human reading of the transcript. The structural checks still reject a missing handoff, a missing repair, and any member that fails its fresh probe. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b7446969d7 |
fix(evals): give choose-capable-target a saveable computer and grade it structurally (#20881)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that agents answer real user questions through the product. > - The case `execution-environments.choose-capable-target` asks "Which computer can run Morgan with Claude Code?" while environment reads are board-only. > - Its setup saved a built-in fake sandbox as the "unsupported" computer. The product refuses to save fake sandboxes (since #4449), so setup failed with 422 before any agent run, and the test API hid the reason. > - No computer that the product can save is advertised as unable to run Claude Code, so the original contrast cannot be built through supported APIs. > - This pull request changes the contrast to advertised support versus runtime health, grades the reply by structure instead of keywords, and makes setup failures show the product's validation reason. > - The benefit is that the case can measure its user promise, and a future setup 422 says why. ## Linked Issues or Issue Description **What happened** In the full chat catalog run, setup failed with `POST /api/companies/<id>/environments returned 422; body withheld`. A focused server test shows the reason: "Built-in fake sandbox environments are reserved for internal probes and cannot be saved." The fixture payload was `driver: "sandbox", config: { provider: "fake" }`. **Expected behavior** Setup builds its computers only through supported APIs, the agent run starts, and the grader checks the user promise without reply keywords. A setup validation failure states the product's reason. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.execution-environments.choose-capable-target` on master. 2. Setup posts a fake sandbox environment and gets 422. 3. `cd server && pnpm exec vitest run src/__tests__/environment-routes.test.ts -t "rejects persisted fake sandbox environments"` shows the same rule. ## What Changed - `runtime-capability-choice.ts`: setup saves an SSH computer on a reserved `.invalid` host (no key, never probed) beside the local fixture computer. Production capabilities advertise Claude Code on both. The `enableEnvironments` experimental setting is still set only through `PATCH /api/instance/settings/experimental` and restored in cleanup. - Grader: removed the reply regex and keyword checks. New checks: agent gets 403 on capabilities and probe; the reply renders a link to `/<prefix>/company/settings/instance/environments` (or a target's edit page); the reply has no board-only SSH settings (host, user, path); targets (including status and metadata) and the worker are unchanged; exactly one chat turn and no tasks. `verifyResult` clicks the rendered link and checks that the page shows both computers and loads the same capabilities. - `execution-environments.yaml`: context changed from "one target supports the harness and another does not" to support versus health, with the reason. Recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in the manifest test, and the inventory digest is updated. - `oracleVersion` is now `paperclip.chat-catalog.runtime-capability-choice.v2` in code and in `chat-catalog-descriptors.json`; `sourceFiles` adds `internal-support.ts`, `server/src/routes/authz.ts` and `server/src/services/environment-config.ts`. - `api.ts`: cookie-session 400/409/422 failures now include the product's `error` message, at most 240 characters, with request string values (8 or more characters), the session cookie and long token-like strings redacted. `details` and all other statuses stay withheld. - Includes a cherry-pick of #20880 (agent environment denials link the environment settings page). Without it, the agent has no route to give. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `runtime-capability-choice.test.ts`, `runtime-computers.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`, `api-private-transport.test.ts`, `api.test.ts`, `api-response-reading.test.ts`: 73 passed. - Grader calibration: valid and reworded replies with the settings link pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page links fail. Replies with the SSH host, user or path fail. Unsupported capability, reachable host, 200 boundary, changed target or worker, an extra run and a created task fail. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Paid run still needed: `chat-features.runner-opencode.local.execution-environments.choose-capable-target`. ## Risks - Premise change: the case no longer has a saved computer that is advertised as unable to run Claude Code, because the product cannot save one. The "avoid an incompatible target" clause stays in the expected outcome, but no saved target exercises it. - The reply's claims about health are not graded by words. A reviewer must read the paid run's reply. - The 400/409/422 reason can show product text that the test API hid before. It is bounded and redacted. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
DottaandClaude Opus 5.5
|
068d814735 |
test(chaos): run waves in Cloud's volume-free server mode; make W2 phase 1 yield (#20930)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks against an isolated control plane and injects faults > - Cloud servers run in the volume-free mode (`PAPERCLIP_EPHEMERAL_SERVER=true`, durable agent files, run logs in object storage) > - The harness never set that mode, so every wave ran a different server mode from staging. Paths that exist only in that mode were never tested. One of them is the per-agent directory operation lock, so `--fail-recovery-step directory-lock` could never reach its step with any workload > - The re-wake workloads W2 and W2C also lost coverage: an agent that reported phase 1 with disposition `done` closed the task, and the harness never sent the re-wake (four of eight Claude cells on 2026-11-11) > - This pull request adds `--ephemeral-server` and makes the W2/W2C brief ask for a `yielded` phase-1 report > - The benefit is that chaos waves can run in staging's server mode and the re-wake cells test the re-wake on every provider ## Linked Issues or Issue Description **What is the problem?** The restart chaos harness cannot run the control plane in the volume-free mode that Cloud and staging use. The `directory-lock` recovery step (`agent_directory.operation_lock.before_acquire`) is reached only for disposable remote copies, which need `PAPERCLIP_EPHEMERAL_SERVER=true`. W2 and W2C cells also stop before the re-wake when the agent reports phase 1 as `done`. **What is the expected behavior?** A wave can opt in to the volume-free mode on Daytona, and every restart of the wave keeps that mode. The W2 and W2C cells reach the re-wake. **Steps to reproduce** Run `--fail-recovery-step agent_directory.operation_lock.before_acquire:1` on Daytona with any workload: the check `recovery_step_failed_once` reports "never reached". Run W2 on Claude: the first run ends with the task `done` and `rewake_sent` is "never sent". ## What Changed - `assertIsolatedServerEnvironment` (`tests/runner-e2e/harness-env.ts`) takes `expected.generatedSecretsMasterKey`. It admits `PAPERCLIP_SECRETS_MASTER_KEY` only when that parameter is set and equals the value exactly. `PAPERCLIP_SECRETS_MASTER_KEY_FILE` stays refused. The check runs over the final server environment. - `--k15-fresh-home` implies `--ephemeral-server` and defaults `--storage` to `s3`. A fresh instance home without the volume-free mode loses the agents' instruction bundles, which staging keeps in Postgres and object storage. - `--ephemeral-server` (`config.ts`, `types.ts`, `main.ts`, `supervisor.ts`, new `ephemeral-server.ts` with its unit test `ephemeral-server.test.ts`): it needs `--storage s3` and `--env daytona` and refuses K14. Every server of the wave gets `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES`, `PAPERCLIP_SHARED_UI_ASSETS`, the wave's Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, one stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config key file) and `RUN_LOG_S3_*` in the wave's object store. - `rewake.ts`: the W2/W2C phase-1 brief asks for `paperclip_finish` with disposition `yielded` and a `response_wake` continuation. - W2/W2C tasks are created in `backlog` with their assignee, and a board comment starts phase 1 (`startComment` in `workloads.ts`, used by `observer.ts`). A passive board wait needs a comment-started run; without it the yield queues an immediate continuation and phase 1 wakes itself. The start comment is posted once (saved-comment check plus a stable `clientRequestId`), and the brief keeps phase 2 out of blocking `remainingWork`. - Volume-free mode drops the repository workloads (B1, B5, F1, R1) from the default list and refuses them when listed: their fixture is a `local_path` project source, which a volume-free server refuses. - `README.md`: a section for the new flag and a note on the brief. - `shape-cells.test.ts`: option parsing and refusals for the new flag. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/shape-cells.test.ts` passes (16 tests). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live proof wave (Daytona, Codex, W2, K1 and K15 with a fresh home, `--storage s3 --ephemeral-server`): wave `w-20261111T205259-s6301` (2 tasks). The first server and both replacements (K15 with a fresh home, then K1) started and reached ready in the volume-free mode. The server process had `PAPERCLIP_EPHEMERAL_SERVER=true`, `PAPERCLIP_DURABLE_AGENT_FILES=true` and `RUN_LOG_S3_BUCKET` set. Both W2 tasks left phase 1 `in_progress` (yielded), received the re-wake and ended `done`. The wave also found product faults (reported separately, not fixed here): a re-wake after K1 failed with `runner_harness_state_mismatch: backup_without_reusable_lease` and needed Retry. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/support.test.ts` passes, including the new master-key admission test (a matching wave key passes; an inherited key, a mismatched key and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` are refused). ## Risks - Low risk. The flag is opt-in and the harness is test-only. The brief change alters the W2/W2C prompt, so results from older W2/W2C seeds are not directly comparable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ef755cecc6 |
fix(evals): grade case facts by the saved fact, bind case kickoffs to the CEO run, stop block-on-design on a missing blocker (#20948)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys and grades saved
product state
> - Three cases failed in runs 43088288923 and 43101125481. In two of
them the product did the right thing
> - `cases.update-case-facts` used a word allow-list on a saved field.
`cases.create-case` read a field the product never fills for a new task
> - `task-coordination.block-on-design` timed out in the browser on a
missing row, which hid the real cause
> - This pull request grades what the user asked for, binds records by
structure, and stops with the real cause
> - The benefit is that each case fails only when the product fails, and
says why
## Linked Issues or Issue Description
**What happened**
- `cases.update-case-facts`: the CEO saved `browser: "Mobile Safari only
(bug reproduces only on mobile Safari)"` and kept the other fields. The
grader allowed only the words "safari" and mobile qualifiers, so "bug
reproduces" failed the first phase. The earlier rewording of the
scripted follow-up was not the cause: it is sent only after this phase.
- `cases.create-case`: the CEO created the case, the `workflow` and
`customer_brief` documents and Morgan's kickoff in one run, before
Morgan started. The grader required the kickoff's `sourceRunId` to be a
CEO run. The product sets `sourceRunId` to the run that holds the task
being handed off, so it is always null when the kickoff creates the
task. The grade never passed, so the harness kept waiting. Morgan then
found no app to test, handed the case back to the CEO, and the extra CEO
run broke the run bound.
- `task-coordination.block-on-design`: the CEO saved no Design blocker.
The scripted human then waited 30 s for the "Blocked by" row and failed
with a locator timeout.
**Expected behavior**
Each case accepts every valid product behavior and rejects the failures
that matter. A failure names its cause.
**Steps to reproduce**
1. Regrade the retained observation of run 43101125481 for
`cases.create-case` with the old grader: `real-first-morgan-kickoff` and
`required-documents-persist-before-investigation` fail although the
records are correct.
2. Call `corrected({ browser: "Mobile Safari only (bug reproduces only
on mobile Safari)" })` with the old grader: it returns false.
3. Run `task-coordination.block-on-design` when the CEO saves no
blocker: it fails on a UI locator instead of on the missing blocker.
## What Changed
- `cases.update-case-facts` (oracle v8): grade the saved correction
structurally, with no word lists. The browser fact differs from the
case's own setup value, and no field keeps the setup's stale literal
"not yet confirmed", in the saved case and in the reviewer's handoff
context. A person reads whether the wording says mobile Safari.
- `cases.create-case` (oracle case-create.v2): bind the kickoff and its
documents to the CEO chat run that was active when the kickoff was
saved, not to `sourceRunId`. Admit one CEO run on the primary task woken
by Morgan's own case handoff back to the CEO, and add a check that binds
it to that handoff. Run bound is now 2 to 3.
- Shared run policy (`chat-catalog-run-policy.ts`,
`chat-catalog-contract.ts`): new optional `maximumLeadCaseHandbackRuns`
for `source-settled-with-active-work`. The default is 0, so other cases
grade exactly as before. No other oracle version changes.
- `task-coordination.block-on-design` (oracle v9): after the first turn,
stop with "The CEO did not save Design as a blocker on Website" when the
edge is missing.
- Descriptors updated for the three cases.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`integrator-case-create`, `integrator-case-facts`, `runtime-dependency`,
`chat-catalog-run-policy`, `chat-catalog-readiness`,
`chat-catalog-failure-retirement`, `chat-catalog-consistency` and
`chat-ordinary-phase-callers` tests: all pass.
- Calibration in both directions:
- case facts: the retained values of runs 43088288923 and 43107042937
and four rewordings pass. The unchanged setup value, a kept stale
literal (in any field), empty facts and a first turn that left the value
unchanged fail.
- create-case: the retained run 43101125481 regrades to pass. Documents
from another run, a kickoff outside the CEO run, a kickoff by Morgan, a
handback with no handoff, a handback woken another way and two handbacks
fail.
- run policy: an undeclared handback, a lead run woken another way and a
second handback fail.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. Only eval code changes. The shared run-policy option is
opt-in, and its default keeps the old result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
9c56e5465c |
fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question and wait for the answer. Native task status comes from the server's status arbiter, not from model prose > - When a native turn ends without `paperclip_finish`, the Runner sends a default report: `done`, objective satisfied, with the agent's final message as the only evidence > - A later comment from the person moves an open question into history (#15392), so the question no longer gates completion > - Together these close a task as `done` while the agent is still waiting for its own answer and has produced nothing. No Retry appears and nothing flags it > - This pull request keeps such a task `in_review` on the agent's own unanswered question when the only completion evidence is the agent's final message > - The benefit is that a "waiting for your answer" turn can no longer become a silent false completion ## Linked Issues or Issue Description Refs #15392 **What happened?** Restart-chaos wave on master |
||
|
DottaandClaude Opus 5.5
|
65ebd96bcf |
fix(chat): keep the person's brief as a document on a new case's primary task (#20954)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - With Cases enabled, a person can ask chat to open a case from a
brief and hand it to an agent
> - The case scenario expects the brief and a workflow document on the
primary task before work starts
> - In one run the CEO copied the brief's facts into case fields but
saved only the workflow document
> - The Cases chat guidance says nothing about where the person's brief
belongs
> - This pull request adds one guidance sentence: save the brief with
`initialDocuments` in the same case-handoff request
> - The benefit is that the assignee always has the person's source,
saved before its run begins
## Linked Issues or Issue Description
**What happened**
In the chat eval `cases.create-case` (run 43107042937), the person
attached a customer bug brief and asked the CEO to open a case and have
Morgan investigate. The CEO created the case, its fields, the workflow
document and the kickoff. It did not save the brief on the primary task.
In an earlier run (43101125481) the CEO did save it, so the behavior
depends on the model's choice.
**Expected behavior**
The new primary task has the person's brief as a document, saved in the
request that creates it, before the assignee starts.
**Steps to reproduce**
1. Enable Cases. Attach a brief in chat with the CEO.
2. Ask: "Open a case for this customer bug, record the facts from my
brief, and have Morgan investigate first."
3. Read the primary task's documents: in some runs only `workflow`
exists.
## What Changed
- `CASES_CHAT_DIRECTIVE` (rendered only while Cases is enabled): one
sentence. When the person opens a case from a brief, save the brief as a
document on the primary task with `initialDocuments` on `POST
/api/companies/{companyId}/case-handoffs`. Case fields do not replace
the brief.
- `case-history-api-discovery.test.ts`: a regression test for the new
guidance.
## Verification
- Red: the new test failed before the change. Green: `npx vitest run
src/__tests__/case-history-api-discovery.test.ts
src/__tests__/agent-conversations.test.ts` (67 passed).
- The route already accepts `initialDocuments` for a newly created
primary task (`packages/shared/src/case-handoffs.ts`).
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. The sentence renders only when Cases is enabled. No API or
permission changes.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
6cfb1f3d87 |
fix(environments): resume a warm sandbox that its finished predecessor is still releasing (#20953)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm Paperclip Runner session keeps its Daytona sandbox between runs. The next run of the same task resumes that sandbox through a `reuse_by_environment` lease. > - Acquisition only resumes a lease that is `released`, `retained`, or held by the same run. A finished run releases its lease only after it saves its instruction and workspace files, which takes several seconds. > - A board wake that arrives in that window does not see the warm lease. It creates a fresh sandbox. The session's durable backup cannot be restored there, because its source lease is still held. The run fails with `native_session_recovery_failed: runner_harness_state_mismatch: backup_without_reusable_lease` and the task needs Retry, which starts a new provider session. > - This pull request makes acquisition wait, for a short bounded time, for a lease that a recently finished predecessor is still releasing, and then resume it. > - The benefit is that the follow-up continues the same warm session in the same sandbox, with no Retry, no restore and no lost provider context. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T205259-s6301` (Codex, Daytona warm, volume-free server, `--ephemeral-server` from #20930), task t01. The board re-wake run `e7129ec7` started at 20:57:44. Its predecessor `5cf8c306` reported `run.terminal` at 20:57:38 but saved its instruction copy until 20:57:51, so its warm lease was still `active`. At 20:57:53 the new run recorded `startup reuse: environment (fresh)`. It then failed with `runner_harness_state_mismatch: backup_without_reusable_lease` (error code `native_session_interrupted`). Recording the `workspace_finalize` operation then failed with a Postgres deadlock, and the run settled `native_finalization_missing`. Retry run `79aa18ec` worked with a fresh provider session. The K1 restart in that wave was at 20:55:32 and is not the cause: six warm runs after it reused the sandbox normally. **Root cause** `reusableLeaseCanBeResumed` in `server/src/services/environment-runtime.ts` excludes an `active` lease held by another run. That rule is correct for a running predecessor. For a predecessor that has already ended, the `active` state is a short release tail. The new run creates a sandbox, and `retiredSourceAuthorizesNativeBackupRestore` (`native-retired-backup-source.ts`) correctly refuses to restore the backup from a source lease that is not yet `released` or `retained` (`idleWarmSourceRetiresBackup`). The executor's last branch (`native-session-executor.ts`, host preparation, created allocation) then throws `backup_without_reusable_lease`. **How often** Seen in 2 of 361 retained wave result directories on this machine (`s6301` today, `s5201` earlier). It needs a board wake within a few seconds of the predecessor's terminal event, so it is timing dependent. ## What Changed - `server/src/services/environment-runtime.ts`: add `waitForSettlingReusableLease`. Before the plugin and built-in sandbox paths select a reusable lease, acquisition polls (250 ms) for up to 20 s while a matching `reuse_by_environment` lease is `active` under a different run that is `succeeded`, `failed`, `cancelled` or `timed_out` and finished in the last two minutes. It then continues with the normal selection, which resumes the released lease. - A running predecessor causes no wait (unchanged: the new run gets a new sandbox). A predecessor that ended more than two minutes ago and never released causes no wait. - `server/src/__tests__/environment-runtime.test.ts`: two red/green cases (succeeded and failed predecessor release the lease 400 ms after the successor starts acquiring) and one case for an old unreleased lease (no wait, new sandbox, old lease untouched). - `doc/execution-semantics.md`: describe the wait next to the existing warm-reuse rules. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/environment-runtime.test.ts -t predecessor` fails both cases with `expected 'warm-2' to be 'warm-1'`: the successor created a second sandbox instead of resuming the first. - Green: the same command passes (3 tests). `environment-runtime.test.ts`, `plugin-warm-tasks.integration.test.ts` and `environment-run-orchestrator.test.ts` pass (288 tests). Server `tsc --noEmit` passes. - Not verified in a chaos wave. The window is a few seconds wide and appeared in 2 of 361 waves, so one wave cannot show the change. ## Risks - A successor can start up to 20 s later when its predecessor's release is slow. Creating a sandbox and restoring a backup takes longer, and failing needs a Retry. The two-minute bound keeps a stuck, never-released lease from delaying every later run. - If a caller held a lock that the predecessor's release needs, the wait ends at 20 s and acquisition behaves as before. - Compared with #20863 (fresh-sandbox fallback) and #20811 (unsaved provider state marker): this change does not alter either. It reduces how often a warm continuation reaches the fresh-sandbox branch. The backup restore proof is unchanged, so an unproven backup still fails closed with a working Retry. - The `workspace_finalize` deadlock in the same run is a separate defect and is not fixed here. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e01ba16bfd |
fix(native): continue the interrupted turn after a relaunch that failed before its runner existed (#20944)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs survive a server restart: recovery verifies the old runner stopped, then resumes the same provider session and its interrupted turn > - Recovery is a chain of steps. When one step fails once, the next attempt must still be allowed to finish the work > - Here the first recovery step was correct, its runner spawn failed once (EAGAIN), and the next attempt refused to continue the interrupted turn. The run died and needed a manual Retry > - This pull request lets the scheduled same-run resume prove that the failed launch left nothing behind, and then continue the turn > - The benefit is that a single failed spawn during recovery no longer turns a recoverable lost turn into a terminal failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T190113-s5303` (local, codex, A1, fault K1 with `--fail-recovery-step runnerd-start:1`) on master `683fd1e9a5`. Run timeline: 1. K1 killed the server mid-turn. Recovery recorded `native.local_process_stopped` for the dead runner and chose `resume_dead_runner`. 2. That attempt wrote `native.process_start_requested`, then the spawn failed with `spawn paperclip-runnerd EAGAIN`. The executor persisted a scheduled same-run resume (`recoveryMode: exact_checkpoint_resume`, `providerSessionEstablished: true`, `checkpointExists: true`, `recoverable: true`). 3. The scheduled attempt resumed the session. The runner reported `provider_turn_lost_on_restore` (recoverable). The run ended `native_provider_terminal_failed` with "Automatic recovery stopped", and the task needed Retry. The same seed with K1 and no injected step passes. Root cause: in `executePaperclipNativeSession` (`native-session-executor.ts`), `resumeInterruptedTurn` is `restartRecovery.kind === "resume_dead_runner" || scheduledResumeContinuation`. The second attempt is not a restart claim, so it depends on `scheduledResumeContinuationAllowed` (`scheduled-resume-continuation.ts`). That decision refused with `local_stop_unverified`: `readNativeLocalProcessStop` reads the newest launch event, and the newest one was the failed `process_start_requested` with no process identity and no stop receipt. The checkpoint evidence was correct; the stop evidence was the gap. The refusal was not logged. **Expected behavior** When a launch fails before its runner exists, a scheduled same-run resume proves on this host that no runner for the run exists and that the launch wrote no runner state, records that stop, and continues the interrupted turn. A live runner for the run still blocks it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads A1 --faults K1 --fail-recovery-step runnerd-start:1 --fault-after-seconds 10 --concurrency 1 --tasks 1 --seed 5303` ## What Changed - `scheduled-resume-continuation.ts`: when the decision refuses only for `local_stop_unverified` on a local target, `scheduledResumeContinuationAllowed` calls the existing `recordUnidentifiedLocalRunnerStop` (the same proof explicit continuations use). It requires that the newest event is this host's launch request, that no process carries `--run-id <runId>`, and that no runner state for the run changed after the launch request. On success it records `native.local_process_stopped` (`basis: unidentified_runner_absent`) in the claim transaction and decides again. All other checks (exact checkpoint, recoverable, session established, checkpoint exists, no replacement, admitted attempt) still apply first. - `native-session-executor.ts`: logs the refusal reason when a scheduled `exact_checkpoint_resume` will not continue the interrupted turn. - `native-scheduled-resume-continuation.integration.test.ts`: two cases. A failed relaunch after a verified stop now continues the turn (`resumeInterruptedTurn: true`, attempt 2). A failed relaunch while a process with `--run-id <runId>` is alive still refuses. The fixture now advances `nextEventSeq` after inserting events directly. - `tests/restart-chaos/README.md`: records what the existing `runnerd-start:1` cell found and what now passes. ## Verification - RED on master: "continues the lost turn after a relaunch that failed before its runner existed" fails with `expected [ false ] to deeply equal [ true ]` (the runtime gets `resumeInterruptedTurn: false`, which is the path that produced `provider_turn_lost_on_restore` → `native_provider_terminal_failed`). - GREEN: `scheduled-resume-continuation`, `native-scheduled-resume-continuation.integration`, `native-exact-resume-failure`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `native-session-executor`, `native-session-resumption`, `heartbeat-native-restart-suspension` pass (1797 tests). `pnpm --filter @paperclipai/server typecheck` passes. - Chaos rerun of seed 5303 with `runnerd-start:1`: result is added in a comment on this PR. ## Risks - The proof is host-local: it only applies on the host that wrote the launch request, the same as explicit continuations. Remote targets and restart claims are unchanged. - Old saved state: a run already stuck in a scheduled resume after a failed spawn now continues on its next attempt instead of failing. That is the intended recovery; the checkpoint and attempt limits still bound it. - Sibling recovery steps (`teardown`, `directory-lock`) are Daytona-only. The saved waves that armed them (`s5311`, `s5312`) never reached the step, so this PR does not change or prove them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d9d5958d09 |
fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path - Paperclip's controller and the sandbox Runner exchange numbered commands; the Runner refuses a command whose number is not exactly one past the last it processed. - Staging's most common "Runner gone" failure was a follow-up controller whose journal was behind the Runner. #20928 removed the cause on the server (a warm follow-up now launches from the journal its retired predecessor saved) and added a defence that reconciles a journal behind only by the predecessor's settled close. - The Runner still treated an out-of-order command as fatal: `durable/state.rs` returned an error and the process exited. A controller with a stale journal from any other cause (state saved by an older release, a mid-run deploy) therefore lost the Runner process and the run could only fail as "Runner gone" (`runner_remote_process_identity_unavailable`). - This pull request makes the Runner refuse such a command with a precise result instead of exiting, and makes the server rest that run with a precise code and a Retry whose next run reconciles the journal (the #20928 defence) before it opens the session. ## Linked Issues or Issue Description - Refs #20928 (the server-side root-cause fix and the journal reconciliation this relies on) - Refs #20907 (makes runnerd exit text visible in the run log) - Duplicate search: searched PRs for "controller sequence contiguous", "controller_sequence_mismatch" and "out-of-order command"; none changes the Runner's handling. ## What Changed - `runner-core/src/durable/state.rs`: a new command whose `controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap) now yields `CommandDisposition::Reject` with `status: "rejected"` and `{ code: "controller_sequence_mismatch", message, expectedControllerSeq, receivedControllerSeq }`. The command is not executed or journaled and the last sequence is unchanged; the process keeps running. Replay of an already-processed command and the compacted-history rejection are unchanged. - `native-session-executor.ts`: a failure carrying `controller_sequence_mismatch` is classified as `native_controller_journal_behind_runner`, a permanent (terminal) failure with a Retry action that says no provider work began and that Retry reconciles the journal from the saved checkpoint or stops with a precise error. - Tests: - Rust `tests/durable_recovery.rs`: out-of-order (behind and gap) commands are rejected with the precise code, the journal is unchanged, and the next contiguous command executes. The existing `command_gaps_and_identifier_reuse_fail_closed` unit test now asserts the rejection instead of an error. - Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a follow-up controller with a journal missing the predecessor's close now gets `"code":"controller_sequence_mismatch"`, the Runner process stays alive and its durable sequence is unchanged; with the current journal it resumes. - Server: the classification and terminal disposition of the new code. ## Verification - Red (master Runner): the Rust test panics on `expect("an out-of-order command is not fatal")`; the real-runnerd test receives `runnerd exited unexpectedly with code 1: paperclip-runnerd: controller sequence must be contiguous: expected 6, received 4`. - Green: `cargo test -p paperclip-runner-core --test durable_recovery` (4 passed); `cargo test -p paperclip-runner-core` lib suite (the only other failure, `task_memory::tests::monitor_bounds_notices`, is a timing test that passes alone); `npx vitest run src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built from this branch (2 passed); server `native-session-executor.test.ts` focused tests (71 passed). - `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`: profiles match the release manifest. No ACPX profile, attestation, pin or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the change is Runner source behaviour with an existing command-result shape, so no profile version or `commandDigest` changes. ## Risks - A controller that is *ahead* of the Runner (a gap) is now rejected instead of fatal. Neither case executes the command; the Runner journal is unchanged, so this is strictly safer than exiting. - The rested run needs a Retry. Before this change the same run failed after its retries as "Runner gone" with no precise cause. - Runner source changes the Daytona image content ID; a new image is needed for chaos waves on this revision. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2719145ae2 |
fix(chat): save a requested start order as a first-class blocker (#20947)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People use chat to tell the CEO agent how work must be ordered
> - A person said "Do not start the website until the design is
approved"
> - The CEO only promised to hold the work. It did not save a blocker,
so nothing stopped the Website task
> - Chat guidance names blockers only when chat creates new tasks. It
says nothing about an order between existing tasks
> - This pull request adds one guidance line: save the order as a
blocker on the waiting task
> - The benefit is that Paperclip enforces the order and wakes the
assignee when the prerequisite is done
## Linked Issues or Issue Description
**What happened**
In the chat eval `task-coordination.block-on-design` (run 43088288923),
the person asked the CEO to keep Website from starting until Design is
approved. The CEO read both tasks and replied "I'm holding both until
the design is actually approved". It saved no blocker. Website stayed
startable.
**Expected behavior**
The CEO adds Design to Website's `blockedByIssueIds`. Website cannot
start while Design is open. Its assignee wakes when Design is done.
**Steps to reproduce**
1. Create Design and Website tasks in backlog. Assign Website to an
agent.
2. In chat with the CEO, say "Do not start the website until the design
is approved."
3. Read Website: `blockedBy` is empty in some runs, because the guidance
does not tell the agent to save the order.
## What Changed
- `AGENT_CHAT_DIRECTIVE`: one paragraph. When the person says an
existing task must not start until another is done or approved, add the
prerequisite to the waiting task's `blockedByIssueIds` with `PATCH
/api/issues/{id}`, and keep existing blockers. A comment or a promise
does not stop the work. Link both tasks.
- `agent-conversations.test.ts`: a regression test for the new guidance,
in the directive and in the rendered chat prompt.
## Verification
- Red: the new test failed before the guidance change.
- Green: `npx vitest run src/__tests__/agent-conversations.test.ts
src/__tests__/case-history-api-discovery.test.ts` (66 passed).
- Local scripted-human check (dev server, Playwright): with the blocker
saved, the Website task shows the Design row under "Blocked by", a Todo
click returns 200 without a Morgan run, Design done wakes Morgan with
`issue_blockers_resolved`, and the follow-up chat message sends.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. The change adds about 470 characters to the chat prompt. It
changes no API or permission. The route and field already exist and
agents already use them.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandPaperclip
|
6a43890326 |
Fix image attachments in remote native workspaces (#20939)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs must receive the files that users attach to their messages. > - Remote wake preparation returned file metadata without readable bytes. > - The historical chat reader also rejected every remote target. > - Command-backed sandboxes and SSH targets already provide a bound command transport. > - This pull request stages verified files through that transport and clears them after the turn. > - Agents can then open attached images with their normal image reader. ## Linked Issues or Issue Description Refs #19740. This change covers SSH and command-backed sandbox execution, including Daytona. Plugin Runner tasks without a command transport still need a separate versioned file-transfer contract. This PR does not close that broader issue. The open attachment-recovery PR #20778 concerns provider-session attachment journals, not input files. **What happened?** A user uploaded an image and sent it to an agent. The remote native run received its filename and size, but no readable file. Wake preparation reported `remote_workspace_staging_unsupported`. The historical read fallback also refused remote execution. **Expected behavior** Copy the exact admitted attachment into the bound remote workspace. Let the provider read the image. Preserve company, task, source-comment, size, and hash checks. Clear temporary file contents when the turn ends. **Steps to reproduce** 1. Select a native agent on a command-backed sandbox or Linux SSH environment. 2. Attach a PNG to its current task or in-app chat message. 3. Ask the agent to inspect it. 4. Check that wake preparation supplies a readable workspace-relative image path. ## What Changed - Add a remote staging owner with bounded commands, stdin transfer, hash verification, opaque paths, and safe image extensions. - Confine Linux filesystem operations through directory descriptors. Refuse symlinks and replacement files during cleanup. - Bind current wake files and authorized historical external-chat reads to the selected lease transport. - Join active writes during cancellation. Retain failed cleanup work until scope close. - Tell providers to open image files with an image-capable reader. - Handle stdin stream errors in the shared SSH transport. Failed uploads return failure instead of raising an unhandled EPIPE or reporting success. - Add database, transport, cancellation, integrity, cleanup, executor-wiring, and Linux filesystem regressions. Document support and limits. ## Verification - Prepared an isolated source checkout with `node scripts/agent-preflight.mjs --skip-rust-debug`. - Focused suites: `pnpm exec vitest run server/src/services/native-runtime/remote-attachment-staging.test.ts server/src/services/native-runtime/native-runner-file-handoff.test.ts server/src/services/native-runtime/chat-attachment-read.test.ts` — 74 passed, 4 Linux-only tests skipped. - Executor checks: `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t 'attachment staging to the remote|stages from the authenticated|verifies frozen staged admission'` — 7 passed. - Server typecheck: `pnpm --filter @paperclipai/server typecheck`, followed by the server layered check after the final changes. - `git diff --check` passed. The local secret, PII, and private-reference diff scan found no matches. - `pnpm verify` returned exit 2 because the documentation change requires broad CI. No full local suite is claimed. - The four Linux filesystem tests were skipped locally because this macOS host has no available Docker daemon. Linux CI executed all 17 remote-staging tests successfully, including those four confinement tests: [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266). - SSH regression: the real pipe test reproduced two unhandled EPIPE errors before the fix, plus false success when the peer exited with code zero. The fixed SSH tests and existing argv tests pass (12 tests). The remote-staging transport suite also passes (13 tests; 4 Linux tests require CI). - `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the SSH change. - Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all applicable CI jobs passed, including typecheck, build, general and serialized tests, Runner checks, canary dry run, and all eight browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43102035096). Linux CI also passed all 6 native SSH tests: [server shard 5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349). - Greptile reviewed that final revision at 5/5 with no remaining findings. The review thread is resolved. GitHub reports the PR mergeable with a clean merge state. - No live model turn or deployed release is claimed. ## Risks - Remote staging requires Linux, Node, and a command transport bound to the selected lease. Plugin Runner tasks without that transport retain their unavailable result. - Each file uses bounded independent commands. Slow transports can delay preparation or cleanup. File bytes stay out of command arguments and provider-facing error messages. - An abrupt controller or remote-host failure can bypass cleanup. Ephemeral sandbox disposal remains the backstop. Retained files are never blindly overwritten or deleted. - Previous-release provider sessions and checkpoints retain their format and identity. New turns receive newly staged input files. Existing restart-admission checks still apply. No migration, runner-image change, or account-credential transfer is required. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, code execution, and tests. No subagents were used. The context-window size is not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
757a20d2f1 |
fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at
|
||
|
|
5e2243f081 |
fix(ui): align sidebar avatars and navigation icons (#20926)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The left navigation mixes agent avatars, task status icons, and page icons. > - Agent avatars used 20px boxes. Task and page icons used 16px boxes. > - The different widths moved agent labels 4px to the right of other labels. > - This pull request gives these icons the same 16px square slot. > - Shared label positions make pinned and recent items easier to scan. ## Linked Issues or Issue Description Refs #19685, which added mixed agent and page pins. No duplicate alignment PR was found. **What happened?** Agent avatars in pinned and recent navigation were wider than task and page icons. Their labels started 4px farther to the right. Inline icon layout also produced different box heights. **Expected behavior** Use the existing 16px task and page icon size for all leading navigation icons. Center each icon in a square slot. Start the labels at the same horizontal position. **Steps to reproduce** 1. Pin an agent conversation and a page such as Routines. 2. Open agent conversations and tasks so both appear in Recents. 3. Compare the icon boxes and label positions across the rows. **Paperclip version or commit** Base commit: `c5e6ab5ca813`. **Deployment mode** Local source checkout. This is a client layout issue. ## What Changed - Use 16px agent avatars for pinned and recent conversation rows. - Give `SidebarNavItem` a fixed 16px square icon slot with centered contents and left-aligned text. The existing token scale supplies the size. - Add light and dark Storybook examples with mixed icons and long labels. The browser assertions compare slot sizes, label positions, and glyph bounds. - Add a production `CoordinatorSidebar` regression test that compares pinned and recent avatar dimensions with a task status icon. ## Verification - `pnpm exec vitest run ui/src/components/SidebarNavItem.test.tsx ui/src/components/CoordinatorSidebar.test.tsx ui/src/components/SidebarAgents.test.tsx` passes: 59 tests across three files. - Regression proof: the new production-avatar test fails with the original 20px avatars (`expected 20 to be 16`) and passes with 16px avatars. This addresses the initial review's test coverage gap. - `pnpm check:token-gates` passes. - `pnpm --filter @paperclipai/ui typecheck` passes. - `pnpm --filter @paperclipai/ui build` passes on the final source tree. - `pnpm verify --base origin/master` returns exit 2 because the new Storybook file needs broad CI coverage. The focused local tests run separately. CI must supply the remaining coverage. - Browser inspection on macOS checked all 13 mixed rows in light and dark themes. Every icon slot measured 16 by 16 CSS pixels. Every label started at the same position. Before and after captures use production components with fixture data; they do not prove server persistence or permissions. - Review `sidebar-icon-alignment--dark` and `sidebar-icon-alignment--light` in Storybook to inspect the mixed rows and run their layout assertions. - [Greptile review](https://github.com/paperclipai/paperclip/pull/20926#issuecomment-6806138954): 5/5 on `e779f4707e7826d577b23a2991f3f36755e954be`. The initial 4/5 review requested production Recents coverage. The added regression test addresses that finding; the thread is resolved. - CI queue recovery: the superseded first run was canceled. The next run stalled before runner assignment; GitHub accepted force-cancel but left it queued and refused the rerun request. The final empty commit requests fresh CI without changing the source tree. Required CI is tracked in [run 43095809539](https://github.com/paperclipai/paperclip/actions/runs/43095809539). Its jobs are still queued; CI completion is not yet claimed. - External blocker: after more than 15 minutes, the fresh run still has zero jobs. Other repository PR runs are also queued. A repository Actions or runner administrator must restore dispatch before the required typecheck, test, build, and E2E gates can report results. The root cause of the dispatch delay is not established. ## Risks - Low risk. The shared icon slot also affects other navigation rows. Existing badge, rail, and expanded-state tests cover the shared component. - Agent avatars become smaller in pinned and recent navigation, as requested. No data or API contracts change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected. This change is a scoped UI bug fix. ## Model Used - OpenAI GPT-6 via Codex. The host does not expose the exact served model snapshot or context window size. Used reasoning, code editing, shell execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
612cc7e3e6 |
fix(heartbeat): renew every legacy controller lease in one shared statement (#20924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy run holds a durable controller lease while it prepares. The server renews the lease every 10 seconds, and it stops the run if no renewal succeeds within the 60 second lease. > - Each run renewed in its own transaction on a private pool of two connections. Each renewal used six round trips (begin, statement timeout, transaction id, notify, update, commit) and woke live-run recovery for that run. > - When many runs prepare at the same time on a loaded database, the renewals queue behind each other. The queue grows with the number of runs, and every run loses its lease at about the same time. > - A healthy server must not fail its own runs because it has many of them. > - This pull request renews all leases of one server process through one shared queue. Requests that arrive while a renewal is in flight share the next statement. > - The benefit is that the renewal cost no longer grows with the number of live runs, so a burst cannot starve the lease. ## Linked Issues or Issue Description **What happened?** On a staging server that did not restart, 22 runs failed with `controller_lease_lost` in about 13 minutes. Every run was a legacy run in the `preparing` stage with `providerWorkStarted: false`. Every loss had cause `renewal_deadline`. The renewal had been pending 28 to 48 seconds (`renewalPendingMs`). The deadline callback ran on time (`deadlineLateMs` 3 to 861 ms), so the event loop was not blocked. The successful renewals before the loss had each taken 20 to 30 seconds, because the deadline fired only 25 to 48 seconds after the last renewal (`sinceRenewalMs`). About 30 runs were live in the process. The first wave started when a user retried about 14 blocked tasks at once. The automatic retries of the first wave then failed the same way, which made the second wave. 12 of 16 affected tasks became blocked. **Steps to reproduce** 1. Start about 30 legacy runs in one server process. 2. Make each renewal transaction slow (for example, a loaded database). 3. Wait one lease period. **Expected behavior** The leases stay owned. The renewal cost does not grow with the number of runs. **Actual behavior** All runs fail with `controller_lease_lost` (`renewal_deadline`). ## What Changed - `server/src/services/legacy-controller-lease.ts`: All controllers of one process renew through one queue per database. One flush is in flight at a time. Requests that arrive during a flush share the next flush, so a renewal waits behind at most one other statement. A flush is one transaction on the private lease pool: `statement_timeout` and one statement that renews all requested rows. - The statement locks the requested rows with `for update skip locked`. A row that another transaction holds is reported busy, and only that run retries after 250 ms. It does not stall the other runs in the batch. A row that is not this boot's live legacy lease is reported rejected, as before. - Renewal no longer emits the `active-execution-recovery` notification. A renewal only moves the deadline later. Recovery already has a deadline at the earlier expiry and rereads the moved deadline then. This removes one recovery pass, `pg_current_xact_id` and `pg_notify` per renewal. - The lease contract does not change: 60 second lease, 10 second renewal, boot-id fence, expired leases cannot renew, the local deadline is measured from the attempt start, and the loss causes and timings are the same. - `doc/run-log-events.md`: Documents the shared renewal queue. Upgrade story: no persisted shape changes. The row columns, lease length and boot-id fence are the same. A lease held by the previous release has a different boot id, so the new batch rejects it and leaves it untouched. A test covers a previous-release lease in the same batch. ## Verification - Red on `origin/master`, green with this change: - `keeps thirty preparing controllers owned when every renewal takes four seconds on a two-connection pool`: before, all 30 controllers lost the lease with `renewal_deadline` (the staging shape). After, none lose it. - `renews fifteen concurrent controllers in one statement` (real Postgres): before, 15 update statements. After, 1. - `renews through one private statement without waking live-run recovery`: before, the renewal woke `active-execution-recovery`. After, it does not. - `live-recovery-work.test.ts` `renews without waking recovery and re-reads the moved controller deadline at the old one`: before, 2 recovery passes per renewal. After, none, and recovery still moves to the new deadline at the old one. - Safety cases: a row lock on one run does not delay another run; a previous-release lease in the same batch stays untouched; a terminated lease connection is retried; existing deadline, retry, rejection and revocation tests pass. - `npx vitest run src/services/legacy-controller-lease.test.ts src/__tests__/live-recovery-work.test.ts`: 88 passed. - `pnpm verify`: see the CI checks on this head. ## Risks - Medium-low. All renewals of a process now share one statement. A failed flush fails every request in it; each watcher retries independently, as it did before. - A busy row is retried every 250 ms until the lease deadline. Before, the renewal waited on the row lock for up to the statement timeout. The loss behavior is the same. - Recovery wakes at the old lease expiry instead of after each renewal. It finds the live lease and moves to the new expiry. - Not in scope: the downstream workspace recovery that blocked tasks after a lost lease (`legacy_execution_requires_reconciliation`, `source_unavailable`). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
228b0169be |
fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question (`request_human_input`) and wait for the answer > - A later comment from the person moves an open question into history (#15392). The agent still needs the answer, so it reports a blocker and the task becomes Blocked > - The person then answers. Dispatch cancels the answer's continuation with `issue_not_in_progress`, because only a dependency-blocked task may take an answer while Blocked > - The task rests Blocked with a failed run and a Retry. The answer is lost until a person clicks Retry > - This pull request lets the answer continue the task when the same agent blocked it to wait for that answer, proven from durable records > - The benefit is that the person's answer always reaches the agent that asked, without a Retry ## Linked Issues or Issue Description Refs #15392, Refs #20872 **What happened?** Restart-chaos waves with no fault injected fail every W2Q task on every provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude task t03 on master |
||
|
DottaandClaude Fable 5.2
|
b01959cb15 |
fix(recovery): release a handed-off run's saved-files hold for the person it handed the task to (#20941)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can hand its own task to a person for review. With #20884, a native run that does this keeps running and finishes its turn. > - A restart can land between that hand-off and the end of the same turn. Restart recovery then settles the run as failed and keeps its unsaved sandbox files, so no work is lost. > - Automatic workspace recovery saves those files. But the workspace-repair hold then stayed on the person's task, because the hold only released for a newer agent owner. > - This pull request releases that hold for the person who received the task through the stopped run's own recorded hand-off, once the files are saved. > - The benefit is that the person's task rests in review with no stale "Saved work recovered" hold, and the agent's files are still saved first. ## Linked Issues or Issue Description Follow-up to #20884 (merged). It was reviewed there as a known limit: "Restart blocks the person's task". It uses the `resultJson.humanSelfHandoff` record that #20884 added. Related: #19184, which released the same hold for a newer agent owner. **What happened?** A native run handed its own task to a person, and a restart landed before the run finished its turn. Restart recovery settled the run and retained its files. After automatic recovery saved them, the task still had a workspace-repair hold ("Saved work recovered. Send a message to continue."). The person had nothing to do with that run's work, and nothing released the hold. **Expected behavior** The hold protects the workspace until the files are saved. After that, the task rests with the person and has no execution blocker. **Steps to reproduce** 1. A native run records a hand-off to a person, and the task is now `in_review` with that person. 2. A restart settles the run as failed, with its prepared remote workspace retained. 3. Automatic workspace recovery completes. 4. Before this fix, `getExecutionBlocker` still returns the run's workspace-repair hold. ## What Changed - `server/src/services/execution-blocker.ts`: `restoreHoldReleasedForCurrentOwnerPredicate` now also releases a recovered file-preservation hold in one more case. The task must be assigned to a person, and the stopped run's `resultJson.humanSelfHandoff` must name that task and that person. The existing proof is still required: the completed-recovery receipt bound to the action, and a copyback rather than a snapshot-only save. Until the files are saved, the hold stays. - `server/src/services/workspace-recovery-admission.test.ts`: two embedded-Postgres tests. - The hold blocks the person's task before recovery and releases after it. - A hand-off recorded for a different person keeps the hold. - `doc/execution-semantics.md`: one sentence on the restart case. ## Verification - Red: with the `execution-blocker.ts` change removed, `releases a handed-off run's recovered files hold for the person it handed the task to` fails. `getExecutionBlocker` still returns the hold after `recoverRetainedWorkspace`. - Green: `cd server && npx vitest run src/services/workspace-recovery-admission.test.ts src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/explicit-native-continuation.test.ts src/services/native-runtime/native-human-self-handoff.test.ts`. All 617 tests pass. The server typecheck passes. - Chaos: the H2 workload from #20884 covers the hand-off with a K1 restart after it. Locally the runs had no unsaved files, so no hold appeared. This PR's case needs a remote sandbox with unsaved files. **No Daytona wave was run.** ## Risks - The release applies only when the stopped run's own record names the current person and task, and only after the completed-recovery receipt. Other holds, snapshot-only saves and unsafe archives are unchanged. - No schema change. A run without the record behaves as before. - This branch is rebased onto master after #20884 merged. The only conflict was an adjacent import line. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude
|
0bf3236bc8 |
test(restart-chaos): staging class R cells CR1-CR3 and sandbox capture on every Daytona wave (#20909)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
48d23d1530 |
fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
884a9e19ea |
fix(native): launch a warm follow-up from its predecessor's saved journal, and restart a Runner that exited before session.open (#20928)
## Thinking Path - Paperclip runs agent work through a controller (server) and a runner (`paperclip-runnerd`) that exchange numbered commands; the runner refuses any command whose number is not exactly one past the last it processed. - Staging's most common failure is "Runner gone" (`runner_remote_process_identity_unavailable`, `native.remote_runner_recovery_failed` with `process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`). Chaos wave `w-20261111T192658-s5201` (Daytona, warm, Codex, W2/W2Q/W2C re-wake cells) captured it live: the new runner wrote `controller sequence must be contiguous: expected 13, received 11` (and `expected 5, received 3`) and exited 0.15 s after authenticating. - In all three captured failures the follow-up run started 1–2 s before the predecessor's warm session was closed (`transport_close_requested` for the predecessor arrives after the follow-up's `run started`). The close is issued by the follow-up's own admission. - Admission first materializes the controller projection for the session root. With an idle retained warm owner the root only holds the external-persistence marker, so `restoreWarmRunnerdController` copies that owner's saved checkpoint into the root. Later in the same admission, `executePaperclipNativeSessionWithinScope` finds the owner cannot be reused (configuration, process or GitHub access changed) and retires it. Retirement sends `runner.drain` and `runner.suspend`; both are saved to the owner's checkpoint before they are sent, but the root still holds the projection taken before the close. - The follow-up then opens its controller from that projection. Its journal is two commands behind the runner's durable `lastControllerCommandSeq`, so its first command repeats sequence 11; the runner treats this as fatal and exits, and every later attempt only reattaches to a process that is gone. - So the saved controller journal was persisted correctly; it was not selected: the projection is never refreshed after the retirement that extends it. ## Linked Issues or Issue Description - Refs #20874 (the runner-exit proof this extends from a rejected `session.open` to a Runner that exited before answering it) - Refs #20907 (makes the runnerd exit text, including `controller sequence must be contiguous`, visible in the run log; independent of this fix) - Refs #20853 (related: resumes a remote Runner that died at a settled `ready` cut; this PR covers the unsettled `ready` state left by an exit before `session.open`) - Duplicate search: searched open and closed PRs for "controller sequence contiguous", "runner_remote_process_identity_unavailable" and "Runner gone"; none fixes the stale controller projection. ## What Changed - `server/src/services/native-runtime/warm-controller-journal.ts` (new): - `refreshRetiredWarmOwnerProjection` reloads the retired owner's canonical checkpoint into the session root after retirement, only when the root still projects that owner and the saved journal extends the projected one. A diverged journal holds admission with `native_runner_checkpoint_projection_held:retired_owner_journal_diverged`; a projection changed underneath holds with `...:journal_changed`. - `reconcileControllerJournalWithRunner` is the defence for state already saved in the bad shape (old-saved/new-resumed, mid-run deploys). When the remote runner state shows its journal ahead of the controller projection and the runner's last command is a completed `runner.suspend`, it reconciles only if every extra sequence is a completed `command_close_drain_*`/`command_close_suspend_*` in the runner's processed-command journal and the predecessor's canonical checkpoint holds exactly those commands. Anything else refuses before launch with `native_controller_recovery_missing_evidence:controller_journal_behind_runner` instead of guessing. A runner whose last command is not a completed suspend is left to the existing live-adoption checks. - Defence in depth for the server side of the same failure (`native_session_recovery_failed: runnerd exited while waiting for session.open`, then `runner_remote_process_identity_unavailable` on every same-run retry): a Runner proven to have exited while this attempt's opening command (`session.open`, or the recovery probe) was still unanswered is a failed fresh start, not a live Runner to reattach. - `runnerd-codex-transport.ts`: `runnerExitConfirmed()` (#20874's proof) now also holds for a resumed session whose provider identity was preloaded, when the attempt's opening command is still pending at the proven exit. - The Codex driver's recovery result, `HarnessDriverBackend` and `native-session-runtime` carry that proof on the `native_session_recovery_failed` error. - The executor records `runnerExitedBeforeSessionOpen` on the settled failure; restart recovery puts `exitedBeforeSessionOpen` on the `reattach_remote_runner` claim; `verifyRemoteRunnerOrSettledExit` and `verifyStagedRemoteRunner` then start a new Runner when the process is gone and the saved Runner state has no pending terminal delivery, provider cleanup or warm transition (any lifecycle other than an active turn). Attempt budget and identity checks are unchanged. - `native-session-executor.ts`: calls the refresh right after the configuration-changed retirement of a warm owner, and the reconciliation in `restoreRemoteSuspendedRunnerCheckpoint` with the runner state it already reads (no added remote read, no wait before `turn.start`). - Tests: - `native-session-executor.test.ts`: new `changed-external-checkpoint` scenario of "rotates an owned warm codex remote runner before checking changed configuration" with a real `DurablePrpControlPlane`: the retired owner's close saves drain/suspend, and the journal the follow-up launches from must end with them. - `warm-controller-journal.test.ts`: refresh, divergence hold, untouched marker/other run, reconcile of a settled close, refusal when the canonical journal cannot prove the close, refusal for any other extra command, and no action for a live runner. - `packages/paperclip-runner/src/live/runnerd-stale-controller-journal.test.ts`: real `paperclip-runnerd` + fake Codex. A follow-up controller on the predecessor's suspended runner with a journal that lacks the close commands reproduces `controller sequence must be contiguous` and loses the runner; with the current journal it resumes. ## Verification - Red for the added part: `verifyRemoteRunnerOrSettledExit` with the staging state (process gone, lifecycle `ready`, unsettled outbox) rejects with `runner_remote_process_identity_unavailable` without the change; with `exitedBeforeSessionOpen` it resolves to a fresh start, and stays fail-closed for a pending cleanup, terminal delivery, active turn or exhausted budget. Without the propagation, the backend recovery result lacks `runnerExitConfirmed`; the real-runnerd stale-journal test asserts the transport's exit proof after the contiguity exit (`expected false to be true` on the old transport). - Red on master (executor unchanged, new scenario): `AssertionError: expected [] to deeply equal [ 'runner.drain', 'runner.suspend' ]` — the follow-up launched from the projection taken before the predecessor's close. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "rotates an owned warm codex remote runner before checking changed configuration"` (14 passed), `npx vitest run src/services/native-runtime/warm-controller-journal.test.ts` (7 passed). - Real runner: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-stale-controller-journal.test.ts` (stale journal → `controller sequence must be contiguous`; current journal → resumes). - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript`; `pnpm verify` selected broader verification (a new runner test file is outside the resolved graphs), so broad coverage comes from required CI. ## Risks - The refresh replaces the root projection only with a checkpoint whose identity matches and whose journal extends the projection; otherwise it holds admission with an existing `native_runner_checkpoint_projection_held:*` code, which is safer than launching a runner that will exit. - The defence refuses launch when a runner is ahead by anything other than a provable settled close. Before this change that launch always ended in a runner exit and a failed run, so the refusal only changes the error code and the moment of failure. - This PR does not change the runner. The runner still exits on an out-of-order new command; a follow-up runner PR will turn that into a rejected command result. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
edd00526e7 |
fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude
|
80b2369f26 |
fix(native): keep remote runnerd exit evidence and report it on failed recovery (#20907)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a detached `paperclip-runnerd` in the sandbox, and the controller adopts it after a restart > - On staging, recovery fails with `runner_remote_process_identity_unavailable` when the Runner is gone, but nobody can see why it exited > - The launch script sends runnerd's stdout and stderr to `/dev/null`, so only a panic or a graceful error reaches `runnerd.stderr.log`, and the recovery event does not read that file > - This pull request keeps runnerd's stderr in a bounded file in its private diagnostics directory, and adds the exit reason to the `native.remote_runner_recovery_failed` event > - The benefit is that the reason a Runner exited shows in the run log on staging without SSH, which the class R investigation needs ## Linked Issues or Issue Description **What happened?** Staging runs fail with `runner_remote_process_identity_unavailable` after the remote Runner exits (`native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`). The event does not say why the Runner exited. **What did you expect?** The run log names the Runner's exit reason: its durable `recoverableFailure` and diagnostics, and what it wrote to stderr. **Steps to reproduce** Restart-chaos re-wake and stall cells reproduce the refusal. Each failed run log shows only `runnerLifecycle` and `runnerSettled`. Refs #20893 ## What Changed - `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to `diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700 directory) instead of `/dev/null`. - At launch, a file above 1 MiB is rotated once to `.1`. - While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB, it keeps the last 1 MiB. It checks every 15 s and ends when runnerd ends. It is double-forked with its own argv, so it is not a child of runnerd and does not carry runnerd's arguments. runnerd's own exit reason goes to `runnerd.stderr.log`, which is never trimmed. - A symlink or non-file at that path keeps the old `/dev/null` behavior. - Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch shell's PID, so the identity marker and fingerprint are unchanged. - `remote-runner-recovery-diagnostics.ts` has a read-only `REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of `runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the session diagnostics directory first, then the shared runtime-root directory used by external Runners (Dot, Muse, grok_bot). It never reads through a symlink. Each file keeps its own share of the 4 KiB budget. - When recovery's process verification finds `process_not_running`, the diagnostic adds these fields: - `runnerRecoverableFailure`: a code-shaped string from `runner-state.json`; - `runnerDiagnostics`: the last 4 entries, redacted, 500 characters each; - `runnerExitEvidence`: the redacted file tails, 4 KiB at most. `native.remote_runner_recovery_failed` carries them. - The live process monitor's exit report now puts the durable state summary first, then the file tails. Before, it used one or the other. - `run-failure-diagnostics.ts` is unchanged. It still copies only the closed labels, so no free text reaches failure reports. ## Verification - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`: the launch writer keeps stderr in the file, writes the identity marker with the child's PID, appends across relaunches, rotates an oversized file, trims a growing file while the child runs, and never writes through a symlink. - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`: the evidence script output is bounded and refuses symlinks. Redaction and bounds hold, and the new fields survive the event payload. - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts` passes (1374 tests). One expectation was updated for the combined exit report. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Older runner images: the script is server-side. It adds only `wc`, `tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed `node`, `date`, `chmod`, `mv` and `printf`, so any image that launched runnerd before has these tools. The runner content ID is unchanged, so the current Daytona image runs it unmodified. The restart-chaos waves run on that image. ## Risks - Low. A sandbox disk that is already full could make the stderr redirect fail. `exec ... 2>>file` then fails, and the launch reports `runner_remote_process_identity_unavailable` the same way a launch that cannot write its identity marker does. The marker is written first, in the same directory tree. - Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes into the instance's own run log (run-log data path; not Telemetry and not Observability). - No wait is added before `turn.start`. The evidence read runs only on the recovery failure path, with its own 10 s bound. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
b2779054d4 |
fix(issues): let a native run hand its own task to a person without cancelling itself (#20884)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task has one assignee. An agent often finishes its work by handing its own task to a person for review. > - The task update route stops the outgoing run before it saves a reassignment. > - Since #16082, a legacy run that hands its own task to a person is exempt, so it finishes its turn. Native runs were not exempt. > - On staging, a native run (ACPX Claude, 11.5 minutes of work) sent `PATCH /api/issues/{id}` to hand its task to a person. The route cancelled that run with `issue_reassigned`. The agent saw `api_transport_failure`. The task moved to the person, but it kept a stale execution blocker ("Saved work recovered. Send a message to continue."). > - The native runtime assumes a running run still owns its task. So widening the exemption alone is not enough: the run then fails after its result is accepted, and its finalizer can act on the person's task. > - This pull request applies the exemption to native runs and makes the native terminal fence, restart recovery and finalizer handle a run that gave its task away. > - The benefit is that an agent can hand its task to a person and finish normally, with no blocker on the person's task. ## Linked Issues or Issue Description Refs #16082 (legacy exemption). Related: #20541 (agent-to-agent self handoff) and #18293 (agent key handoff). Those PRs cover other actors and targets. This PR covers native runs that hand their task to a person. **What happened?** A native agent run handed its own task to a person (`assigneeAgentId: null`, `assigneeUserId`, status `in_review`) through its signed run token. The route cancelled the run that sent the request (`issue_reassigned`, "Cancelled before issue reassignment"). The cancelled native run then went through workspace recovery. The person's task kept an execution blocker with `restore_failed`. **Expected behavior** The run that hands its own task to a person keeps running and finishes its turn. The task rests `in_review` with the person. No execution blocker or recovery action stays on the task. The outgoing run cannot change the task after the hand-off. **Steps to reproduce** 1. Run a native agent on its assigned task. 2. From that run, send `PATCH /api/issues/{id}` with `assigneeAgentId: null`, a person's `assigneeUserId`, and status `in_review`. 3. Before this fix, the route cancels the run with `issue_reassigned`. **Why the exemption was legacy-only** #16082 limited the exemption to the run shape of its incident and kept a verified stop for every other case. The new H2 chaos workload showed why native needed more than the route change. With only the route widened: 1. The run's terminal fence `assertCurrentWakeCommentsRead` requires the run to still own the task. It threw `native_current_wake_comments_binding_changed` after the result was accepted. The run then failed with `native_session_lease_lost`, and its finalizer recorded `workspace_finalization_failed` as an agent-owned recovery on the person's task. 2. A restart after the hand-off left the run `running`, because restart recovery refused it with `issue_execution_lock_changed`. 3. The native finalizer only superseded a `done` decision. A `needs_review` report moved the person's task back to `in_progress` and queued a continuation for the outgoing agent. 4. The REST guard against the outgoing run's later mutations (`issue_handed_off`) applied to legacy runs only. ## What Changed - `server/src/routes/issues.ts`: the signed current-run human self-handoff exemption now applies to native runs. All other checks stay: exact run, agent, company and task, `agent_jwt` source, human recipient. For a native run the route first records the hand-off on the run (`resultJson.humanSelfHandoff`, new `native-human-self-handoff.ts`). Board reassignment, agent keys, and handoffs to an agent or to nobody still stop the outgoing run. - `current-wake-comments.ts`: the terminal fence accepts a run whose recorded hand-off matches the person who now owns the task. Without the record (for example a board reassignment), or after a later reassignment, it still refuses. - `native-restart-recovery.ts`: after a restart, a handed-off run is not resumed. It passes the execution-lock check and is then settled by the existing ownership gate (`task_ownership_or_status_changed`) without changing the person's task, instead of staying `running`. - `status-arbiter.ts` and `native-run-finalizer.ts`: `supersedeHandedOffDecision` keeps the person's task status and strips retries, continuations, owner wakes, recoveries, blockers and reviewer bindings, also from decisions that already preserve the status. Two effects stay: a governed request's interaction, and the workspace-finalization repair, so saved work is not lost. Review runs are not affected. - `server/src/run-task-ownership.ts` (renamed from `legacy-run-task-ownership.ts`): `assertHandedOffRunTaskOwnership` rejects later task mutations from the outgoing legacy or native run, in the route and under the update transaction's locks. - `tests/restart-chaos/`: new opt-in workload `H2`. The agent saves a brief, then hands the task to its creator in review as its last step. The oracle requires the task to rest `in_review` with the creator, every run to settle without `issue_reassigned`, readable non-empty run history, and no execution blocker or active recovery action. - Docs: `doc/execution-semantics.md`, task feature map and agent access map, chaos README. ## Verification - Red on `origin/master` (same tests, master source): the route cancels the native run (500, and 409 `runner_goal_reassignment_stop_unconfirmed`); the finalizer moves the person's task to `in_progress` with a continuation for the outgoing agent; a late `done` from the outgoing native run closes the person's task. - Red chaos wave on master's server (`--server-from` a master checkout, `--adapters claude --workloads H2 --faults K1 --matrix`): 0/2 pass. Both runs ended `cancelled` / `issue_reassigned` (`T:handoff_did_not_cancel_run`). - Green focused tests on this branch: - `native-human-self-handoff.test.ts` (embedded Postgres): hand-off through the real issue service; the run stays `running`; `done`, `needs_review` and failed-turn reports leave the task `in_review` with the person, with `getExecutionBlocker` null and no recovery action; the terminal fence accepts only the recorded hand-off; a late change from the outgoing run fails with `issue_handed_off`; pure policy cases. - `issue-agent-mutation-ownership-routes.test.ts`: native and legacy self-handoffs persist without `cancelRun`, and native records the hand-off before the update. Board, agent key, other run, task, agent and company still stop the run. - Native runtime suite (restart recovery, wake comments, arbiter, stale-owner completion), `issues-service`, `low-trust-red-team-routes`: pass. Server and restart-chaos typecheck pass. `node scripts/check-maps.mjs` passes. - Green chaos waves on this branch (local, `--adapters claude --workloads H2`): - `--matrix --faults K1 --seed 4459`: 2/2 pass (K0 baseline and a K1 crossing). - `--tasks 3 --faults K1 --fault-after-seconds 60` and `--fault-after-seconds 100`: 3/3 pass each. In the first, all three restarts landed after the hand-off while the runs were live; the runs settled and the tasks stayed with the person. - The final waves ran on the pre-rebase head `8a2a3b6a74`. The rebase onto master only merged the chaos workload lists (`types.ts`, `config.ts`, `journeys.ts`). No Daytona wave was run. ## Risks - The outgoing native turn keeps running after the person owns the task. Its native task tools already require the run to own the task, so they refuse (`paperclip_runner_tool_binding_not_authorized`). Its REST mutations are rejected with `issue_handed_off`. - A restart after the hand-off settles the run as failed (`native_restart_recovery_blocked`) instead of resuming it. The person's task is not changed and needs no Retry. - The finalizer rule covers any native run whose task belongs to a person at finalization, not only a self-handoff. Before this change such a run could still move the person's task and queue its own continuation. - Saved state: `resultJson.humanSelfHandoff` is a new optional run key. An older server ignores it, so a run handed off on the new server and finished on an old one fails its fence as today. No schema change. - No API shape change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
d7b04a8b6d |
refactor(runner): rename codex-runtime-adapter to acpx-runtime-adapter (#20938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner opens agent sessions through one ACPX runtime adapter > - The adapter began as the Codex ACPX adapter, so its file is `codex-runtime-adapter.ts` > - The adapter now opens sessions for every ACPX agent: Claude, Codex, Pi, Cursor, Grok, Kimi, Copilot and Antigravity > - The Codex name makes readers look for Claude and other agent behavior in the wrong place > - This pull request renames the module to `acpx-runtime-adapter.ts` and changes no behavior > - The benefit is a file name that tells readers what the module does ## Linked Issues or Issue Description No issue exists. Refs #20933, which adds Claude restore behavior to this module. **What existing behavior does this improve?** The name of the shared ACPX runtime adapter module in Paperclip Runner. **Subsystem affected** Paperclip Runner, ACPX driver (`packages/paperclip-runner/src/drivers/acpx/`). **Current behavior** The module that opens ACPX sessions for all agents is named `codex-runtime-adapter.ts`. Its test file, which holds Claude, Pi, Grok, Cursor and Copilot tests, is named `codex-runtime-adapter.test.ts`. **Proposed behavior** The module is `acpx-runtime-adapter.ts` and its test file is `acpx-runtime-adapter.test.ts`. All imports, scripts and docs use the new name. **Reason and benefit** The name matches the module's scope. Reviewers and contributors find agent-specific session behavior faster. **Breaking changes** None. No exported name, behavior or contract changes. ## What Changed - Rename `src/drivers/acpx/codex-runtime-adapter.ts` to `src/drivers/acpx/acpx-runtime-adapter.ts`, and its test file in the same way. Git records both as renames with no content change, except the test file's import path. - Update the imports in `acpx-runtime-sidecar.ts`, `codex-acpx-driver.ts` and four test files. - Update the `dist` import in `scripts/grok-native-smoke.mjs` and the test file key in `scripts/vitest-lane-durations.json`. - Update the links and file names in `doc/RUNNER-INTEGRATION-PLAYBOOK.md`, `doc/architecture/runner-cursor-capabilities.md` and `doc/architecture/runner-rich-acp-capabilities.md`. - Do not change the historical profile identity fixtures under `packages/paperclip-runner/test/fixtures/`. They record the sources of earlier releases with the old path, and tests check them by hash. ## Verification - `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`: no errors in changed files. The only errors are in `src/eval/workflow-harness.ts`, because `@paperclipai/paperclip-eval-kernel` was not built in the local worktree. - `npx vitest run src/drivers/acpx src/cli/acpx-runtime-sidecar.test.ts`: 93 files, 1859 passed, 9 skipped. - `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11 passed. The pack contract digest reads protocol schemas, the Rust crate and the ACPX patch, not TypeScript sources, so the rename does not change it. - `node --check scripts/grok-native-smoke.mjs`, and `scripts/vitest-lane-durations.json` parses. - `git grep codex-runtime-adapter` finds the old name only in the historical fixtures. - Merge check with #20933: a local merge of that branch into this one completed with no conflicts. Git applied #20933's changes to the renamed files. The merged adapter tests passed (133 passed). ## Risks - Low risk. The change renames files and updates references only. - An open PR that edits the old file path must merge or rebase after this PR. Git rename detection handled #20933 with no conflicts. A PR that adds a new import of `./codex-runtime-adapter.js` fails typecheck until it uses the new path. - `openCodexAcpxRuntime` keeps its name. It is an alias of `openQualifiedAcpxRuntime`, and this PR does not change exported names. - `acpx-runtime-adapter.ts` and its test file were already longer than 1,000 lines. This PR does not add lines to them. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1111.0-canary.14 |
||
|
DottaandClaude Opus 5.5
|
7cd5558e41 |
fix(authz): send agents denied a cost-recording route to the Costs page (#20922)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent is denied a board-only action, the denial names the
board page where a person can do it, and agents relay that link
> - Cost recording retry ("Retry cost recording") is board-only and
lives on the run's row on the Costs page
> - The denial for every `/companies/:id/costs` route named the Budgets
page instead
> - So an agent asked to fix a run's missing cost sent the person to
Budgets, where the control does not exist
> - This pull request maps cost routes to the Costs page and names that
page in the retry route's catalog entry
> - The benefit is that the person lands on the control the agent told
them to use
## Linked Issues or Issue Description
**What happened**
In paid chat eval run 43086418488
(`budgets-costs.repair-cost-recording`), the CEO tried `POST
/api/companies/{companyId}/costs/retry-accounting`, got `403
board_access_required` whose remediation said `[Open
budgets](/<prefix>/activity/budgets)`, and told the person to use the
retry on Budgets. The retry control is on the Costs page.
**Expected behavior**
Board denials for cost routes name `[Open
costs](/<prefix>/activity/costs)`. Budget and budget-incident routes
still name Budgets.
**Steps to reproduce**
1. As an agent, call `POST
/api/companies/{companyId}/costs/retry-accounting`.
2. Read the 403 `remediation`.
3. It links `/<prefix>/activity/budgets`, a page with no cost retry
control.
## What Changed
- `server/src/routes/authz.ts`: `COMPANY_COSTS_PAGE`
(`/<prefix>/activity/costs`); `/companies/:id/costs/*` denials use it.
Budgets and budget incidents keep `COMPANY_BUDGETS_PAGE`.
- `server/src/routes/openapi.ts`: the retry route's description says a
board user retries with Retry cost recording on the run's row on the
Costs page.
- `board-access-denial.test.ts`: regression row for the retry route (red
before the change, green after).
## Verification
- `pnpm exec vitest run src/__tests__/board-access-denial.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/agent-refusal-finality.test.ts
src/__tests__/cost-data-quality-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (server): all pass. The
new row fails without the `authz.ts` change.
- End-to-end: the chat eval case `budgets-costs.repair-cost-recording`
grades that the agent's guidance links the Costs page; it will be rerun
after this merges.
## Risks
- Low. Only the link text and target in cost-route denials and one
catalog description change. Agents that previously relayed Budgets now
relay Costs.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c1edb9ec02 |
fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - On staging, `runner_remote_process_identity_unavailable` was the most frequent failure. The tasks stayed blocked until someone clicked Retry > - In 31 of the 34 failed runs the reattachment found the Runner gone with an unsettled `ready` journal (`runnerSettled: false`). The Runner was not lost by accident. The controller's own close stopped it with `SIGKILL`: close could not prove suspension within its 10 s budget, so it force-stopped the Runner in the middle of a turn > - The server then scheduled a same-run reattachment that could never succeed, because the close had just removed the Runner it needed > - This pull request keeps an unreachable remote Runner alive when a failed run closes. A Stop, a cancelled turn and a `stop_cleanup` recovery still stop it > - The benefit is that the same-run reattachment finds the live Runner. The Runner then settles into a `suspended` journal, which the existing relaunch path resumes. The task continues without Retry, and no provider work is replayed ## Linked Issues or Issue Description Refs #20853, #20907 **What happened?** Remote native runs failed with `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. The tasks were left blocked. In the failed runs the first attempt failed with a recoverable transport error, for example `provider_transport_failed: PRP command turn.start timed out`, `native_session_interrupted` or a server restart. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The second string comes from `waitForProcess`, which sends `SIGKILL` when its deadline passes. The same-run reattachment ran 20–60 s later and reported `process_verification` / `process_not_running` with `runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs had done provider work before the failure (turns and tool calls), so the automatic Retry correctly refused them (`provider_turn_unsettled`). **What did you expect to happen?** When a run fails for a reason that the server resumes on the same run, the close does not destroy the Runner that the reattachment needs. The reattachment adopts it, and the run continues without Retry and without a repeated tool effect. **Steps to reproduce** 1. Start a remote native run and let a turn start. 2. Make the Runner unreachable (a broken transport or a slow event pump), so that the turn fails and the close cannot prove suspension within its budget. 3. The close sends `SIGKILL` to the Runner. The same-run reattachment then fails with `runner_remote_process_identity_unavailable`, and the task is blocked. **Relation to open PRs** - #20853 covers the 3 runs whose journal was a settled `ready` cut (`runnerSettled: true`). This PR does not change that path. - #20907 makes the Runner's exit reason visible. It does not change recovery. This PR does not overlap with it. ## What Changed - `runnerd-codex-transport.ts`: `close(reason, { retainUnsettledRemoteRunner })` accepts a new option. When the option is set, the Runner is remote (a process launcher or an adopted Runner), the close is not a `stop_cleanup`, the Runner is still alive and suspension is not proven, the close does not stop the Runner. It skips the `waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It still releases the controller route, writes the unsettled checkpoint and throws `NativeSessionCloseUnrecoverableError`, so the session stays quarantined and is never reused. The settlement diagnostic records `runnerRetained: true`, and a new diagnostic records `remote runner left alive unsettled for same-run reattachment`. - `HarnessNativeSession.close` asks for retention only when the session was not explicitly cancelled. After `cancel()` (Stop), it asks for the old behavior, so Stop still stops the Runner. - `CodexHarnessSession.close` passes the option to the transport. `HarnessSession.close` and `CodexAppServerTransport.close` accept the optional field. Callers that do not set it keep the old behavior. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md` describe the rule and the `runnerRetained` field. How the run continues: the reattaching controller adopts the live Runner. The Runner then completes the `runner.drain` and `runner.suspend` that the failed close queued, and exits with a settled `suspended` journal. That attempt fails as `native_runner_process_exited`, which the server resumes on the same run. The next reattachment finds the process gone with a settled `suspended` journal and relaunches the Runner over it (the existing path from #19745). If the Runner was only slow, it suspends by itself and the first reattachment relaunches. ## Verification - Red, on `master` with only the tests added: `runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5 (`expected 'gone' to be 'alive'`): the close killed the Runner. `harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2. - Green with the fix: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-remote-retention.test.ts src/backends/harness-driver-backend.test.ts`. - The staging shape: a real `paperclip-runnerd` started through a process launcher, a turn in progress, the Runner frozen with `SIGSTOP` during close. The close throws `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`, the Runner stays alive, and its journal stays `ready`. - A second controller attaches through `adoptExistingRunner` (no duplicate launch). The Runner settles into `lifecycle: suspended` with an empty outbox and no pending terminal delivery, then exits. - An explicit cancellation, a caller that does not ask for retention, and a `stop_cleanup` close still stop the Runner. - Also run: `src/backends`, `src/drivers/codex`, `src/drivers/acpx/codex-acpx-driver.test.ts`, `src/drivers/acpx/codex-runtime-adapter.test.ts`, `src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/` (files that timed out under full-parallel load pass at `--maxWorkers=2`). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The fix is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Medium. A failed close now leaves an unreachable remote Runner alive instead of killing it. This is the same state a restart detach leaves today, and the same-run reattachment or a Stop cleanup owns it. - If the server does not resume the run on the same Runner, the Runner stays alive until something reattaches or the sandbox is released. Remote Runners have no reconnect deadline (`runnerProcessReconnectGraceMs: null`). Before this change they were killed. The close still quarantines the session, so it is never released for reuse. - Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and `stop_cleanup` close without retention, so they still stop the Runner. - Previous-release story: this change adds and changes no persisted state. State that an older release left (a killed Runner with an unsettled `ready` journal) keeps its current handling: the named refusal `runner_remote_process_identity_unavailable` and a working Retry. Tasks already blocked on staging keep their Retry button. A deploy during a run changes nothing, because the decision is made in the closing process. - No wait is added before `turn.start`. The change only affects the failure path of close. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
e8e548e856 |
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
fc14f1ade2 |
fix(execution): keep resumed native runs alive past a stale finalization deadline (#20920)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The execution control reconciler bounds how long a run may spend
settling after its provider work ends
> - A run deadline from one attempt stayed on the run row after the run
was handed to a same-run resume or a restart claim
> - The reconciler then failed the resumed, live run with
`execution_finalization_deadline_exceeded` and told a person to inspect
the provider
> - This breaks two product rules: recovery must work without a person,
and users must not get operator instructions
> - This pull request makes a live native owner or a scheduled same-run
resume the owner of the run, and it replaces the operator text with
plain wording and Retry
> - The benefit is that runs that recover after a restart keep running,
and a genuine expiry shows plain words and one working Retry
## Linked Issues or Issue Description
No public issue exists. This is the bug description, in the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On a hosted deploy, 11 tasks became blocked with
`execution_finalization_deadline_exceeded` in two hours. Three runs had
the same sequence. The native session failed or the server restarted. A
same-run resume or a restart claim then took the run. The new controller
held a live native lease and prepared the run again. The reconciler then
found the earlier attempt's run deadline and failed the run. The
recorded original failure code was empty, so no real failure caused the
block. The run error told a person to inspect the failed run and verify
that its provider stopped.
**Expected behavior**
A run that a live controller owns, or that has a scheduled same-run
resume, continues. A deadline that really expires shows plain wording
and a working Retry. Users do not get operator instructions.
**Steps to reproduce**
1. Save a running native run with an `executionControlDeadlineAt` in the
past.
2. Give its finalization row a live lease (a restart claim), or phase
`retryable_failure` with no lease (a scheduled resume).
3. Run `reconcileAbandonedExecutionControl`.
4. The run fails with `execution_finalization_deadline_exceeded`, and
its error tells a person to inspect the provider.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
64afd22d82 |
fix(runner): keep the OpenCode MCP bridge on keep-alive so OpenCode keeps its Paperclip tools (#20921)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode agents call Paperclip tools through a local MCP
bridge in the runner's OpenCode proxy
> - #20886 made the bridge send `Connection: close` on every response,
as hardening for one stalled parallel tool call that was never
reproduced
> - After #20886 merged, many native OpenCode runs started without any
Paperclip MCP tools
> - Agents then said the Paperclip tools were missing, worked only with
bash and file tools, and never called paperclip_finish
> - This pull request removes the `Connection: close` header and keeps
the delivery diagnostics from #20886
> - The benefit is that OpenCode agents get their Paperclip tools again,
and a recurrence of the original stall still shows in the run log
## Linked Issues or Issue Description
Refs #20886.
**What happened**
In chat eval runs on CI, every OpenCode session before #20886 sent a
first model request of about 34k–36k input tokens (prompt plus the
Paperclip tool schemas). After #20886, about 9 of 25 sessions sent only
about 8k–15k tokens on the same prompts (for example 14,711 for
questions-and-approvals.necessary-question, 15,481 for
cases.cases-disabled, 14,096 for cases.update-case-facts). The prompt
envelope size did not change (about 32.7k characters). Those sessions
made no Paperclip MCP calls: zero tool-delivery records, no
paperclip_finish. The agents' own reasoning said "The paperclip tools
aren't in my toolset". Sessions on a commit without #20886 (
|