mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
c67909cb0f1fa973e0991ba068d9c8b2f8f64924
7207
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
c67909cb0f |
fix: keep tool approvals valid through OAuth refresh and delayed review (#20340)
## Thinking Path > - Paperclip lets people manage agent work and approve app actions. > - An approval must bind the action that the person reviewed. > - OAuth access tokens can expire while that person is away. > - The old target snapshot treated each token refresh as a new authorization. It also imposed a one-hour review deadline. > - This pull request separates token renewal from authorization identity and keeps the human decision durable. > - Current access, the reviewed arguments, meaningful target changes, task closure, and execution receipts still control dispatch. > - People can approve later without repeating a decision after routine connection maintenance. ## Linked Issues or Issue Description **What happened?** A pending app action returned `approved_tool_target_changed` after its OAuth tokens refreshed. The tool and connection settings were unchanged. The same card also expired after one hour. A sign-in problem could discard an accepted decision before provider dispatch. **Expected behavior** The original card remains reviewable after routine OAuth refresh and a server restart. A later approval runs the exact reviewed action once. A connection repair preserves the decision while Paperclip verifies the same authorization. Meaningful changes require a new review. **Steps to reproduce** 1. Connect an OAuth MCP app and configure an action to require approval. 2. Request the action, then refresh its access and refresh tokens before approving. 3. Approve the original card. Before this change, it returns HTTP 409 even when the authorization is unchanged. 4. Wait more than one hour. Before this change, the card expires without a task-specific deadline. **Paperclip version or commit** Reproduced from master at `76254ead2229`. The final branch is rebased onto current master. ## What Changed - Add versioned approval target comparison. New snapshots bind credential identity epochs, consent, permissions, ownership, connection configuration, tool contracts, and credential references. Authenticated OAuth refresh preserves identity. Manual credential replacement does not. - Remove the implicit one-hour deadline from ordinary app action reviews. Keep explicit deadlines and terminal-task expiry. Synchronize the nullable interaction deadline in shared contracts and UI. - Keep accepted, undispatched actions when sign-in is required. Show a saved-approval repair state. Resume on committed connection changes after target and access revalidation, without an idle retry loop. - Preserve the existing execution claim, result receipt, and uncertain-outcome no-replay rules. - Explain meaningful target changes without inviting the person to retry the stale card. - Add a production task-thread Storybook example and document compatibility and recovery behavior. ## Verification - Red: production-refresher invalid-grant handling and formal-approval withdrawal during a sign-in wait also reproduced as failing regressions before their fixes. - Red: before implementation, the local OAuth HTTP fixture failed for both public and confidential clients with `approved_tool_target_changed` after token refresh. - Green: the extended fixture passes for public and confidential clients with personal and shared grants. It covers actual local HTTP token exchanges, multiple refreshes, a rebuilt gateway, an ended originating run, approval after 25 hours, duplicate approval, preserved sharing arguments, sign-in waiting, and event-driven repair. The gateways use the production OAuth refresher; an invalid refresh preserves the accepted decision before provider dispatch. Omitted-scope responses cover equivalent scope-reporting changes. The provider is an isolated local fixture, not a deployed Notebook service. - Negative controls refuse changed consent, scopes, tool definition, destination, and manual credential replacement before provider execution. - Existing gateway service, connected-tool acceptance, secret owner, secret service, unsigned-review queue, and connection-health suites passed. Explicit-deadline fixtures still prove expiry behavior, including deadlines on accepted sign-in waits. A withdrawn formal approval cancels a waiting action before connection recovery. - Approval card, task-chat, error-copy and target-comparison tests passed. Server and UI typechecks passed. Token gates and Storybook catalog checks passed. - Inspected the new saved-approval repair card in local Storybook using the production task thread. It shows a reconnect link and no second approval button or false running state. - `pnpm verify --explain` returned exit 2 and requested broad CI coverage. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507) passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build, typecheck/release registry, general and serialized tests, and all eight browser shards. - Greptile reviewed the same final head at 5/5. Both findings from the first review are fixed and their threads resolved; no open follow-ups remain. ## Risks - No database migration is needed. The new comparison reuses the vault identity epoch. Only authenticated OAuth refresh code can preserve that epoch; HTTP credential-write schemas cannot select that behavior. - Earlier signed requests keep their original exact-version checks and saved deadlines. Already-expired cards are not revived. A historical card whose credentials already changed needs a fresh review. - A new consent, account, permission set, destination, or manually replaced credential must not inherit the old authorization. Reconnection continues automatically only when revalidation proves the same saved authorization. - The interaction enrichment now permits a null deadline. Existing string deadlines remain valid. The web client and shared validator change together. - This PR does not deploy to staging or perform a live Notebook publication. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e325e8d52 |
fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a67b936885 |
fix(server): redact invitation bearer paths from HTTP logs (#20343)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People join a company through an invitation link. The link carries a bearer token in its URL path. > - The HTTP logger removes query secrets and URL user information. It keeps the path. > - So invitation bearer tokens reach the HTTP log in the request URL, the log message, and the referrer header. > - PR #17848 fixed this, but it did not merge. Since then, #13742 changed the same redactor file. > - This pull request carries the #17848 commits onto current master. The original author is kept on both commits. > - The benefit is that invitation tokens do not stay in HTTP logs. The chat eval for membership management can then accept a real invitation safely. ## Linked Issues or Issue Description Supersedes #17848 (same commits, rebased onto current master after #13742). **What happened?** The HTTP logger writes invitation bearer tokens to logs. A synthetic POST to `/api/invites/<token>/accept` puts the token in `req.url`, in the log message, and in the `Referer`/`Referrer` header record. **Expected behavior** Logs keep the invitation route, the method, and the HTTP status. Logs do not keep the bearer token. **Steps to reproduce** 1. Check out master. 2. Run `server/src/__tests__/logger-invitation-redaction.test.ts` from this PR. 3. The synthetic token appears in the request URL, the message, and the referrer. 12 tests fail on master. **Paperclip version or commit** Master at the merge base of this branch. **Deployment mode** Source server. The tests use a disposable local HTTP listener. They use no database and no provider. ## What Changed - Remove the bearer segment from `/api/invites/<token>/...`, `/invites/<token>/...`, and `/invite/<token>` URLs, including mounted paths and absolute URLs. - Apply the same URL policy to `referer` and `referrer` request headers. - Keep record IDs on the non-token revoke route `/invites/<inviteId>/revoke` and on company-scoped invite routes. - Add regression tests through the real HTTP logger for 202, 400 and 500 responses, with both referrer header spellings. - Original work by @cryppadotta in #17848. This PR only rebases it onto master. ## Verification - From `server`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/__tests__/logger-invitation-redaction.test.ts src/__tests__/redact-sensitive.test.ts src/__tests__/http-log-policy.test.ts` — 46 passed. - With the master `logger.ts` and `redact-sensitive.ts`, the same new tests fail: 12 failed. - `http-log-aborted-response`, `cloud-principal-log` and `setup-token-session` tests pass locally. Some other HTTP-log suites could not load locally because workspace package builds were missing in this worktree. CI covers them. ## Risks - Low risk. Referrer query and fragment values are now removed from request logs, as they already are for request URLs. This removes some diagnostic detail to protect secrets. - A route that uses `/invites/<id>/...` for a non-bearer record ID, other than `revoke`, now shows `[REDACTED]` in logs for that segment. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution, used to rebase and verify. The original change was authored in #17848. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
79bf733012 |
fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner keeps a lifetime fence until the original ACPX provider stops. > - A recovered Runner must prove that fence is free before it clears saved cleanup state. > - Suspension and other recovery paths already wait for transient listener holders. Recovered shutdown used one immediate probe. > - That difference made parallel Rust tests fail after they dropped their original listeners. > - This pull request uses the existing bounded proof for recovered shutdown and removes the repeated test retry. > - The benefit is one cleanup rule, with the same fail-closed proof and no new work on normal task dispatch. ## Linked Issues or Issue Description Fixes: #20210 Thanks to @genius-0963 for the test-only fix proposed in #20321. That related alternative adds a test-only shutdown retry. This change applies the existing bounded cleanup proof in the shared shutdown path and removes the same retry pattern from native-provider integration tests. It also tests a single shutdown call while the original quorum is still held. ## What Changed - Use `await_provider_lifetime_fence` for a recovered executor with no live session and unconfirmed provider exit. Delete the unused one-shot helper. - Keep the acquired quorum held through the existing durable state write. A failed proof leaves `providerExitUnconfirmed` set. - Give the shared timeout message wording that applies to shutdown and suspension. - Extend the recovery test with a worker-start signal. Hold the original listeners until the single shutdown call is waiting, then release them and assert successful cleanup without a provider session. - Use the same bounded proof in the retired-quorum fixture test. Remove the duplicate retry loop from the native-provider test helper. - Document the bounded recovery cleanup wait. ## Verification - macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only targeted checks ran locally. - `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`: passed. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on the rebased head (5.21 seconds). The initial head also passed five more parallel-test runs (45 tests per run), with no failure. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test native_provider_backend acpx_`: 14 passed on the rebased head (9.43 seconds). These tests use the real Rust executor and controlled sidecars. - Negative control: restore only shutdown's original one-shot proof. The new regression fails at `shutdown must wait while the original lifetime still holds a quorum`. Restore the fix and the affected module passes. - Independent parent verification on current commit `368fc814f7`: the three exact recovery, held-quorum, and retired-quorum tests passed (3 tests, 5.06 seconds). Independent source review confirmed the unchanged three-file scope and found no actionable issue. - The active-turn recovery test still holds a live quorum through the five-second bound. It checks that shutdown refuses cleanup, saved state remains unconfirmed, and the provider never starts. It then releases the quorum and confirms cleanup. - Standard Greptile scored current commit `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads. - Initial GitHub general-server shard 7 failed in unchanged `remote-workspace-nested-verification.test.ts`, at its 15-second wait for a child-process import checkpoint (expected one row, got zero; both Vitest attempts). [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815). That fixture runs Node, PostgreSQL, Git, and storage migration; it does not run the native Runner or the changed Rust cleanup. The cause of the checkpoint timing miss remains unknown. The focused retry passed on the unchanged source ([successful retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)). This proves recovery, not the cause of the original timing miss. GitHub kept the reusable-workflow aggregate failed after that job-only retry and rejected both a final-gate retry and a failed-job retry. A subsequent run-level retry materialized only the gate and aggregate jobs; its aggregate inputs reported prior jobs as `abandoned`, so it failed without a new test failure. A new `full-ci` label event requests a fresh workflow graph on the same commit. The reason for those rerun-state results is not established. - The fresh `full-ci` run [42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208) then lost several runners mid-job. The maintained [runner-loss recovery 42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525) classified all four failed jobs as runner loss: typecheck, E2E shards 2 and 8, and server shard 4. The evidence was a shutdown log or lost-communication annotation. It retried the failed jobs together; canceled sibling jobs were also requeued. This is runner-interruption evidence, separate from the earlier checkpoint assertion. - The maintained recovery attempt again failed the same partial-checkpoint assertion in unchanged `remote-workspace-nested-verification.test.ts`, twice ([job 129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)). Neither checkpoint-failure job logged runner shutdown or lost communication; their resource summaries reported no significant constraint. The child phase at the timeout remains unknown. No further blind retry or TypeScript change is included here. Related startup-handshake work exists in #20339; its controlled-delay evidence establishes a deadline-boundary defect but does not establish the phase of these original CI failures. - Rebased onto current master after upstream #20339 merged. That separate change fixes a demonstrated child-startup/checkpoint-deadline boundary. It does not prove which child phase caused the earlier zero-checkpoint assertions. This PR still changes only its original Rust and documentation files. Fresh GitHub CI on the new base will establish recovery, not retrospective cause. - Full local tests and a local release build were not run. Current-head [PR CI 42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411), [Runner Windows 42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803), and [Docker Runner 42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601) all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check list has 59 successes and 5 intentional skips. GitHub supplied broad typecheck, build, and optimized Rust coverage. This is new-base recovery; the cause of the original checkpoint assertion failures remains unproved. ## Risks - A genuinely unconfirmed original provider can now hold recovered shutdown for up to the existing five-second proof deadline. The proof returns immediately when two candidate ports are free. - The changed branch requires both no in-memory provider session and unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch keep their existing paths. Outer controller close deadlines remain authoritative. - State saved by the previous release carries forward unchanged. No schema, launch identity, profile digest, protocol, or retry-budget change. - No live provider session or hosted deployment was tested. The tests cover cleanup proof, refusal, durable state, and controlled provider execution. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
0f71ab796e |
fix(issues): name the existing order in dependency-cycle denials (#20342)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks can wait for other tasks through blocking dependencies, and the server rejects a dependency that closes a cycle > - The rejection said only "Blocking relations cannot contain cycles" > - In chat, an agent asked to "make Website wait for Design too" got this 422 while Design already waited for Website > - The agent had no sign that the existing order must stay, so it guessed a swap: it removed Design's dependency on Website and added the reverse > - This pull request makes the 422 name the existing order, say that nothing changed, and say to ask the requester which order they want > - The benefit is that agents keep the existing prerequisite and ask the person, instead of silently reversing it ## Linked Issues or Issue Description **What happened** A chat agent was asked to make Website also wait for Design. Design already waited for Website. The server correctly returned 422 "Blocking relations cannot contain cycles". The agent then removed the existing Design-waits-for-Website dependency and added Website-waits-for-Design. The person lost the existing prerequisite without being asked. **Expected behavior** The rejection keeps both saved dependency sets as they were. The agent explains the conflict with the real task names and asks the person which order they want before it changes any dependency. **Steps to reproduce** 1. Create tasks Website and Design. Make Design blocked by Website. 2. PATCH Website with `blockedByIssueIds: [Design]`. 3. See that the 422 says only "Blocking relations cannot contain cycles", with no task names and no next step. ## What Changed - `server/src/services/issues.ts`: the cycle check now records the path it finds. The 422 message names the existing chain by task identifier (for example "COM-2 waits for COM-1, so COM-1 cannot also wait for COM-2"). It never includes titles, because the chain can pass through tasks the caller cannot read. Identifiers are the same fields that locked relation stubs already show. It says no dependencies changed and the existing ones are kept. It says not to remove or reverse an existing dependency, and to ask the requester which order they want. - The 422 `details` now carry `code: "blocking_cycle"`, the updated issue, the requested blocker, and the existing chain as ids and identifiers. - `server/src/__tests__/issues-service.test.ts`: a regression test checks the message and details for a direct cycle, the chain text for a two-step cycle through a task with a private title, that no title appears in the error, and that the existing dependency is unchanged. ## Verification - `cd server && pnpm exec vitest run src/__tests__/issues-service.test.ts` passes (152 tests). The new test fails on the old message. - Chat eval `task-coordination.reject-dependency-cycle` ran end to end with the first commit (run 42678089404, which named tasks by identifier and title). The agent kept Design waiting for Website and asked the person which order to keep with a question card. The saved dependency sets did not change. The second commit removes titles from the message; the rest of the text is the same. ## Risks - Low risk. The status code (422) and the first words of the message do not change. Clients that match the full old message exactly would no longer match. No code in this repository does that. - The check reads the identifiers of the issues in the chain only when it rejects a request, so the success path is unchanged. - The error can show the identifier of a task that the caller cannot read. Relation responses already show this identifier in locked stubs, so this shows no new data. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
251c6d5c69 |
fix(recovery): settle a remote workspace save whose process died mid-save (#20326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.
**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.
**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.
**Paperclip version or commit**
`master` at
|
||
|
Dotta
|
0eacad0d0c |
fix: refresh preserved harness receipt facts without copying archives (#20322)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Volume-free deployment must preserve historical controller files before removing the old disk. > - A WEB preservation receipt binds an archive and source files to the selected session and database facts. > - Later metadata changes can make those facts stale while the preserved files and archive remain unchanged. > - The existing owner refuses every stale receipt, so preservation cannot converge. > - This pull request refreshes only facts freshness after it verifies the same source bytes, checkpoints and archive. > - Existing locks and an old-receipt digest comparison prevent a concurrent replacement from being overwritten. ## Linked Issues or Issue Description Refs #19814 for WEB evidence preservation without settling remote recovery. Refs #19832 for bounded larger inventories. Related harness receipt PRs were searched; no existing change provides this guarded refresh. **What happened?** A valid WEB preservation receipt becomes stale after later run metadata changes. Its original-volume owner refuses another preservation attempt because an invalid receipt already exists. This leaves an eligibility blocker even when the source and saved archive are unchanged. **Expected behavior** The original-volume owner can refresh stale facts only after it proves every other persisted binding unchanged. It retains the prior receipt and remote recovery state. **Steps to reproduce** 1. Preserve a settled session's WEB files and controller evidence. 2. Add later run metadata so its database facts digest changes. 3. Preserve the same scope again. The previous implementation refuses at the existing-receipt check. **Paperclip version or commit** The failure was reproduced on master `12523fb677585fcd6da929f868014514a4e96e87`. This branch includes master `1b7fe33432016b3b27ccdfbf7b8944da4b40726d`. **Deployment mode** Managed Linux server with PostgreSQL and durable object storage. ## What Changed - Reuse the exact prior archive for existing WEB receipts. Recheck current source bytes, controller history, checkpoint material and archive bindings. Read back ciphertext and verify its size and hash. - Permit only database/local-observation freshness changes. Selected session, source run, controller, archive, WEB entries, and assets must still match the prior proof. Local inode and timestamp observations may change across attempts; preserved bytes, paths and modes must match. - Recheck current facts and source files under the existing finalizer, task and run fences. Accept a matching concurrent winner or require the exact previous receipt digest before replacement. - Retain the prior database and local facts digests, witness fingerprint, observation time and receipt digest in the same company-scoped activity transaction. Require those four fields to reconstruct the exact prior proof from unchanged current bindings before replacement. Normal audit redaction stays enabled. - Preserve current-key checks and archive reads on ordinary idempotent calls. Keep active-source, changed-evidence and cancellation refusals. - Document the refresh contract. No schema, saved-run format or public API change. ## Verification - Genuine RED: the new real PostgreSQL/encrypted-archive case failed at the old previous-invalid gate. - Before the audit review follow-up, the full local owner suite passed **94/94 integration cases in 50.01 seconds**. It covers same-archive refresh with no PUT, audit history, current-key failures, unreadable archives, cancellation, changed files, replaced volume, active sources, matching concurrent winners and conflicting receipt replacement. - The cases also cover a source becoming active during an idempotent readback. Earlier 91- and 93-case runs preceded these final review guards and are not the final result. - Server typecheck, module boundaries and diff checks passed on the final production/test bytes before the clean master merge. Required CI supplies final-head breadth. Four narrow post-merge refresh/current-key/idempotent regressions passed in 5.70 seconds on `82e270f29fff98442c4184b96082f72f18a88721`. - The fixtures use a real disposable PostgreSQL database and actual local encrypted archives/storage. Controlled faults affect the stated storage or concurrency boundary. They do not prove a hosted refresh or archive plaintext cold restore. - The existing integration file now exceeds 1,000 lines. Keeping these cases with its private original-volume, controller, archive and PostgreSQL fixture avoids duplicating that setup. - Initial worktree preflight stalled on the global package-manager wrapper. Preflight completed with the existing pinned toolchain. No setup failure is counted as a product test pass. - Review found that activity redaction could change a full prior-receipt copy. A new regression enabled the real username censor and reproduced that mismatch. The fix stores only the four changed scalar fields and their original proof digest; a prepublication canonical reconstruction guard rejects dropped or unknown fields. - On the audit follow-up, 16 affected real PostgreSQL cases passed in 11.98 seconds; 80 unchanged cases were intentionally skipped. They include real redaction with a username-containing WEB path, reconstruction across two refreshes, unknown prior fields, CAS, active sources, current-key failures, unavailable archives and cancellation. Final server typecheck and boundaries passed. This was an intermediate revision; the final-head results appear below. - A second review found that comparing old inode/timestamp observations also broke ordinary valid retries. Three real cases reproduced valid-receipt touch/inode failures and stale-facts touch failure. The fix permits metadata freshness changes across attempts while keeping exact byte/path/mode/checkpoint/archive coverage and the existing locked within-attempt metadata fence. - Final local owner suite on the metadata follow-up: **102/102 integration cases passed in 44.04 seconds**, including valid touch/new-inode retries, stale refresh with the same archive and no PUT, four-scalar audit reconstruction, changed bytes/mode/entries, and a touch during archive readback that must refuse. Server typecheck, module boundaries and diff checks passed. The final-head CI and review results appear below. - Final head `c81475c927d0862dcb8d2c2356dc41ab4a27e63f`: [PR workflow attempt 5](https://github.com/paperclipai/paperclip/actions/runs/42667287803/attempts/5) completed successfully at 19:24:28 UTC on 2026-11-09. All 51 jobs are accounted for: 49 passed and 2 were skipped. The actual [final ci / verify](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129382061558) and [E2E aggregate](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129381486342) passed. Typecheck, build and the selected test lanes passed. The exact-head rollup is successful with no pending or failed checks. - Attempt 5 resolved both trusted workflows to `3a2f71db31aa99633610dc241f55f34ac65124de`, after #20339 merged. Its routing gate passed and real jobs started. This verifies successful startup and completion of this rerun; it does not exercise the empty-runner fallback because routing succeeded. - Earlier CI failures on `82e270f29fff98442c4184b96082f72f18a88721` and attempts 1–4 of this final-head workflow remain nonpasses. Attempt 1 had interrupted leaf jobs and failed aggregates. Attempts 2 and 3 ended in workflow failure without a final verify job. Attempt 4 failed startup with zero jobs and empty-runner errors. Those failures are not counted as passes or labeled unrelated. - [Greptile](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746395090) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746327827) reviewed the unchanged final head at **5/5**. Both historical review threads are resolved; no inline thread remains open. - Hosted application to an actual stale scope remains pending normal release delivery. ## Risks - Refresh must not bless different preservation evidence. The unchanged-binding checks, fresh file and checkpoint proof, exact archive reuse, source capability and locked receipt comparison remain mandatory. - A fresh capability has a new witness fingerprint and expiry. The audit retains the old fingerprint, database/local facts digests and observation time. All unchanged proof bindings remain on the current receipt. Their canonical reconstruction must match the old digest. The new capability is validated at entry and publication. - Readback verifies ciphertext size/hash and current-key checkpoint material. This change does not claim a new archive GCM/plaintext restore. - Archive identity and rows remain unchanged, so retention ownership does not change. Failed refreshes retain the previous receipt, original files and unresolved remote recovery obligations. - Older receipt shapes are not rewritten automatically. Normal validation and named preservation refusals remain in force. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
908d2aae7a |
test(chaos): run waves against S3-compatible storage with a fault proxy, and click Retry in the real board UI (#20288)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud deployments keep run logs, traces, attachments, artifacts and
saved workspace files in S3-compatible object storage
> - The restart chaos harness ran only with local disk storage, so
storage failures and slowdowns were never tested
> - It also clicked Retry through the API, so a missing or broken Retry
button in the board UI went unnoticed (staging PAP-1683: "the
saved-files recovery Retry does nothing")
> - This pull request runs waves against a loopback MinIO behind a fault
proxy, adds store-outage, latency and throttling faults, and clicks
every Retry in the real board UI
> - The benefit is that storage outages and broken Retry paths now
surface in chaos waves. The first ST1 wave found that a run starting
during a store outage fails immediately and exhausts recovery
## Linked Issues or Issue Description
**What happened?**
Staging saves to object storage and has failures there that local chaos
waves cannot reproduce, because the harness used `local_disk`. The
harness also clicked Retry through the API, which skips the board's
actual button.
**Expected behavior**
Chaos waves can run the server in cloud storage mode, inject storage
faults, and press Retry the way a user does.
**Actual behavior**
They could not.
## What Changed
- `--storage s3`:
- starts a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, from
`~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`)
with its data under the instance temp root and random root credentials;
- creates the bucket;
- puts a fault proxy in front;
- points the server's `PAPERCLIP_STORAGE_*` settings at the proxy, with
path-style addressing and a per-wave prefix. The supervisor declares
these as explicit workspace storage for the isolation check.
- `storage-proxy.ts`: a transparent, non-retrying S3 fault proxy that
keeps the signed Host header. Rules match by method, path and remaining
count. Faults are outage (503 `ServiceUnavailable`), latency with
jitter, a 5xx or `SlowDown` error rate, connection reset, and 403
`ExpiredToken`. It counts requests by method and status, written to
`storage-proxy.json`.
- `--storage-faults ST1,ST2,ST3` (`storage-chaos.ts`): seeded 60–180 s
windows of outage, 1–10 s latency, or 30% 503 / `SlowDown`, applied
while the wave runs. They're logged in the timeline and in
`storage-faults.json`.
- `board-ui.ts`: every task and subtask Retry the grader needs is now
clicked in the board UI. The harness uses headless Chromium through
Playwright, opens `/issues/<identifier>`, waits for the visible Retry
button and captures the `POST /issues/:id/retry` the click sends.
- R3 fails a missing button as `ui_retry_button_missing (user saw: …)`
and a click that sends nothing as `ui_retry_no_request`.
- Crash-point waves, which serve no UI, keep the API path.
- README: a cloud storage section.
## Verification
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 244
passed. New tests:
- the proxy against a fake upstream: passthrough, bounded outage,
`SlowDown`, `ExpiredToken`, latency and reset;
- live MinIO through the proxy: `If-None-Match` returns 412, and an
injected 503 reaches the client;
- Playwright against local pages: a working button, a missing button,
and a dead button;
- the observer clicking through the board, and R3 signatures for UI
failures;
- window planning, apply and clear;
- `--storage` and `--storage-faults` parsing.
The observer and grader tests failed before the change.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Live waves on master
|
||
|
DottaandPaperclip
|
3a2f71db31 |
fix(ci): handle missing runner output and await import fixture readiness (#20339)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI checks must report real result failures after their setup phase. > - Required summaries must run even when the routing gate has no runner output. > - Process-recovery tests must start their checkpoint deadline after the child loads its modules. > - An empty runner prevents summaries from starting. Slow child startup can consume the checkpoint deadline. > - This pull request adds a fallback for the two summaries and a ready signal for the recovery fixture. > - Existing routing, checkpoint, key and recovery assertions remain in place. ## Linked Issues or Issue Description Refs #20322. Related open PRs #12512 and #12513 propose a broader runner contract and caller activation. This change does not adopt that contract or change normal routing. No current repair for an empty aggregate runner was found. **What happened?** The PR workflow can fail before starting any jobs when the routing gate has no runner output. GitHub reports an empty `runs-on` value for the `verify` and `e2e` summary jobs. Their `always()` conditions make this path possible when a dependency cannot run. A real workflow rerun showed both errors at lines 801 and 1784 of the trusted workflow. The first run of this PR also failed two process-recovery cases with no saved child checkpoints. The fixture starts its 15-second checkpoint poll before a new Node process imports the server modules. A controlled 16-second pre-import delay reproduces the same zero-checkpoint assertion. That evidence proves a deadline-boundary defect. The original CI run has no phase events, so it does not prove which stage caused those two CI failures. **Expected behavior** The summaries can start on `ubuntu-latest` when routing has no runner output. They must still fail when routing or required work did not succeed. The process fixture waits for a bounded child-ready signal before its unchanged checkpoint deadline. It must still prove checkpoint reuse, process termination and refusal under a changed master key. **Steps to reproduce** 1. Start or rerun PR CI with no runner output from the routing gate. 2. Let the two `always()` summary jobs evaluate their runner expressions. 3. Observe that the previous expressions produce an empty value and GitHub refuses the jobs. 4. For the fixture defect, temporarily delay the child before imports for 16 seconds. The previous fixture fails its 15-second checkpoint poll before the child is ready. The updated fixture passes with the same delay. The temporary delay is excluded from this PR. **Paperclip version or commit** Reproduced with the workflow from `f56b097fbd25607b3b84e418085ab42c2cbb758d`. The caller and both reusable workflow files are byte-identical between that revision and the affected PR head. This fix is based on current master `4a349ea44754f0b1fb8e5e04dc6cdcb05d1c3396`. **Deployment mode** GitHub Actions PR CI. ## What Changed - Add `ubuntu-latest` as the fallback only for `verify` and `e2e` when the routing runner is empty. - Extend the existing workflow contract test to require a usable fallback and execute each real summary script with a failed, cancelled or skipped gate and absent routing outputs. - Wait up to 30 seconds for the recovery child to finish imports and construct storage before starting the existing 15-second checkpoint poll. Keep the 30-second work abort, 60-second test timeout, and every kill, key, state and proof assertion. - Preserve normal runner selection, task selection, permissions and all result assertions. No production code, public command or product contract changes. ## Verification - The new local workflow contract failed before the two runner-expression changes. The existing summary scripts rejected failed routing while the test identified the missing fallback. - **148 focused tests passed** across CI planning, routing and E2E sharding in 16.00 seconds. These execute the actual summary Node scripts and preserve failure and cancellation checks. - `actionlint -oneline -shellcheck= -pyflakes=` passed for the trusted workflow, routing workflow and caller. `git diff --check` passed. - The canonical worktree preflight completed with the pinned toolchain and shared build cache. - This test does not implement a GitHub expression evaluator. The real GitHub startup error establishes the absent-output failure; the source contract and actionlint validate the proposed expression. Hosted execution of this fallback remains to be verified after the trusted workflow reaches master. - The unchanged two process cases passed locally before the fixture change: 2 passed in 31.06 seconds. A controlled 16-second startup delay then reproduced the zero-checkpoint failure in 21.81 seconds. The ready handshake passed with the same delay in 37.11 seconds. The temporary delay and phase logging were removed. These are measured test durations, not a measurement of the original CI child startup. - The final full nested-verification fixture passed: **38/38 cases in 132.30 seconds**, with disposable local PostgreSQL. This includes partial checkpoint reuse, changed-key refusal and all existing recovery controls. - Fresh final head `da35ea034bdee603c021c4eba5b0f78ee6ef52d9`: the [full PR workflow](https://github.com/paperclipai/paperclip/actions/runs/42678538191), [final ci / verify](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129374227690) and [incremental qualification](https://github.com/paperclipai/paperclip/actions/runs/42678537930) passed. The [previously failing server lane](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129366881302) passed. The final rollup has 61 successful checks, 6 expected skips, 2 neutral checks and no pending checks or failures. Snyk passed. - [Greptile](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748154727) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748242241) both reviewed that exact final head at 5/5. There are no inline review threads. Ponytail permits retaining the source check with actionlint, which is the validation used here. - The [first workflow run](https://github.com/paperclipai/paperclip/actions/runs/42674612619) failed the two cases described above; it is not counted as a pass. No CI retry was used to obtain the final result. CI uses the merged trusted workflow; its policy tests validate this proposed source file. ## Risks - A routing failure now produces a normal failed summary instead of an invalid runner expression. The unchanged assertions require successful routing, policy, valid scope and selected suite results. - Only lightweight summary jobs can fall back to GitHub-hosted capacity. Heavy jobs and authorization rules retain their existing route. - The new readiness allowance is bounded. A child that does not become ready, fails its work deadline, or cannot produce the expected checkpoint still fails. The original CI failure cause remains unproved beyond the observed zero-checkpoint assertions. - The fallback assertion is an exact source contract. A formatting change can require an assertion update. The separate actionlint check validates expression syntax; the text check does not emulate GitHub. - This does not repair a GitHub internal error or runner interruption. Those failures remain nonpasses. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
bce7503a82 |
perf(workspaces): time a restore's sandbox upload and extraction separately (#20331)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote (sandbox) workspaces are restored from a durable snapshot before a later turn, so restore time delays that turn's first token > - #20280 added run-log timing for the storage steps of a save and a restore > - A Daytona wave with a Paperclip-sized workspace (~750 MB) spent 39.9 s preparing the workspace of a restored run. The storage steps (download, decrypt, verify, publish) took 2.4 s, and `workspace.snapshot.sandbox_restore` took 35.6 s > - The archive helper unpacks the same archive locally in 3.6 s, so most of that time is elsewhere, probably copying 750 MB into the sandbox. One span cannot say which > - This pull request splits that span into the upload into the sandbox and the extraction in it > - The benefit is that the next measurement shows which part to fix ## Linked Issues or Issue Description Refs #20280 **What happened?** A restored run on Daytona spent 35.6 s in `workspace.snapshot.sandbox_restore` (wave `w-20261109T170632-s9921`, R1 workload, 754,838,327-byte snapshot). The run log records only that total, so it cannot show whether the copy into the sandbox or the unpack inside it is slow. **Steps to reproduce** 1. Run a remote workspace task with a large repository on Daytona and let a later run restore it. 2. Read the restored run's `run.performance.span` events. 3. Only `workspace.snapshot.sandbox_restore` covers the time after the storage download. **Expected behavior** The run log separates the upload into the sandbox from the extraction there. ## What Changed - `remote-workspace-transport.ts` (`restoreRemoteWorkspaceDirectory`): the archive copy into the sandbox runs inside `workspace.snapshot.sandbox_upload`, and the in-sandbox restore command inside `workspace.snapshot.sandbox_extract`. Both are children of the existing `workspace.snapshot.sandbox_restore`. - `execution-preparation-timing.ts`: the two new phase names. A traced `measureRunPhase` scope also carries its preparation ancestor's run (`recordRunId`). Without it, with an OTLP endpoint set, a step nested inside another step (here, upload and extract inside `sandbox_restore`) never reached the run log. Execution attempts still record nothing. - `doc/run-log-events.md`: the two rows in the durable storage timing table. - No behavior change. This is a run-log change: the rows stay in the instance database. ## Verification - New test `times the restore's sandbox upload and extraction as separate run-log steps` (`remote-workspace-transport.test.ts`). It restores a real archive through the real helper, inside the production `sandbox_restore` step with an OTLP endpoint set, with a 60 ms pause in each step, then reads the drained run phases. It fails without the spans or without the scope fix. - New test `records a step nested inside another step when an OTLP endpoint is set` (`execution-preparation-phases.test.ts`). It fails without the scope fix. - `npx vitest run` on `execution-preparation-phases`, `execution-preparation-timing`, `execution-timing-context`, `database-operation-attribution`, `native-workspace-finalization-work`, `remote-workspace-transport` and `remote-workspace-snapshot-storage`: 256 passed. ## Risks - Low risk. Two more measured steps on the restore path; spans under 50 ms are not recorded. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4d2a77cf5a |
perf(workspaces): verify saved Git objects without a single-threaded full fsck (#20329)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote (sandbox) workspaces are saved to durable storage as encrypted snapshots, and restored before a later turn > - Saves and restores are a reported source of slowness on staging, so #20280 added run-log timing for each step > - A Daytona storage wave with a Paperclip-sized workspace (R1, ~750 MB, 19.5k entries) showed each save taking 58–68 s. Storage was only 2–3 s of that; capture inside the sandbox was 44–47 s > - The first save runs while the first run prepares its workspace, so it delays the first token > - A local profile of the same capture found one cause: `git fsck --full` re-hashes the whole history on one thread on every save > - This pull request keeps the same object checks but verifies packs on several threads > - The benefit is about 10 s less per save on a Paperclip-sized repository, with the same corruption protection ## Linked Issues or Issue Description Refs #18990, #20280 **What happened?** Saving a remote workspace with a large Git history is slow. On Daytona (3 vCPU), a Paperclip clone's save took 58–68 s; capture was 44–47 s of it. The first save is on the first run's critical path: `heartbeat.prepare.workspace` took 90–110 s before the first turn. **Steps to reproduce** 1. Clone Paperclip (`git clone https://github.com/paperclipai/paperclip.git`, ~750 MB, one 459 MiB pack). 2. Run the generated archive helper: `node archive.cjs capture <clone> <out.pcws>` (from `remoteWorkspaceArchiveScript()`). 3. It takes ~30 s. A wrapper on `git` shows 22.5 s in one `git fsck --full --no-dangling` on the normalized copy. **Expected behavior** Capture verifies saved Git objects as #18990 requires (a missing or corrupt object blocks the save), without spending most of the save re-hashing unchanged history on one thread. ## What Changed - `remote-workspace-snapshot-archive.ts`: both `git fsck --full --no-dangling` calls (the offline source check and the normalized-copy check) now call a new `verifyGitObjects`. It runs: - `git fsck --connectivity-only --no-dangling`: every object reachable from refs, the index and saved Git state is present. - A direct hash of each loose object (inflate, then SHA-1 or SHA-256 per the repository's object format) against its file name. A mismatch or unreadable object fails with the existing `invalid_git_repository` reason. - `git index-pack --verify --threads=N` for every pack (N = available CPUs, at most 8). This re-hashes every packed object, on several threads. - `fsck --no-full` was not used: in a shallow clone it reports reflog entries that point at packed objects as invalid. - Tests (`remote-workspace-snapshot-archive.test.ts`): - New: capture refuses a repository missing a staged object, or a committed object. - New: capture refuses a repository with a corrupt byte inside a pack. - The existing #18990 test `rejects corrupt available objects in full/shallow/partial Git before publishing a receipt` (valid zlib, wrong hash) still passes. It failed with connectivity-only fsck alone, which is why the loose-object hash is there. ## Verification - `npx vitest run` on `remote-workspace-snapshot-archive`, `remote-workspace-transport`, `remote-workspace-migration`, `remote-workspace-online-source` and `remote-workspace-readonly-capture-budget`: 526 passed. - Local benchmark: a fresh clone of `https://github.com/paperclipai/paperclip.git` (753,506,675-byte archive, 19,506 entries), alternating runs on one host: | Capture | Run 1 | Run 2 | |---|---|---| | master helper | 32.04 s | 30.05 s | | this branch | 21.07 s | 20.09 s | Both produce the same archive: equal byte size, manifest digest (`e5facf033409…`) and sha256 (`a4eb843bff1f…`). - Component timings on the same clone: `fsck --full` 20.5 s; `index-pack --verify` 19.5 s on 1 thread, 10.7 s on 3 threads; `fsck --connectivity-only` 0.9 s. - Daytona evidence for the problem (wave `w-20261109T170632-s9921`, R1, 3 vCPU, MinIO): save `captureMs` 44–47 s, `transferMs` 11–20 s, storage `putMs` 1.3–1.8 s, `encryptMs` 0.4–0.7 s. I did not rerun Daytona on this branch: its image must match the checkout's runner content. The local helper run is the same code the sandbox executes. ## Risks - Same guarantees as before for missing and corrupt objects. `fsck --full` also applies its object syntax checks to every packed object; `index-pack --verify` checks hashes and pack integrity. Commits and trees are still parsed by the connectivity walk, so malformed ones still fail. - More CPU threads during capture, for a shorter time. Capture already runs inside the sandbox, off the control plane. - A follow-up could skip re-verifying a pack that an earlier save of the same workspace already verified. Most saves keep the same clone pack, so that would remove the remaining ~10 s on repeat saves. Not in this PR. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4a349ea447 |
fix(chat): tell agents that instruction saves keep no history (#20328)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents can read and edit other agents' instruction files with dedicated tools > - #14420 changed instruction saves: Paperclip now keeps only the current file and records no revision history > - The agent-facing descriptions still describe a "canonical instruction revision" history > - In a live eval, a chat agent asked to undo an instruction change found an empty history, read it as a test artifact, guessed the earlier text from the current file's wording, wrote it, and reported a restore > - This pull request tells agents the real contract in chat guidance and in the tool and API descriptions > - The benefit is that an undo with no saved version leaves the instructions alone and asks the person, instead of inventing an earlier version ## Linked Issues or Issue Description Refs #14420 Refs #19515 (open: would list save records from the activity log. Those records hold no content, so they still cannot restore earlier text) **What happened** A person asked a chat agent: "Undo Morgan's last instruction change." Morgan's history was empty, because saves do not record revisions. The agent's reasoning said the history "is only populated with canonical revisions" and that there was "no revision tracking in this e2e harness". The agent then removed the last line of AGENTS.md and replied "Undone. Morgan's AGENTS.md is back to …". No saved source had the earlier text, so the agent guessed it. **Expected behavior** When there is no saved earlier version, the agent leaves the instructions unchanged. It says plainly that no earlier version was saved, and it asks the person for the text they want or which part to remove. **Steps to reproduce** 1. Create an agent with an AGENTS.md file. Edit the file once through `PUT /api/agents/{id}/instructions-bundle/file`. 2. `GET /api/agents/{id}/instructions-bundle/history?path=AGENTS.md` returns `{"revisions":[]}`. 3. Ask a chat agent to undo the agent's last instruction change. The agent rewrites the file from a guess. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says three things: - Saves keep only the current instruction files. - An empty history means no earlier text is retained. - An undo with no listed revision must leave the file unchanged, say that no earlier version was saved, and ask for the wanted text. - The `get_agent_instruction_history` contract now says that it lists only pre-upgrade revisions, and that an empty list means Paperclip retains no earlier version. - The `restore_agent_instructions` contract now says that only listed pre-upgrade revisions can be restored. - The OpenAPI summary for `GET /api/agents/{id}/instructions-bundle/history` says the same. - The semantic tool contracts, the protocol manifest and the seeded eval catalog hash are regenerated. - Regression tests check the new guidance and both live tool contracts. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts server/src/__tests__/board-access-denial.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/runner-api.test.ts` passes. - In `packages/paperclip-runner`, `pnpm exec vitest run src/semantic-tools/paperclip-semantic-tools.test.ts src/catalog src/evals/native-execution.test.ts` passes. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-protocol-manifest.mjs --check` and `generate-replay-goldens.mjs --check` pass. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - The live result is the chat eval `agents.restore-instructions`, run with this change cherry-picked: run 42666621831 passed. The agent replied that Morgan's history is empty and no earlier version exists. It left AGENTS.md unchanged and asked for the exact text or the line to remove. Run 42660236086, without this change, failed: the agent guessed and rewrote the file. ## Risks - Low risk. Only agent-facing text changes, and no behavior changes. - The runner semantic catalog hash changes. This is the same kind of change as #19983. - If #19515 lands, history lists save records without content. The guidance wording must then say that listed saves without content cannot be restored. Whichever PR merges second must reconcile the two texts. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f56b097fbd |
fix(recovery): move stranded legacy subtasks off a parent's held workspace (#20314)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents split work into subtasks. Since #19860, every subtask gets its own execution workspace from the project base, and a subtask that an older release bound to its parent's workspace moves to its own workspace on its next admission > - Some legacy subtasks never get a next admission. Another task held the shared workspace with unsaved files, so their runs failed setup before any provider work. Stranded-work recovery then blocked them and handed them to the board > - On staging, a parent and two of its children stayed blocked for four days this way. The children could have run in their own workspace at any time > - This pull request makes the scheduled recovery-action sweep move such a child to its own workspace and continue it, with no person > - The benefit is that a legacy subtask never stays blocked on a workspace that it should not share, and the parent keeps its own workspace ## Linked Issues or Issue Description Refs #19860 (subtasks get their own workspace; legacy children move at admission) Refs #20295 (the held workspace's exhausted save now settles automatically) Refs #20273 (stall wakes wait while a workspace hold refuses a run) **What happened?** A parent task and three children shared the parent's remote workspace, because an older release created the children before #19860. One child's ended run held the workspace with unsaved files. The other two children were woken, but each run failed setup with "The workspace still has an unfinished writer or save operation." (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`). Stranded-work recovery then set each child to Blocked with a board-owned escalation (`board_escalation_no_takeover_v1`) that asked a board operator to retry. The #19860 move happens only at the next admission, and a blocked child gets no admission. **Expected behavior** A subtask that failed only because of a legacy shared-workspace binding continues in its own workspace from the project base automatically. The parent and the shared workspace stay unchanged. **Steps to reproduce** 1. With a release before #19860, create a parent task and children that share its remote workspace. 2. Let one child's run end with unsaved files whose save is not settled. 3. Wake another child. Its run fails setup on the workspace hold, and stranded-work recovery blocks it with a board escalation. 4. See that the blocked child is never moved to its own workspace and never runs again. **Root cause** `server/src/services/execution-workspaces.ts` (`releaseAncestorWorkspaceBindingAtAdmission`) moves a legacy child only when a run of the child is admitted. `server/src/services/recovery/service.ts` (`reconcileStrandedAssignedIssues`) ignores Blocked tasks, so nothing admits a child that a setup failure on the shared hold blocked. ## What Changed - `server/src/services/execution-workspaces.ts`: extract the binding release into `releaseAncestorWorkspaceBinding(tx, …)`, which runs inside the caller's transaction. Admission uses it unchanged (same guards: no other running run of the task, no unsettled workspace recovery, only a relative's workspace). - New `server/src/services/recovery/legacy-workspace-rehome.ts` (`rehomeStrandedLegacyWorkspaceTasks`). It selects a board-owned stranded escalation whose latest run failed setup on the workspace writer hold before any provider work, for a Blocked task bound to a relative's workspace. In one transaction it releases the binding, resolves the escalation with the outcome `legacy_shared_workspace_rehomed`, sets the task to `todo`, records a pending wake (`legacyWorkspaceRehome.wake: "pending"`), and records `issue.legacy_workspace_rehomed` system activity. After commit it wakes the assignee once (`issue_recovery_action_restored`, with a fresh session) and marks the wake `delivered` only when a run is created. A wake lost to a crash, refused (`null`), or failed stays pending and backs off (5 minutes, doubling, at most an hour); the pass returns the earliest owed wake as its retry deadline, so the recovery worker schedules it. Candidates are paged by action id. - Guards: the Blocked status must come from that escalation (set after the failed run, and no status decision at or after the block), the shared dependency-readiness rule must allow work (cancelled blockers and done blockers that still save files keep it blocked), the failed run must belong to the current assignee, and the binding release must be allowed. Otherwise nothing changes. - `reconcileActiveRecoveryActions` (the company-scoped sweep that the scheduled issue-recovery worker runs for each company with an active recovery action) runs this pass first. `recoveryActionSweepCondition()` also selects a company whose moved task still owes its wake. - `doc/execution-semantics.md`: documents the move and what a moved child sees. ### What a moved child sees A fresh workspace from the project base, exactly like a new top-level task in the project. It does not get the parent's changes. It does not get its own earlier files from the shared workspace either, because nothing of its work ran there (its runs failed setup before any provider work). The child that still holds the shared workspace with unsaved files is not moved by this change: its save recovery owns the hold (#20295 settles it automatically), its files stay in the shared workspace and in that recovery's saved-work receipts, and it moves on its first admission after that recovery settles (the #19860 rule). ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - **Migrated, no schema change.** The change reads escalations and task bindings that older releases saved, and it changes only the task's binding row, the task status, and the escalation. It does not change saved run identity (native execution input, leases, checkpoints, sessions). The regression test seeds the exact staging shape (Nov 4–5 evidence fields, `retryReason: workspace_busy`, `latestRunErrorCode: setup_failed`). - **Mid-run deploy.** A child with a running run or an unsettled workspace recovery is not moved (the same guards as the admission release). ## Verification - Red (the move disabled), `npx vitest run src/__tests__/legacy-workspace-rehome.test.ts`: 1 failed, 2 passed. The staging-shape case failed because the stranded child stayed Blocked and bound to the parent's workspace. - Green: 8 of 8 pass. All cases run the scheduled path: the company is selected by `recoveryActionSweepCondition()` and the pass is `recoveryService(...).reconcileActiveRecoveryActions({ companyId })` with the real heartbeat wake. - Staging shape: a parent and 3 children share the parent's workspace. One child holds it with a dirty writer run. Two children are blocked by board escalations after setup failed on that hold. After the stranded-work sweep, both children are unbound from the shared workspace, their escalations are resolved by the system with text that asks no person to act, and each child is woken exactly once with a fresh session. Each child runs once through the real heartbeat path outside the shared workspace and completes. The parent and the writer child keep their bindings, the shared workspace row is unchanged, and nothing runs for them. A later pass changes nothing. - A move whose wake was lost to a crash: the scheduled worker still selects the company, delivers the wake once, and the child runs and completes. - A refused wake (`null`) or a thrown enqueue stays pending, backs off, does not hold up other wakes, is reported to the worker as the next deadline, and keeps the company selected after a restart; once accepted, it settles. - Thirty rejected escalations do not hide a later eligible one (paging). - A child with a cancelled blocker stays blocked with its board escalation. - A child whose last status decision came before the block moves; one decided after the block stays blocked. - A stranded child whose setup failed for another reason stays blocked with its board escalation. - A task bound to the workspace it created (the parent) is never moved. - Related suites pass: `subtask-workspace-independence`, `heartbeat-workspace-busy`, `heartbeat-remote-workspace-busy`, `heartbeat-held-comment-wake`, `recovery-deadline-acceptance`, `heartbeat-retry-scheduling`, `task-workspace-binding` (278 tests), and after the review changes `issue-blocker-stall-wakeups`, `issue-recovery-deadlines`, `issue-recovery-work-lifecycle`, `recovery-durable-wait-projection`, `owner-gone-attestation` (124 tests with the two workspace suites). - `pnpm verify`: exit 2 ("broader verification required", because `doc/execution-semantics.md` is a global CI input). Required CI supplies the broad coverage. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - The pass changes a Blocked task to `todo` only under narrow guards, and only for a task bound to a relative's workspace whose last run failed setup on that workspace's writer hold. A Blocked status from another decision, or an unresolved blocker, keeps the task blocked. - The moved child starts again from the project base. This is the product rule for subtasks. Nothing of its own work is dropped, because its runs failed before any provider work. - The pass runs inside the existing active recovery action sweep and selects at most 25 escalations and 25 pending wakes per pass. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only API checks) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
76254ead22 |
fix(runner): allow Codex interruption to settle governed waits (#20330)
## Thinking Path > - Paperclip manages AI agents and their work. > - A native run must stop provider work before it waits for a human response. > - The Codex transport allows 30 seconds for a control command. > - The governed-wait loop used its 15-second default for Codex. > - A valid interruption terminal at 18 seconds therefore failed the run. > - This change gives Codex the existing 30-second terminal drain allowance. > - The run still fails if it cannot prove that the provider turn ended. ## Linked Issues or Issue Description **What happened?** A Codex run can save a human wait, accept an interruption, and fail before the provider sends its terminal. Its saved wait remains intact, but normal finalization does not occur. **Expected behavior:** Allow terminal delivery through the transport's bounded command window. Keep missing-terminal cases closed. **Steps to reproduce:** Run the new protocol regression against the old backend. The real Runner binary uses a controlled Codex process that delays its interruption terminal for 18 seconds. The run fails at the 15-second drain deadline. The same process passes with this change. **Paperclip version or commit:** Reproduced against `1b7fe33432`. **Deployment mode:** Local test of the native Runner path. The incident was observed in a hosted sandbox. **Installation method:** Source checkout. **Operating system:** macOS arm64 for local verification. Hosted incident runtime was Linux. **Additional context:** No matching repair was found. This addresses a reproducible deadline mismatch. The hosted log ends terminal observation at 15 seconds, so it cannot prove whether that provider would have settled within 30 seconds. The approved local live-provider check passed. This PR does not claim recovery of the hosted task. ## What Changed - Give Codex a 30-second governed-wait terminal drain through the existing backend option. - Keep other providers and the caller's shorter execution bound unchanged. - Add real Runner protocol cases for prompt, delayed, and absent terminal delivery. Check retained wait state and one provider turn. - Document the bounded wait and terminal-proof requirement. ## Verification - Red: the 18-second controlled-provider case failed with `governed_wait_provider_terminal_unproven` on the old backend. The prompt case passed. - Green: 282 tests passed across the native runtime, harness backend, and new Runner protocol suite, including a repeat on the rebased PR revision. `pnpm verify --explain` returned exit 2 because the execution-contract document requires broad CI; the [required CI run](https://github.com/paperclipai/paperclip/actions/runs/42668235142) passed on the final head. All 58 applicable checks passed; the three skipped checks were the two unrelated Storybook jobs and the selected-test planner replaced by broad coverage. - The missing-terminal case still fails without `completeRun` or a successful run terminal. The saved governed wait remains available. Only one provider turn and one tool call occur. - Existing runtime tests cover shorter deadlines, foreign or unproven terminals, complete usage without a terminal, and checkpoint recovery without another model turn. - Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64, disposable local workspace. One `request_human_input` tool call returned a pending interaction fixture. The real provider emitted `turn.interrupted`; the controller accepted the yielded wait and its run terminal. The check passed in 8.14 seconds. The tool callback and persistence port were controlled; no hosted server permission or Notebook publication claim is made. - Initial local probe setup failed before model execution: first the explicit launch lacked `app-server`, then it omitted the Runner permission configuration. The successful check used the Runner's normal launch arguments and isolated home with the approved existing account. - Hosted sandbox acceptance and recovery remain unverified. The 18-second delayed shutdown is a controlled protocol regression; the live model check proves normal shutdown, not that specific latency. ## Risks - A Codex provider that never stops can occupy the settlement window for up to 15 additional seconds. It still fails closed and is quarantined. - No persisted schema, provider identity, permission, or recovery rule changes. Previous-release checkpoints use the same format and existing recovery checks. - This change cannot fix a provider that never emits a terminal. It also does not grant missing setup permissions or decide pending approvals. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool calls, code edits, and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
34c1e6469c |
fix(storage): retry small uploads and resume broken downloads (#20313)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task's files live in a sandbox. Paperclip saves them to object storage when a run ends and restores them when the next run starts > - Object storage has short blips: a single 500, a reset connection, a download cut off halfway. Each of these used to fail the whole save or restore once, with no second try > - This pull request makes a short blip invisible. Longer outages are handled by the storage-unavailable wait (#20299 and a follow-up) > - The benefit is that tasks keep moving through ordinary storage failures without anyone clicking Retry ## Linked Issues or Issue Description Refs #19831. Storage chaos waves `w-20261109T145628-s9902` (ST1) and `w-20261109T151711-s9903` (ST2). - A single-request upload (files up to the multipart threshold, which includes most workspace archives) sent once. A 500 or connection reset failed the save. - A download whose body broke mid-stream failed the restore, even though S3 can resume from the byte offset. Related: #20299 (storage outage is shown as an infrastructure wait instead of a failure). ## What Changed - `server/src/storage/s3-multipart.ts`: `sendFileRequestWithRetry` resends a single file request and records throttles and backoff in the save timing. It uses the same throttle backoff as multipart parts for 429/503/SlowDown and up to the same part attempt count for other failures. It honours the caller's abort signal. - `server/src/storage/s3-provider.ts`: - A file-backed `putObject` under the multipart threshold uses that retry and reopens the source file for each attempt, so a resend never reuses a half-read stream. The Content-MD5 is computed once from the file and still verified by S3 on each attempt. - `getObject` resumes a body that fails with a transient socket error (reset, timeout, premature close and similar). It continues with a ranged GET from the byte where the stream stopped, at most three times. The resumed request carries `If-Match` with the first response's ETag, so a changed object fails instead of mixing bytes. Body-failure diagnostics are still recorded on every response. Destroying the returned stream closes the S3 body it is reading, even mid-read, so read timeouts and client disconnects still release the connection. - Tests: - `server/src/__tests__/storage-s3-transient-retry.test.ts` (new) runs against a local HTTP server speaking S3. ## Verification Red/green (each new test fails against master and passes with this PR): - `storage-s3-transient-retry.test.ts`: - A small upload is resent after a 500 and after a connection reset, re-reading the file. The second PUT carries the full content. - A download cut off mid-body resumes with `Range: bytes=<offset>-` and `If-Match`, and returns the complete bytes. - A download whose object changed before the resume fails with a precondition error and does not return mixed bytes. - A stalled download that its reader destroys closes its connection. Regression: the storage and workspace-finalization suites pass locally, and `tsc --noEmit` passes for `server`. ## Risks - Resending an upload is safe because each attempt writes the full object with the same Content-MD5. Resuming a download is bounded to three resumes and refuses a changed object. - No schema migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
a3e7e63330 |
Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. **The current fix has a causal red/green check.** On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: **5 passed, 0 failed;170 filtered rows excluded**. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed **25/25**, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, **15 selected accepted-result/restart controls passed**. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete **17-case maintenance-server file**. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed **29/29**. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
365c771030 |
fix(chat): give project and workspace tool results their page links (#20324)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask a chat agent to take them to a project or to the files of a task > - The agent finds the right project or workspace, but it cannot link the page > - The `list_projects` and `list_workspaces` tool results have no page route, and the chat link guidance names no project or workspace format > - The agent then writes a bare path or a shortened directory that the person cannot open > - This pull request adds the page `href` to these tool results and adds the project link format to the chat guidance > - The benefit is that chat replies give a clickable link to the real project or workspace page ## Linked Issues or Issue Description **What happened** In chat, a person asked "Take me to the Launch project and show its open tasks." The agent found the project. Its reasoning said "There's no project page route documented explicitly". The reply wrote the project as a bare path `/COM/projects/<id>`, which is not clickable. In a second chat, a person asked where the files of a task are. The agent named the correct workspace but gave a shortened directory and no link. **Expected behavior** The reply links the project page (`/<prefix>/projects/<id>`) and the task's workspace page (`/<prefix>/execution-workspaces/<id>`) as Markdown links, so the person can open them. **Steps to reproduce** 1. Create a company with a project and a task that has a saved execution workspace. Enable isolated workspaces. 2. In chat, ask the agent to take you to the project, or to say where the task's files are. 3. Look at the reply. It has no project or workspace link. Related: Refs #17985 (an earlier open PR that adds project and goal link wording for newly created projects only). This PR covers project discovery and workspaces and adds tool-result hrefs. ## What Changed - `list_projects` rows now carry `href: /<prefix>/projects/<id>`. Both the native runner tool authority and the MCP project-tools route pass the company prefix. - `get_workspace` and `list_workspaces` now carry `href: /<prefix>/execution-workspaces/<id>`, only while isolated workspaces are enabled. The workspace route redirects to the dashboard when the feature is off, so no href is offered then. - The chat link guidance (`AGENT_CHAT_DIRECTIVE`) adds the project link format. It tells the agent to link a returned `href`, and to name a workspace that has no `href` instead of linking it. - Regression tests: guidance text against the registered UI route, `callProjectTool` href, and an embedded-Postgres authority test for project hrefs and the workspace flag on/off. ## Verification - Red then green: the three new tests fail on `origin/master` source and pass with this change. - `npx vitest run` in `server/` for: `agent-conversations`, `task-workspace-api`, `project-tools-pagination`, `task-workspace-tools`, `chat-project-tools`, `dot-runner`, `heartbeat-list` (prompt-size limits), `task-markdown`, `muse-native-visibility`, `paperclip-runner-tool-authority`, `runner-api`, `codex-local-execute`, `board-access-denial`. All pass. - `npx tsc --noEmit -p server` passes. - Live chat eval: see the comment on this PR for the paid runner E2E run of `navigation-preferences.find-project-view` and `workspaces.locate-files` with this change. ## Risks - Low risk. The change adds one optional field to read-only tool results. The output schemas allow extra fields. - The guidance grows by two sentences. The prompt-size test in `heartbeat-list.test.ts` passes. - `list_projects` and the workspace tools now do one extra small read (company prefix, and the instance experimental settings for workspaces). ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
0f84df91da |
fix(native): accept a completed child's file in the parent's completion report (#20325)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In onboarding, the agent plans the user's first task, creates one child task for the work, and waits for it > - The `first-task` skill tells the agent to publish a requested file on the child task. When the parent wakes, the agent links the child's file and does not publish a copy > - Native `register_deliverable` is bound to its current task, so the parent cannot register the child's file again > - The native completion check required each cited attachment to be on the task being completed > - So when the parent cited the child's download link, `paperclip_finish` failed, and the agent reported the finished onboarding task as blocked > - This pull request lets a completion report cite an attachment on a direct child task that is done > - The benefit is that onboarding closes after its child task delivers a file, and the user does not see a false "blocked" status ## Linked Issues or Issue Description Related: #18258 (onboarding artifact completion instructions). **What happened?** An onboarding run created a child task to build and publish a project. The child task registered `project.zip` with `register_deliverable` and was marked done. The parent woke with reason `issue_children_completed`. The agent called `paperclip_finish` with `reportedWorkDisposition: "done"` and cited the child's `/api/attachments/<id>/content` link. The call failed two times with: ``` Completion cites no registered attachment on this task. Use register_deliverable for the requested file and cite deliverable:<attachmentId> from its receipt. ``` The skill does not let the agent register the file on the parent, so the agent called `paperclip_block` with "Paperclip completion validation" as the owner. The parent task stayed `blocked`. Nothing wakes it again. **Expected behavior** The parent completes and links the child's file. **Steps to reproduce** 1. Create a task with a child task. 2. In a native run on the child, register a file with `register_deliverable`, and mark the child done. 3. In a native run on the parent, call `paperclip_finish` with `reportedWorkDisposition: "done"` and the child's `deliverable:<attachmentId>` or `/api/attachments/<attachmentId>/content` as evidence. 4. The call fails with "Completion cites no registered attachment on this task". ## What Changed - `server/src/services/native-runtime/native-deliverable-feedback.ts`: when a cited attachment is not on the task, `validateNativeDeliverableEvidence` also looks for it on a direct child task with status `done`. If it is there, the citation is accepted. - The child's file is context, the same as an attachment from an earlier run on the same task. It does not count as this run's publication of a file that the parent's own request names. - With `lockSources`, the finalization path locks the attachment and child task rows with `FOR SHARE NOWAIT`, as it does for other sources. - The error text now says "on this task or its completed child tasks". - This applies to the `paperclip_finish` check and to the two finalization checks, because they use the same function. - `doc/AGENT-ARTIFACTS.md`: the native runner section states that a completion report can cite a file on a completed direct child task. ## Verification - Extended "prepares one verified same-run attachment and replays without duplicates" in `server/src/services/native-runtime/native-runner-file-handoff.test.ts`: - An attachment on an unrelated task is still rejected. - An attachment on a child task that is not done is still rejected. - An attachment on a done child task is accepted, through `paperclip_finish` and through the locked finalization path. - When the parent's own request names a file, the child's file alone does not satisfy it. - Without the source change, the test fails with the same error as the run above. - `npx vitest run` on `chat-attachment-reuse`, `native-completion-feedback`, `native-completion-format-feedback`, `native-deliverable-feedback`, and `native-runner-file-handoff` tests in `server/src/services/native-runtime`: 281 passed. - `npx vitest run src/__tests__/native-result-commit-work.test.ts src/__tests__/native-status-arbiter-corpus.test.ts` in `server`: 167 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Only direct children with status `done` are accepted. A grandchild's file, or a file on a child that is not done, is still rejected. - A parent that names a requested file in its own request still needs its own delivery evidence. This keeps the current rule for earlier outputs. - A task that is already blocked by this error does not change. The user can mark it done, or comment so the agent runs again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, database queries, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
47dccbf3eb |
fix(recovery): re-prove an exhausted save's agent stop and free the workspace (#20295)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run ends, Paperclip saves the files that the run changed in its remote workspace. Until that save settles, the run holds the workspace, so no other task can write it > - When the save fails, a bounded system recovery proves that the agent stopped (a provider stop receipt) and then saves the files again > - If the stop cannot be proven before the 30-minute recovery window ends, the recovery is exhausted with `agent_stop_unconfirmed`. Exhaustion was terminal: Paperclip never proved the stop again, even after the run ended and its sandbox had a stop receipt > - The hold then stayed forever. On staging, a child task's save was exhausted this way on Nov 5. Its parent and its two sibling tasks share that workspace, and all of them stayed blocked for four days > - This pull request re-arms such an exhausted save automatically, on a bounded backoff, once the run is provably over. After the bounded re-arms, the hold continues from the last save with the unsaved files attached > - The benefit is that a dirty workspace hold always settles without a person, and no unsaved work is discarded silently ## Linked Issues or Issue Description Refs #20273 (the stall sweep stops failing wakes against this kind of hold) Refs #20278 (a settled restore hold returns its warm sandbox) Refs #19860 (subtasks get their own workspace) **What happened?** A child task's run wrote its remote workspace and failed to save it (`native_workspace_sync_out_retry_exhausted`). The save recovery made one attempt. That attempt failed with `agent_stop_unconfirmed`, and the recovery window ended before a second attempt. The recovery became `exhausted`. Ten minutes after the first attempt, the sandbox lease got a provider stop receipt, so the agent was provably gone. But nothing looked at the exhausted recovery again. The run kept the workspace's writer hold. The parent task and every other task on that workspace could not start ("The workspace still has an unfinished writer or save operation."). The blocker told a person to press Retry. **Expected behavior** An unproven stop is a wait, not a refusal of the saved files. When the run is over, recovery must prove the stop again and finish the save, so the hold settles automatically. If the stop still cannot be proven, Paperclip must keep the unsaved files and then release the hold. A person must not need to act. **Steps to reproduce** 1. Run a task in a remote (sandbox) workspace. Make its final save fail. 2. Keep the workspace recovery from proving the agent stop until its 30-minute window ends (for example, keep the finalization ownership busy). 3. See that the recovery action is `exhausted` with `agent_stop_unconfirmed`, and that it stays so after the lease gets a stop receipt. 4. Wake another task bound to the same workspace. See that its setup fails because the workspace is held. **Root cause** - `server/src/services/workspace-recovery.ts` (`recoverRetainedWorkspace`): an `exhausted` state only goes to the sibling settlement and to the retained-source fallback. The fallback accepts only permanent source refusals (`isUnverifiableSourceFailure`). Every other exhausted state is handed to the board and never attempted again. - `recheckStoppedWorkspaceRecovery` skips `exhausted` states, and it only accepts acknowledged cancellations. A failed run whose lease got a stop receipt later is never re-examined. - `retainedWorkspaceCandidates(…, obligationsOnly)` and `nextWorkspaceRecoveryAt` do not select exhausted `agent_stop_unconfirmed` holds, so the recovery worker never wakes for them. ## What Changed - New `server/src/services/workspace-recovery-stop-proof.ts` (`rearmExhaustedStopProof`). For an exhausted save whose last failure is `agent_stop_unconfirmed` (`recheck_evidence`), it starts a new bounded system recovery generation. It does this only when the source run is provably over: the run is terminal, no local execution remains, and no controller lease or finalizer lease is live. The new generation repeats the strict stop proof (a provider stop receipt) and the save. It records an `issue.workspace_recovery_rearmed` system activity. - Re-arms are bounded: at most 4. The first is due at once. The next ones are due 30 minutes, 2 hours and 8 hours after the previous re-arm. The `stopProofRecovery` marker in the recovery action evidence records the count and the next due time. - `workspace-recovery-fallback-discovery.ts`: new discovery conditions (`stopProofRearmPendingCondition`, `stopProofRearmDueAt`, `stopProofRearmDueCondition`) and `retainedHoldFallbackEligible`. `retainedWorkspaceCandidates` (worker pass) and `nextWorkspaceRecoveryAt` use them, so the recovery worker wakes when a re-arm is due. - `workspace-recovery-fallback.ts`: when every re-arm is spent and the stop is still unproven, the hold takes the existing retained-source fallback. The unsaved files are attached to the task (unapplied, secrets omitted), the unverified epoch is abandoned, and the task continues from the last save. - `system-recovery-state.ts`: the exhausted `agent_stop_unconfirmed` text now says that Paperclip tries again automatically. It no longer asks a person to press Retry. Retry stays available. - Review follow-ups: the re-arm applies to every workspace recovery the worker owns (restore and historical holds too). A due re-arm is scheduled no earlier than the source run's controller or finalizer lease expiry, so the worker is not woken every 15 seconds while a lease is live. A fallback whose bounded checks are spent is reconsidered once when an allocation of the source run changes, and `settleStopOnlyCleanup` wakes the workspace recovery worker when it records a stop receipt. The automatic-retry text is shown only while a re-arm remains. - `doc/execution-semantics.md`: documents the re-arm, its bounds, the fallback, and the upgrade behavior. ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - **Carried forward, no migration.** A hold exhausted by an older release has no `stopProofRecovery` marker. The recovery worker treats it as due at once and re-arms it on its first pass after deploy. The new regression test saves the exact Nov 5 evidence shape (`attempts: 1`, `retriesUsed: 0`, `state: exhausted`, no marker) and resumes it with the new code. - **Mid-run deploy.** A run that is still running, or whose controller lease or finalizer lease is live, is not re-armed. The test covers a live controller lease. The re-arm uses the same CAS (generation and revision) and the same row locks as the other recovery writers, so an old worker's late write cannot overwrite the new generation. - **No new refusal codes.** No saved run identity changes. The re-arm writes only recovery-action evidence. ## Verification - Red (before the fix), `npx vitest run src/services/workspace-recovery-stop-proof.test.ts`: 3 failed, 1 passed. The staging-shape test failed because the worker did not wake (`nextWorkspaceRecoveryAt` returned `null`) and the hold stayed. The backoff and fallback tests failed for the same discovery gap. - Green (after the fix): 4 of 4 pass. - Staging shape: a parent and 3 children that share one remote workspace. One child's save is exhausted with `agent_stop_unconfirmed`, and its run ended days ago with a stop receipt. Before recovery, the hold refuses the parent (`remoteWorkspaceHoldRefusesStart` is `true`). The recovery worker discovers the hold, re-arms it, proves the stop, saves the files and completes. The workspace is not dirty, it has a new snapshot, and the hold is `free` for the parent. The action stays system-owned and its text asks no person to act. - A live controller lease keeps the hold. No re-arm happens and no lease is resumed. - A re-arm that is not yet due is not discovered. The worker's next wake is in the future, and the text says that Paperclip tries again automatically. - After 4 re-arms, the hold continues from the last save. The unsaved files are captured as a task attachment, and the hold is `free`. - A live controller lease defers the worker's next wake to the lease expiry, and the hold is not selected before it. - A spent fallback is not selected, and a later lease change (a late stop receipt) makes it selected again; the worker then settles it. - `system-recovery-state.test.ts`: the automatic-retry text appears only while a re-arm remains. - Related suites, all pass: `src/services/workspace-recovery*`, `src/services/remote-workspace*`, `src/services/system-recovery-state.test.ts`, `src/services/task-retry.test.ts`, `src/__tests__/heartbeat-observed-workspace-preservation.test.ts` (34 files, 1510 tests), plus the stop-cleanup suites (`environment-retained-orphan-sandbox`, `live-recovery-work`, `retained-native-session-cleanup-work`, `legacy-workspace-restore-recovery`, native workspace warm-reuse, cancelled preservation, reconstruction and failed recovery; 11 files, 622 tests). - `pnpm verify`: exit 2 ("broader verification required", because `doc/execution-semantics.md` is a global CI input). The focused suites above cover the change, and required CI supplies the broad coverage. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A re-armed generation resumes the stopped sandbox to save its files. This is the same physical path as a manual Retry. It runs only when the source run is provably over. - Re-arms are bounded (4) and spaced, so a stop that can never be proven does not loop. After that, the existing retained-source fallback settles the hold. If the retained files cannot be read, the fallback records why. This is the same behavior as for a refused source. - The parent's workspace receives the child's saved files, because the child (bound by an older release) wrote that workspace. This is the existing #19860 behavior: the files that a legacy child left in the shared workspace stay there. #20314 moves the stranded legacy children to their own workspace. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only API checks) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8fe524f796 |
fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a new attempt can start on a task, Paperclip must prove the previous run's agent stopped, so two agents never work the same task at once > - A local native run can be interrupted after it asked to start the runner but before it recorded the runner's process. The runner may have saved state and started a provider, and provider processes can outlive it, so the existing check (#19694) rightly refuses to assume the run stopped > - Today that case has no way out: every Retry is accepted and then waits forever as `process_identity_missing` (storage chaos ST1, t02/t03) > - This pull request makes the wait explicit and actionable. The task explains why it waits, and a board operator who has checked can confirm the run stopped; the parked Retry then starts at the click > - A second shape from the Daytona storage baseline (t06) parked a Retry with no reason at all: a follow-up was interrupted by a graceful shutdown while it was still restoring its workspace. No provider ran, but the restore hold kept the same agent held, so the Retry never started > - The benefit is that no Retry stays accepted-and-waiting with nothing anyone can do ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks t02 and t03. **What happened (CHA-3, run `3c3f6241`):** - The run requested a runner launch (`native.process_start_requested`), and a K2 restart hit before its process identity was recorded. - After the restart, the resumed attempt failed on the storage outage. The task was blocked. - The board's Retry (202) was parked as `executionWait.reason = process_identity_missing`: "The previous run has no verified stop record." - `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this run's runner saved control-plane state. Once that exists, a provider may have started (#19694), and a provider is not guaranteed to carry `--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on purpose). The shared owner prover needs a PID, which the run never recorded. - Nothing could ever clear the wait. **Daytona storage baseline `w-20261109T154810-s9911`, task t06:** - The run succeeded. The follow-up was cancelled (`continuation_task_ownership_changed`), and the next run was interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s into `preparing`. It was restoring the saved remote workspace and had launched no provider. - Restore recovery saved the files (`workspaceRecovery.status: recovered`, system recovery `complete`). But the `legacy_execution_requires_reconciliation` restore hold releases only for a different owner, or for a native Stop fenced before any provider turn, so it kept the same agent held. - The UI Retry returned 202 and parked with `executionWait.reason: execution_recovery`, the generic wait, so the task showed a Retry that never started. Automatic proof needs the runner to record its identity before saving state and to group provider processes under it. That is follow-up runner work; this PR removes the endless wait now. ## What Changed - `server/src/services/native-local-process-stop.ts`: new `confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished native run of the task with no recorded process and whose latest launch evidence is an unconfirmed launch on this host in a local environment. It still refuses while a runner with `--run-id <run>` is running. It records `native.local_process_stopped` with `basis: "operator_attested"` and the attesting user. That is the receipt the existing continuation gate already accepts. - `server/src/routes/issues.ts`: `POST /issues/:id/recovery-actions/confirm-run-stopped` (`runId`, `stopEvidence` of 20 to 12,000 characters). It is board-only and requires issue read plus `runtime:confirm_owner_stop`, like the other stop attestations. It logs `issue.run_stop_confirmed` with the evidence, promotes deferred intents for the issue, and resumes each parked task Retry, so work starts at the click. Refusals return 409 with a code. - `packages/shared`: `confirmRunStoppedSchema`, and the presentation user action `confirm_run_stopped`. - `server/src/services/task-interruption-presentation.ts`: a Retry parked as `process_identity_missing` for a native run that recorded no process now presents as waiting for the user (`canRetry: false`, `userAction: "confirm_run_stopped"`), with a message explaining why and who can act. A run with a recorded process keeps the existing verification wait. - UI: - `ExecutionBlockerNotice` shows the explanation and, for users who can confirm a stop, "Confirm it stopped" with a short evidence form. - It reuses `PreviousServerStopConfirmation`, now with optional `action`, explicit `runId` and `formId`, and a custom button label. - The Inbox names the action. - `server/src/services/execution-blocker.ts`: `interruptedPreparationBeforeProviderWorkCondition`. The restore hold also releases for the same agent when the stopped run was a system-interrupted preparation (legacy runtime never resolved, still `preparing`) with no process identity, no runner or provider events, no finalization or result, and every lease released with a confirmed stop. It mirrors the existing interrupted-preparation proof and still requires the completed workspace-recovery receipt. Auto-replay is unchanged; this only lets the user's Retry or message start. - Review fixes: - `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and lease checks, and the presentation offers `confirm_run_stopped` only when it passes. - The attested stop wakes cost accounting in the same transaction. - The confirmation form is keyed by task and run. - The route is registered in OpenAPI. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`: - The parked Retry presents `confirm_run_stopped` with the explanation. - Confirming records the operator-attested stop and the activity entry, satisfies the stop check, and resumes the parked Retry: its receipt is coalesced into a newly started run. - It refuses with 409 while a runner for the run is running. - Agents and viewers get 403. - `ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`: the notice offers "Confirm it stopped" (and no dead Retry) and posts the run and evidence. A viewer sees the explanation without the action. - `server/src/__tests__/interrupted-preparation-restore-retry.test.ts` (t06 shape): - The restore hold no longer blocks. - A Retry is accepted and resumed into a started run, and the audit hold keeps `replay: blocked`. - The hold stays when a provider turn started, when a runner event was recorded, when an operator stopped the run, or when the sandbox has no stop receipt. - `issue-confirm-run-stopped-routes.test.ts`: - A case per refusal code checks the presentation keeps the generic wait and the endpoint returns 409. - An `alreadyStopped` case. - An accounting-wake assertion. - The UI test switches targets with stale evidence. Regression and static checks: - `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control tests: 74 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A board operator can now attest that an unidentified run stopped. The attestation is permission-gated, requires written evidence, is activity-logged, and is refused while a runner for the run is visibly running. A wrong attestation could start a second agent beside an orphaned provider, the same trust model as the existing previous-server attestations. - The confirmation component now accepts a run without a recovery action. Existing callers pass the action as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ff2d463eed |
feat(cli): manage agent grants and keep remote login on the public origin (#20323)
## Thinking Path
> - Paperclip lets people control their AI agents and permissions.
> - Remote operators need a browser-approved CLI login.
> - Proxies can send that login's approval link to an internal host.
> - Human member commands cannot grant an agent an explicit permission
such as commentary:view.
> - This change keeps approval on the canonical public origin and adds
Board-only single-grant commands.
> - The API checks company access and permission-management authority
and records each change.
> - An operator can sign in, grant one agent permission, and preserve
the agent's other grants.
## Linked Issues or Issue Description
**What happened?**
A proxied CLI login returned an approval URL on the upstream host. The
signed-in browser was on the public host. The CLI also had no command to
add an arbitrary explicit agent grant: member commands manage human
members, while agent permissions:update exposes fixed configuration
fields.
**Expected behavior**
Approval uses the configured canonical public origin. A Board operator
with users:manage_permissions can grant or revoke one explicit agent
permission without replacing other grants or activating an inactive
membership.
**Steps to reproduce**
Configure a public auth origin. Create a CLI challenge through a proxy
that replaces forwarded-host headers. Inspect the approval URL. Then try
to grant commentary:view to an agent through the CLI.
## What Changed
- Use the existing public-origin resolver for CLI approval URLs.
- Add agent permission:grant and permission:revoke commands.
- Add PATCH /api/agents/{id}/permission-grants with strict shared
validation and Board-only OpenAPI metadata.
- Serialize each mutation on the agent row. Preserve unrelated grants
and membership state. Commit the grant and its activity record together
and publish activity after commit.
- Document remote login and single-grant management.
## Verification
- The new approval-link regression failed before the fix and passes
after it.
- All 25 CLI auth route tests pass.
- All 72 affected CLI, login and OpenAPI tests pass.
- All 5 PostgreSQL grant tests pass, including actual commentary reads
(403 before grant, 200 after grant, 403 after revoke), preservation,
audit attribution, concurrent additions, invalid inputs, inactive or
missing membership, agent self-grants, cross-company denial, and
rollback on audit failure.
- Shared and CLI typechecks and server declaration-layer checks pass.
- pnpm verify requests broad CI coverage because doc/CLI.md is a global
verification input. Greptile reviewed final head
|
||
|
Dotta
|
1b7fe33432 |
fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. **What happened?** Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. **Expected behavior** Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. **Steps to reproduce** 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. **Paperclip version or commit** The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. **Deployment mode** Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
0cf70c9d0c |
fix(chat): tell overviews to state when the picture is from (#20301)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask the CEO chat agent for a quick picture of what the company is doing now > - Such a picture is only useful when the person knows how current it is > - The chat guidance asks for accurate status, but it does not ask for a time > - A live overview said "right now" and gave no time, so the person could not tell when it was observed > - This pull request adds one guidance sentence: state the picture's time as a UTC clock time, from the latest message timestamp > - The benefit is that overviews tell the person when their information is from ## Linked Issues or Issue Description Refs #19614 **What happened** A person asked the chat agent "Give me a quick picture of what the company is doing right now." The agent read agents, projects, tasks and approvals and gave a correct summary. The summary did not say when it was observed. The only time word was "right now". **Expected behavior** An overview of current work states when the picture is from, as a clock time. The native prompt already lists each new comment with its timestamp, so the agent has a time it can use. **Steps to reproduce** 1. Open a chat with the CEO agent in a company that has active, blocked and done tasks. 2. Ask "Give me a quick picture of what the company is doing right now." 3. The reply summarizes the work but gives no observation time. ## What Changed - `AGENT_CHAT_DIRECTIVE` status-update guidance: when the agent describes what the company, a project or agents are doing now, it states a UTC clock time, such as "As of 15:26 UTC". It uses the latest message's timestamp unless a record it read gives a later time. - A regression test checks that the directive contains this guidance. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-list.test.ts` passes (includes the prompt-size limits). - The chat-catalog eval `activity.company-overview` now grades a real observation time (stacked eval PR on `chat-eval/master-candidate`). This fix is cherry-picked onto that eval branch for the next live run. No live run has used it yet. ## Risks - Low risk. This is one guidance sentence. Overviews get one extra clause with a time. - The time is the message time, not a server clock read. Reads happen seconds after the message, so the difference is small. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
12523fb677 |
fix(chat): send a task whose automatic recovery stopped to the person's Retry (#20306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person asks the chat agent (the controller) to act on tasks that other agents own > - When a task's automatic recovery stops, for example after three runner losses, the task is blocked and the person gets a Retry control on it. Retry rechecks the task's gates and keeps the failed attempt's evidence > - The blocker told every reader "Try again or send a new message to continue". A controller read that as its own instruction. It found that Retry is board-only, then resumed the task with a comment and reassigned the task to itself > - That skips the assignee's budget and hold gates and hides the failed-run Retry from the person > - This pull request names Retry as the person's control in the blocker text, and tells the controller to point the person to it > - The benefit is that "try that task again" leads to the supported, gated Retry instead of an agent takeover ## Linked Issues or Issue Description Refs #20270 (automatic recovery after a lost local runner) **What happened?** In the chat eval `chat-features.runner-opencode.local.recovery.retry-exhausted`, a person asked the CEO agent to "Try the website task again" after the website task's native recovery was exhausted. The agent read the task's `executionBlocker` (`canRetry: true`, nextAction "... Try again or send a new message to continue once the previous execution has stopped."). It found that `POST /api/issues/{id}/retry` is board-only. It then posted a `resume: true` comment, which moved the task from blocked to todo, and reassigned the task to itself to re-run it. The task lost its failed-run Retry control, and the original assignee's gates were not checked. **Expected behavior** The agent links the original task, says why it stopped, and tells the person to use Retry there. It takes the work over only when the person explicitly asks for that. **Steps to reproduce** 1. Let a native run exhaust its automatic recovery (for example, three runner losses). 2. In the controller chat, ask it to try that task again. 3. The agent comments to resume the task, or reassigns it, instead of pointing to Retry. ## What Changed - `execution-blocker.ts`: the text that is added when a person can retry is now `EXECUTION_BLOCKER_PERSON_RETRY_ACTION`: "A person can use Retry on this task, or send it a new message, to continue once the previous execution has stopped. Agents cannot retry it for them." - `agent-conversations.ts`: a new chat guidance paragraph. When another task's automatic recovery has stopped, the supported next step is the person's Retry on that task. The controller links the task and explains why it stopped. It does not reassign the task, change its status, comment on it to resume it, or create a replacement task, unless the person explicitly asks it to take the work over. ## Verification - `board-access-denial.test.ts`: the directive contains the new rule. - `execution-blocker-read.test.ts`: a retryable blocker's nextAction ends with the person-Retry text. - 54 tests pass locally for these two files. - Paid live run of `recovery.retry-exhausted` with this change and #20270 on the eval branch: the run ID will be added below. ## Risks - Low. Only text changes. Any UI that shows the blocker's nextAction shows the new sentence instead of "Try again or send a new message ...". - No schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
97188f276c |
perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.
## Linked Issues or Issue Description
Refs #20190, #20285
**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).
**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.
**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.
## What Changed
- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
- New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
- The other `DurableRunnerConfig` literals set `turn_scoped: false`.
## Verification
- `cargo test -p paperclip-runner-core`:
- `--lib`: 602 passed.
- `--test acpx_park_preparation`: 22 passed.
- `acpx_park_readiness`: 8 passed.
- `acpx_warm_attachment`: 4 passed.
- `acpx_provider_suspend`: 5 passed.
- `acpx_provider_session`: 12 passed.
- `acpx_provider_turns`: 29 passed.
- `native_provider_backend`: 47 passed.
- `durable_recovery`: 3 passed.
- `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(
|
||
|
DottaandClaude Opus 5.5
|
189fefb991 |
perf(task-chat): warm the whole first frame before a task opens (#20286)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from the issues list and the inbox to continue a conversation with an agent > - The compact task chat shows its first frame only after about ten reads settle. These include recent comments, activity, recent runs, interactions, attachments and live runs > - Navigation prefetch warmed only the issue and its recent comments. The other reads started on the click, and the slowest one decided when the chat appeared. On staging that took 1.1–2.0 s > - This pull request warms the whole first-frame set when an open is likely, under the exact keys the page reads > - The benefit is that a task the person is about to open can render from cache. A pointer that only passes over rows does not add load ## Linked Issues or Issue Description **Problem or motivation** On a busy instance, the composer and the latest messages appear 1.1–2.0 s after a click on a task row. The page waits for about ten reads. Before the click, only two of them are warmed. **Proposed solution** Warm the rest of the first-frame reads when an open is likely: a mouse or pen press, a click, keyboard Enter, or a hover or focus that rests on the link for 120 ms. The page and the prefetch use the same fetchers and query keys. **Alternatives considered** - Warm everything on every hover. This sends about 7 requests for each row that a pointer crosses, so it was rejected. - Warm only on press. The time from press to click is short, so most of the wait would remain. - A single bootstrap endpoint. This is planned separately, because it changes the API contract. **Roadmap alignment** This is part of the task-loading performance plan in #20276. It follows #20277 and #20284. ## What Changed - New `ui/src/lib/issue-chat-history-queries.ts`: - The recent activity and recent run window fetchers moved here from `IssueDetail.tsx`, together with their query keys. The logic is unchanged. - `prefetchIssueChatHistory` warms activity, the run window, interactions, attachments and live runs. It warms the active run only for running work, as the page does. - Live state uses a 5 s stale time. The other reads use the existing 30 s navigation stale time. - The run-window prefetch uses the same conversation mode as the page. Thus a reconnect in an agent conversation reads only the watched runs by id. - `prefetchIssueCommentInputFacts` warms the input facts of the recent comments after those comments arrive. - `ui/src/hooks/useIssueIdBatches.ts`: new `prefetchIssueIdBatches` fills the same batch keys that the hook reads. The hook and the prefetch share one batching function. - `prefetchIssueDetailForNavigation` accepts `history: true` and then also warms that set, plus the comment input facts. - `IssueLinkQuicklook`: - A hover still warms the light set at once. The quicklook trigger does not start a history prefetch. - A hover that rests on the link for 120 ms warms the full set. Leaving the link before 120 ms cancels it. - Keyboard focus uses the same 120 ms dwell as a hover. Blur cancels it. - A primary mouse or pen press and a click warm the full set at once. - A touch can start a scroll, so it warms only the light set. The full set waits for the click. - The keyboard Enter handlers in `IssuesList`, `LegacyIssuesList`, `Inbox` and `LegacyInbox` pass `history: true`. - `IssueDetail.tsx` uses the shared fetchers and keys. ## Verification - `cd ui && npx vitest run src/lib/issueDetailCache.test.ts src/components/IssueLinkQuicklook.test.tsx src/hooks/useIssueIdBatches.test.tsx` passes 29 tests. These are the new ones: - A `history` prefetch fills every key that the compact chat waits on, with the same run-window shape that the page builds. - A `history` prefetch fills the comment input-fact batch keys that the chat reads. - In an agent conversation, a reconnect reads the watched runs by id, not every run. - Prefetched id batches render with no loading state on the first render. - The active run is read only for work that is running. - A light prefetch makes no history requests. - A pointer that passes over a link in less than 120 ms warms only the light set, with the quicklook on and off. - A hover that rests on a link warms the full set. - A mouse press warms the full set at once. - A touch warms only the light set until the click. - Keyboard focus warms the full set only after it rests on the link. - The tests for the review fixes (quicklook hover, input facts, conversation mode, touch and focus) fail without the fix. - The `IssueDetail`, `Inbox`, `IssuesList` and `LegacyInbox` suites pass: 4 files, 412 tests. - `pnpm typecheck` in `ui` shows no new errors. The only errors are existing ones from an unbuilt `plugin-sdk` package in this worktree. - The staging harness from #20276 will measure the gain after deploy, for a hover, a click, and a revisit. ## Risks - More requests when an open is likely: up to 7 extra reads for each task that is pressed with a mouse or pen, or hovered or focused for 120 ms. These are the same reads that the page would send on open, so a completed open costs nothing extra. A hover that does not lead to an open costs those reads. - Freshness: a prefetched result can stay fresh for up to 30 s (5 s for live state). Websocket invalidation still refreshes these keys, as on a revisit today. - No API or schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
90404d2033 |
perf(run-logs): read object-store log tails in parallel, once, and cached (#20284)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted instances keep run logs in object storage as a journal of
content-addressed segments and binary index nodes
> - The task chat reads the last 32 KB of a run's log, aligned to a
record, to show what the agent did
> - A run that flushes often has many small segments. The tail read
fetched each object one after another, and record alignment walked the
index a second time
> - On staging, one tail read took 11–12 s, and the task chat waited for
it (see #20277)
> - This pull request reads the tail in parallel, in one pass, and
caches the objects that cannot change
> - The benefit is that the first tail read is about 6.6x faster, and
repeated reads (live polling, reopening a task) use only the mutable
head
## Linked Issues or Issue Description
**What happened?**
`GET
/api/heartbeat-runs/:runId/log?offset=…&limitBytes=32000&alignStartToRecord=true`
took 11–12 s on staging for a 400 KB object-store log. Without alignment
it took 6.1 s. A read from offset 0 took 2.0 s. The task chat waited for
this read before it showed the run's output.
**Expected behavior**
A 32 KB tail read returns in well under a second, and a repeated read of
the same tail costs about one storage round trip.
**Steps to reproduce**
1. Use object-backed run logs (`s3_segments`).
2. Write a run log in many small flushes, as a live run does.
3. Read its last 32 KB with `alignStartToRecord=true`, and count the
object GETs and how many run at the same time.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
099adb2728 |
fix(recovery): settle a remote workspace save that stopped without finishing (#20291)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.
**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.
**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
ef2b8d5324 |
fix(issues): make every Retry start work or say why (#20283)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a task's run fails, the board recovers it with a Retry button, so that button must either start work or say why it did not > - An audit of every Retry-type button found three gaps. The task Retry shown during a scheduled retry's countdown returned "accepted" but changed nothing. A stale Retry looked like success in the Inbox. "Retry now" skipped the task's read and change checks, so a member who cannot see a private task could start its retry > - This pull request makes the task Retry start a scheduled retry immediately, makes every Retry that did not start work say why, and gives "Retry now" the same authorization as the task Retry > - The benefit is that no Retry click silently does nothing ## Linked Issues or Issue Description Refs #19831. The board reported a dead Retry button. A read-only audit of every Retry button confirmed: - **Task Retry during a scheduled retry was a no-op.** `TaskInterruptionPresentation.canRetry` is true in `retry_scheduled`, so `ExecutionBlockerNotice` shows Retry. `requestTaskRetry` treats `scheduled_retry` as busy and inserts a `coalesced` wake pointing at that run. `resumeTaskRetry` reads only `deferred_issue_execution`, and dispatch uses coalesced user wakes only to carry comments. Nothing promoted the run: the API answered 202 `accepted: true` and the countdown did not change. - **A stale Retry looked like success.** `accepted: false, stale: true` comes back as HTTP 200. The Inbox ignored `accepted` and opened the task as if work had started. The task page showed no reason. - **"Retry now" checked less than the task Retry.** It ran only `assertBoard` and `getAccessibleResource`, skipping `assertIssueReadAllowed` and `issue:mutate`. A company member who cannot read a private task could promote that task's scheduled retry. - **"Retry now" said nothing for some outcomes.** It toasted only `promoted` and `gate_suppressed`. ## What Changed - `server/src/services/task-retry.ts`: when the Retry coalesces into the task's latest run and that run is `scheduled_retry` (with no other Retry already active), the receipt returns `scheduledRetryRunId`. - `server/src/routes/issues.ts`: - `POST /issues/:id/retry` then promotes that run through the existing `retryScheduledRetryNow`, the same path "Retry now" uses. It logs `issue.scheduled_retry_retry_now` with `source: "task_retry"` and returns `scheduledRetry: { outcome, message }`. - `POST /issues/:id/scheduled-retry/retry-now` now runs `assertIssueReadAllowed` and `decideIssueAccess("issue:mutate")`, matching `/retry`. A private task the member cannot read now answers 404. - `ui/src/hooks/useTaskRetry.ts`: new `taskRetryNotStartedReason`. When a Retry did not start work (stale, or a scheduled retry that is gate-suppressed, waiting, or gone), the hook returns that reason as its error. `ExecutionBlockerNotice` and `StalledDependencyNotice` already display it. - `ui/src/pages/Inbox.tsx`: a task Retry that did not start work shows the reason in the error toast and does not navigate. - `ui/src/hooks/useRetryNowMutation.ts` and `IssueScheduledRetryCard.tsx`: every "Retry now" outcome now shows its headline and message, including `no_scheduled_retry` on the card. - `ui/src/api/issues.ts`: `retryTask`'s response type includes `scheduledRetry`. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-scheduled-retry-routes.test.ts`: - "starts a scheduled retry now when the task Retry is clicked during its countdown": 202 with `scheduledRetry.outcome: "promoted"`, the same run becomes `queued`, and no second retry run is created. - "refuses Retry now on a private task the member cannot see, like the task Retry": on master "Retry now" answers 200 `promoted`; now both routes answer 404 and the run stays `scheduled_retry`. - The seed now adds the board user's membership, which the stricter route needs. All 33 tests pass. - `ui/src/hooks/useTaskRetry.test.tsx`: a replaced attempt, a gate-held scheduled retry and a waiting scheduled retry each show their reason; a promoted one shows nothing extra. - `ui/src/pages/Inbox.test.tsx`: an Inbox Retry that did not start work (stale, or gate-held) toasts the reason and does not navigate. - `ui/src/components/IssueScheduledRetryCard.test.tsx`: "explains when there was no scheduled retry left to start". Other Retry buttons: the chat failed-run Retry already reports stopped and failed retries (`agents.test.ts`). Agent lifecycle "Retry now" and cost "Retry accounting" start no task work and already surface their API errors. Regression and static checks: - UI: 7 files, 168 passed. Server: task-retry and scheduled-retry suites, 149 passed. `issue-agent-mutation-ownership-routes.test.ts` passed. - `maintenance-server-routes.test.ts` "assembled maintenance server production wiring" fails identically on master on this Mac (`ensure-build-deps`). - `tsc --noEmit` passes for `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A task Retry during a scheduled retry now starts the run immediately, through the existing "Retry now" promotion gates (pause, budget, cleanup waits). A gate that holds it is reported, not bypassed. - "Retry now" is stricter: a member without read or change access to the task gets 404 or 403, as on `/retry`. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fd2e7482c0 |
fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop
## Linked Issues or Issue Description
Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).
**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.
**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.
**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
e015c2ef21 |
fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path - #20145 prepares the next Claude provider generation at park, and #20190 restarts the ACPX provider when a turn settles. They move the sidecar respawn off the follow-up's critical path. - When that turn-end restart fails, the runner "fails closed": `restart_session_after_turn` saves the session `closed` and drops it. Within the run, the next turn fails exactly like a failed restart at turn start. That's intended. - Across runs it breaks the conversation: - the next run's `run.attach` refuses a `closed` session ("run.attach requires the same settled ACPX provider profile and session"); - it reconnects for about 10 s; - the controller then records a continuity break and starts a fresh Claude conversation (`run.startup.reuse provider_session: fresh/created`). - A local A/B (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`, on a host with load average about 17–27) confirmed it: | | |
||
|
Dotta
|
5c73633a8f |
fix(workspaces): preserve remote recovery holds during online import (#20298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace import preserves old controller-local files while unrelated remote work continues. > - A task can have an old local checkout and a separate durable remote workspace. > - A terminal remote run can retain a file-recovery hold on that task. > - The legacy source checker rejects remote references, so that hold blocks independent local imports. > - This pull request proves that the hold belongs only to the separate remote source and fences its finalizer during import. > - Old files can migrate while the remote recovery obligation stays intact. ## Linked Issues or Issue Description Refs #19873 and #19854 for the existing cohort and source-verification paths. Searched related online migration and remote recovery PRs; no duplicate fix was found. **What happened?** Online import rejects a legacy Git checkout with `online_workspace_source_scope_unproven` when a same-task failed run has a prepared reference to a separate durable remote workspace. The legacy run reader rejects remote references before the hold can be classified. **Expected behavior** Independent local files can import under their source fences. The separate remote workspace, dirty writer, allocation, accepted result and recovery hold must stay intact. **Steps to reproduce** 1. Create related local Git workspaces. 2. Record a terminal failed native run and an exhausted file-recovery hold for a separate durable remote workspace on the same task. 3. Import the local leaf online. The old checker refuses before capture. **Paperclip version or commit** Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`. **Deployment mode** Managed server with PostgreSQL and remote Runner execution. ## What Changed - Add a distinct remote-only preservation proof. Keep remote runs excluded from legacy export sources. - Use the canonical source inspector and prepared-reference matcher. Bind the run, finalizer, action, task, accepted result, authority, allocation and prior committed snapshot. - Acquire the held run's finalizer fence through the existing sorted lock path. Existing source-proof reinspection rejects changes before publication. - Preserve previous-release prepared references with an older baseline snapshot. Do not require that snapshot's epoch or source run to equal the current run. - Add real PostgreSQL import, authenticated cold restore, preservation, refusal, lock-contention and publication-race regressions. Document the behavior. ## Verification - Genuine baseline failure on unchanged product: the real importer refuses with `online_workspace_source_scope_unproven`. - Final focused regression file: 23 tests passed in 13.19 seconds through the canonical isolated PostgreSQL wrapper on Node 24.22.2. - 24 existing affected migration/recovery tests passed against the same final production bytes. Another 83 tests were explicitly excluded by the focused selection; no full-suite pass is claimed. - The positive imports local bytes, confirms the remote finalizer fence is held during upload, leaves all remote records unchanged, parks the original repository and authenticates/restores the archive. - Refusal and race cases cover malformed, foreign, same-local and mixed references; active owners/writers; changed allocation, result, task, action, authority and baseline; and finalizer contention. - Server TypeScript check, module-boundary check and `git diff --check` passed. - An intermediate run had one incorrect assertion that expected an explicit empty legacy-source array. The canonical journal omits that empty field. The final assertion accepts absence and still forbids any exported remote source. Final relevant results are 23 new plus 24 existing passes, rather than a claim that the earlier invocation passed in full. - Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`: 55 successful checks and 4 intentional skips. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42650025033). Greptile reviewed that head at 5/5 with no actionable findings or open inline/review comments. Hosted continuation remains pending normal delivery. ## Risks - A false disjointness proof could admit an unsafe local import. Strict company/task/source binding, the canonical inspector, finalizer fencing and publication revalidation retain refusal on uncertainty. - This does not recover or settle remote files. Dirty authority and the original recovery obligation remain unchanged. - No schema, public API, archive format or saved-run identity changes. Existing persisted state carries forward. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
d10c6e0e8b |
fix(chat): hold a chat message at the agent's daily limit instead of dropping it (#20292)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat sends each person's message to the conversation agent through a durable outbox (`deliverConversationComments`). The outbox keys each wake receipt as `conversation-comment:<commentId>`. > - The outbox treats any receipt with that key as delivered. Queue admission writes a `skipped` receipt when the agent reached its daily run or cost limit. > - So a chat message sent after the agent used its daily allowance was consumed silently. No run started, the chat showed nothing, and nothing resumed it. > - A person's message must never be silently dropped. The daily limit is an operator control, so the message must wait honestly, not bypass the limit. > - This pull request makes the outbox hold the message at the daily limit, like it already does for a paused agent, and resume it when the limit resets. > - The benefit is that a chat message at the daily limit is saved, shown with its reason, and sent automatically at 00:00 UTC. ## Linked Issues or Issue Description No public issue exists. Inline bug report: **What happened** In a chat eval, the conversation agent (CEO) had `maxDailyRuns: 3`. After three chat turns and a separate failed-model phase on another agent, the person sent a fourth message. The comment was saved (POST 201), but no CEO run started. The conversation stayed `active` with no `executionRunId`. The ownership reconciler later reported `I7_visible_next_action`. The server log shows no wake error. Queue admission had written a `skipped` receipt (`heartbeat.daily_run_limit`) under the outbox key, so the outbox counted the message as delivered. **Expected behavior** A chat message that reaches the agent's daily limit is not consumed. It stays saved, the chat shows why it waits, and it is sent automatically when the limit resets. The daily limit still applies: the message does not start a run early. **Steps to reproduce** 1. Set a conversation agent's `runtimeConfig.heartbeat.maxDailyRuns` to N. 2. Send N chat messages and let each turn finish. 3. Send one more chat message. 4. See that no run starts, the chat shows no waiting state, and the message is never answered (a `skipped` wake receipt with reason `heartbeat.daily_run_limit` exists for `conversation-comment:<id>`). ## What Changed - `server/src/services/agent-conversations.ts`: add `conversationDeliveryHold`, which applies the same daily run/cost limit check as queue admission. The outbox returns `deferred` at the limit and writes no receipt, so the message stays owed. It reports the next UTC day start through a new optional `onDeferredDeadline` option. - `server/src/services/recovery/service.ts`: stranded-work recovery passes its `onDeferredDeadline` to the outbox, so the held message is retried when the limit resets. - `server/src/services/heartbeat/queue.ts`: if the limit is reached while an outbox wake is in admission (a race), admission throws a 409 with `reason: heartbeat.daily_*` for `conversation-comment:` keys instead of writing a `skipped` receipt. The outbox already maps a matching 409 to `deferred`. Other wakes keep the current skip behavior. - `server/src/routes/issues.ts` (`GET /issues/:id/queued-comments`): when a chat has no queue receipt but has messages held by the daily limit, return them as a `deferred` queue with `executionWait` (reason and message). Edit, send and discard stay off because the outbox owns delivery. - `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: show the `executionWait` message for such an outbox-held queue (no queue id and no target run). Receipt-backed queues keep the current behavior and do not duplicate the task status. - Tests: three real-database tests in `agent-conversation-delivery.test.ts` (hold and resume after the limit resets, recovery deadline, and a limit reached during real `heartbeatService.wakeup` admission) and one UI test. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversation-delivery.test.ts`: 10 passed. Without the source change, the three new tests fail (`expected 'delivered' to be 'deferred'`; the real queue path writes the skipped receipt). - `cd server && pnpm exec vitest run src/__tests__/issue-recovery-conversations.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-undispatched-execution-wait.test.ts src/services/heartbeat/queue.test.ts src/__tests__/agent-chat-messages.test.ts`: 215 passed. - `cd ui && pnpm exec vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx`: 36 passed. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: clean. `node scripts/check-token-gates.mjs`: clean. - Manual: give a chat agent `maxDailyRuns: 1`, send two messages. The second message shows in the queued messages bar with "<agent> reached its daily run limit. Your message is saved and sends automatically when the limit resets at 00:00 UTC." ## Risks - Behavior change for chat only: a chat message at the daily limit now waits in the outbox instead of being consumed. It is sent after 00:00 UTC. If an operator raises the limit, the message is sent on the next recovery pass for that chat; the UI message does not promise an earlier send. - Saved state from the previous release: no persisted shape changes. Messages that an earlier release already consumed with a `skipped` receipt stay consumed. They are not replayed later, which avoids sending old messages without notice. - The queued-comments read adds one agent read and one daily-limit count for a chat with no queue receipt. This is the same query that admission runs. - The recovery deadline is in-process; instance-sleep scheduling does not get a new deadline from this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
87e755ce8e |
fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes keep a warm sandbox and hand it to
the next run through a chain of reusable leases
> - Acquisition binds that chain to the execution workspace, so a task
can inherit a warm sandbox from an earlier task in the same workspace
> - Saved-files recovery verifies the same chain, but it required every
earlier lease, run, and finalization to belong to the recovering task
> - A valid chain that started with another task therefore always failed
with `allocation_ownership_ambiguous`. That code is `service_required`,
so recovery stopped after one attempt, and each board Retry repeated the
same refusal
> - This pull request makes the chain check match the acquisition rule,
and it checks each predecessor against its own task
> - It also makes the board say plainly when recovery refuses the saved
state, because "failed after automatic retries" was not true and Retry
looked like it did nothing
> - The benefit is that a task blocked with "Saving the agent's files
failed after automatic retries" can recover its saved files, and a
refusal that Retry cannot change is shown with its reason
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn, and its result was
accepted. The copy of its files back to the host failed and was
scheduled to retry. A graceful server restart then interrupted the
retry, so the run settled with
`native_workspace_sync_out_retry_exhausted`, and the task became
blocked. Every saved-files recovery generation then failed on its first
attempt with `allocation_ownership_ambiguous` (`service_required`). The
board showed "Saving the agent's files failed after automatic retries.
Retry to try again". Each Retry replaced the generation
(`issue.recovery_generation_replaced`), and the new generation failed in
the same way within one second. To the user, Retry did nothing. The
sandbox lease chain was valid: same agent, same execution workspace,
same sandbox, and matching handoff receipts on all eight leases. But its
first three leases belonged to an earlier task in the same workspace.
**Expected behavior**
Recovery must accept a warm sandbox chain that acquisition accepted.
Acquisition binds reuse to the execution workspace, and it requires the
same task only when the lease has no workspace. Recovery must check each
predecessor run and finalization against that predecessor's own task. It
must still refuse a chain whose records disagree or that has no shared
workspace.
**Steps to reproduce**
1. Run a native task in a remote sandbox with `reuse_by_environment`,
with an execution workspace that another task also uses.
2. Run the other task in the same workspace first, so the sandbox lease
is handed from that task to this one.
3. Make the last run of the second task fail to copy its files back (for
example, restart the server during the copy-back retry).
4. Wait for saved-files recovery, or click Retry. Recovery fails with
`allocation_ownership_ambiguous`, and Retry repeats the failure.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
6958cb112c |
fix(ui): keep responder avatars visible across stale execution outcomes (#20293)
## Thinking Path > - Paperclip lets people manage agents and send follow-up messages in task chat. > - The responder identity should stay visible while a message starts a run. > - Comment storage, task reads, and run discovery finish separately. > - An older intent can resolve during admission of the new message. > - A reopened task can also retain its previous terminal status until the next read. > - This change keeps those older outcomes from clearing the new startup feedback. > - The avatar stays visible until the new run or its own admission outcome arrives. ## Linked Issues or Issue Description Related fixes: #17885 and #18312. A search found no other PR for these races. **What happened?** After Send, the agent avatar can disappear before the new run appears. A prior intent changing from working to resolved clears the new submission. Sending a follow-up that reopens a done or cancelled task also clears startup feedback against the old task status. **Expected behavior** Keep the responder visible through comment acknowledgement and run discovery. Stop startup feedback on a current failure, blocker, or completion. **Steps to reproduce** 1. Send a follow-up while a previous intent is still reported as working. 2. Let that same intent resolve before the new run read completes. 3. Observe the responder disappear. Alternatively, send a follow-up to a done or cancelled task and delay the reopened task read. **Paperclip version** Reproduced on source master at `33a23e0246`. Rebased onto `3ef45fdd92` before submission. **Deployment mode** Local component tests reproduce the read ordering. An isolated local trusted instance verifies the real Send-to-process-run path in Chromium. It does not invoke a live model provider. No staging message was sent. ## What Changed - Record the previous intent identity. Its later resolution cannot acknowledge the new message unless the observed run belongs to that message. - Pass the existing reopen request into shared submission state on both chat surfaces. - Keep startup feedback through the previous terminal status. Restore ordinary completion handling once the reopened status arrives. - Add six Send regression cases, a live-run handoff assertion, and coverage for later completion and ordinary terminal notes. - Clarify identity continuity in DESIGN.md. ## Verification - Red: all six new Send cases fail on unchanged master. - Green after rebase: 531 tests pass, one existing test is skipped, across TaskChatThread, IssueChatThread, and useRunnerSubmission. - Chromium: the existing compact-startup responder test passes on a disposable local instance. Frame sampling checks that the same identity stays mounted through acknowledgement and process startup. - UI typecheck and token gates pass. Initial typecheck lacked generated plugin SDK declarations; it passed after local server setup built that dependency. - Affected checks passed after rebase on `de9ed52c0b`. Final CI on `de9ed52c0bd6a6f7738614af31aec1eae2b871d0`: 59 checks passed, four skipped, no failures. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42646038090). Greptile and Paperclip review both gave 5/5 with no actionable findings or unresolved review threads. ## Risks Low scope: client startup feedback only. An old intent must not hide a new admission, but a new failure must still end it. Existing backoff and failure tests remain covered. The reopen exception applies only to an explicit reopen and ends when the reopened status arrives. No API, database, permission, or provider behavior changes. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed in this session. Capabilities used: reasoning, source editing, shell execution, and browser verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
052c638b51 |
fix(environments): honor stopped-task proof after warm preparation (#20289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm plugin Runner retains its task for later runs. > - A follow-up can find that the retained task has stopped. > - The resume path already asks settle to check status before it sends stop. > - A saved startup marker disabled this order, even after warm preparation had finished. > - This pull request applies the same stop-proof check to all observed-stop releases. > - The next run skips a repeated stop when status proves both execution and required preparation have stopped. ## Linked Issues or Issue Description Fixes: #20224 Refs #20203 ## What Changed - Remove the startup-marker exclusion from the existing status-first path. Keep the existing shared confirmation check. It still requires `preparationStopped: true` for startup work. - Keep stop followed by status when the first status lacks the required proof. Keep the startup drain, credential writeback, complete, and cleanup failure handling. - Add retained-task coverage with the startup metadata that warm preparation saves. Cover a startup cleanup whose first status lacks preparation proof. - Update the plugin task lifecycle contract. ## Verification - Local macOS, source base `3e8744fa91`: from `server`, `pnpm exec vitest run src/__tests__/environment-task-runtime.test.ts src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts` passed: 3 files, 82 tests. The integration tests use real embedded PostgreSQL and controlled plugin results. - After restoring the fix, the two affected files passed again: 73 tests. - Independent parent verification on commit `19f720ff68`: the two files filtered to the changed startup-proof and retained-task cases passed (4 tests). Independent source review found no actionable issue. - Baseline proof: with only the source condition restored to upstream, the new retained-task test with saved startup metadata fails because it sees an extra `stop`. The case without that metadata passes. The source fix was then restored. - The retained-task regression asserts exactly `environmentResumeLease`, `status`, `complete`, and `environmentAcquireLease`, with both task operations bound to the old task. The startup test also proves that missing preparation proof sends `stop` and a second `status`. - Test setup: `node scripts/agent-preflight.mjs --skip-rust-debug --explain` installed dependencies, then stopped because Cargo is unavailable on this Mac. Initial test collection failed because the plugin SDK output was absent. `pnpm --filter @paperclipai/plugin-sdk build` built that required dependency and the shared package; the targeted test run then passed. - Standard Greptile reviewed commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a` at 5/5 with no review threads. [Review](https://github.com/paperclipai/paperclip/pull/20289#issuecomment-6743829614). - Full local checks were not run, as requested. [GitHub PR workflow 42644023868](https://github.com/paperclipai/paperclip/actions/runs/42644023868) passed on final commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a`. It supplied broad typecheck, build, and test coverage. The separate Snyk status passed. Optional Storybook checks were skipped. - The change removes one provider stop call on the reported path. The issue's measured eight calls would become seven if other calls stay the same. The earlier five-call target is not claimed. No deployed latency measurement is claimed. ## Risks - A provider that cannot prove preparation stopped still gets stop and status. That fallback can cost one extra status call, as the existing status-first contract already allows. - A status failure keeps cleanup pending and blocks replacement. No termination proof or credential handoff is skipped. - No schema, API, dependency, or timer change. No added work on successful task reuse or ordinary task dispatch. - Live provider acceptance remains unrun. The local evidence checks orchestration and stop-proof handling. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository inspection, shell tools, and code execution. The exact serving model ID and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
09412e652a |
fix(runner): allow warm release to drain after provider restart (#20290)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm Runner sessions park after a turn so follow-up tasks can claim the same process. > - Claude restarts its ACPX provider after the controller receives the terminal result. > - That restart can delay the drain receipt beyond the release path's one-second limit. > - A failed drain prevents parking, so the idle timer later closes the retained Runner. > - This pull request uses the existing warm admission budget for release drain. > - The benefit is retained Runner reuse without moving provider restart onto task dispatch. ## Linked Issues or Issue Description Fixes: #20225. Refs #20190 and #20145. Related performance follow-up: #20224. ## What Changed - Align the release drain timeout with the existing 30-second warm admission budget for every provider. Close paths keep their explicit remaining deadlines. - Keep the existing immediate completion when the drain receipt proves settlement. Add no sleep, retry, provider branch or dispatch work. - Add a real Runner regression with a fake ACPX sidecar whose replacement start takes 1.5 seconds. One release call must park the Runner, then a follow-up must claim the same Runner process. - Document the release drain budget. ## Verification - Targeted command in `packages/paperclip-runner`: `vitest run src/live/runnerd-codex-transport-release-resume.test.ts src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed** across 3 files, 157.78 seconds. - Independent parent verification reran the slow-restart and ordinary cached cases: **2 passed**. Source review confirmed that release is the only caller using the changed default; close callers supply explicit deadlines. - Negative control: with the original one-second timeout, the new test fails because the single release call returns `false`. The fixed code passes the new case. - Built only the current-source Runner and two fixture binaries required by these tests. The first sandboxed invocation could not bind its loopback listener (`EPERM`); the test ran with loopback access afterward. - Full GitHub PR workflow [42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993) passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard Greptile scored this same head 5/5 with no review threads. Full repository checks ran in GitHub CI, per the requested local test scope. No live Claude session or hosted deployment was tested. ## Risks - An unresponsive Runner can now hold release drain for up to the existing 30-second warm admission limit. A healthy Runner returns as soon as its receipt arrives. The provider restart stays at turn end; task dispatch gets no fixed delay. - Saved state, protocol and provider identity formats do not change. Previous-release state carries forward under the existing validation. - The controlled fake-sidecar test proves the restart timing race and process reuse. It does not qualify live provider latency or a hosted rollout. ## Model Used - OpenAI GPT-6 through Codex, with repository tools, code editing and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
dc9ad17515 |
perf(storage): time workspace snapshot save and restore steps and count storage retries (#20280)
## Thinking Path
- Cloud durable storage is a major source of failures and slowness on
staging. For example, a staging task failed when saving files after
automatic retries.
- Before measuring or fixing it, we need to see where storage time goes.
Today a saved-workspace restore before the turn appears only inside the
coarse `workspace_environment_realize` phase.
- After the turn, the save's encrypt, put, verify and multipart throttle
backoff have no timing at all. Only `captureMs`, `transferMs`,
`verifyMs`, `uploadMs` and `publishMs` exist, in `saveTiming`.
- Storage retries happen in the SDK and in the multipart uploader,
invisibly.
- This adds the measurement first, so the storage-mode chaos waves
(MinIO behind a fault proxy) can report save and restore p50/p95 and
point at the dominant wait.
## Linked Issues or Issue Description
**What happened?**
Durable-storage saves and restores were opaque in the run log and save
timing. A slow or repeatedly retried storage call could not be told
apart from slow capture, encryption or sandbox work.
**Steps to reproduce**
1. Run a remote task with S3-compatible durable storage, so it saves
after the turn and restores on the next run.
2. Inspect the run log and the `remote_workspace_snapshot` operation's
`saveTiming`. The restore's download, decrypt and verify, and the save's
put, verify and throttle backoff, have no timing.
**Expected behavior**
Each storage step has a duration, and storage request retries and
backoff are counted.
## What Changed
- `server/src/storage/request-stats.ts` (new): an async-local collector
of storage request statistics. Counts only:
- `requests`, `attempts` and `failedRequests`;
- `retryDelayMs` (SDK waits);
- `parts`, `throttled` and `backoffMs` (the multipart uploader's waits).
- `server/src/storage/s3-failure.ts`: `withS3FailureDiagnostics`, which
wraps every S3 send, records each request's SDK `$metadata.attempts` and
`totalRetryDelay`, on success and on failure.
- `server/src/storage/s3-multipart.ts`: records each uploaded part and
each resend wait, flagged throttled when the backend returned 503, 429
or SlowDown.
- `server/src/services/remote-workspace-snapshot-storage.ts`:
- Each step is a `storage.snapshot.*` run phase:
- save: `prior_attempts`, `encrypt`, `digest`, `put`, `verify`;
- restore: `download`, `decrypt`, `verify_plaintext`, `publish`.
- A caller can pass a `timing` record that receives the step durations,
the bytes and the request statistics.
- `server/src/services/remote-workspaces.ts`:
- The save passes a timing record, kept in `saveTiming.storage`.
- The sandbox unpack during a restore is its own phase,
`workspace.snapshot.sandbox_restore`.
- `execution-preparation-timing.ts`: the new phase names.
- `doc/run-log-events.md`: a new "Durable storage timing" section.
Because the restore runs inside run preparation, its steps of 50 ms or
more become `run.performance.span` rows with no extra plumbing. This is
run-log and instance-database data only; there's no Telemetry change.
## Verification
- `server`: `storage-s3-multipart-backoff`, `storage-s3-multipart`,
`storage-s3-multipart-throttle`, `storage-s3-failure-diagnostics`,
`remote-workspace-snapshot-storage` and `execution-preparation-timing`
tests: 86 passed. They include:
- collected stats for a throttled multipart upload: 12 requests, 7
failed, 7 throttled, 35.5 s backoff and 3 parts;
- a save and restore that fill every step duration and the byte count.
- `remote-workspaces.integration.test.ts`: 112 passed. Its `saveTiming`
assertion now includes `storage`.
- Server typecheck: no errors in the touched files.
## Risks
- **Overhead:** one async-local lookup per S3 request, plus timestamps
per snapshot step.
- **Collector scope:** the collector only exists inside a snapshot save
or restore. Other storage callers are unaffected; they record nothing.
- **`saveTiming`** gains a `storage` object. Readers that compare it
exactly (one test, updated) will see the new field.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
3ef45fdd92 |
fix(recovery): let a new task wait for its manually paused assignee (#20281)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A person can pause an agent. Tasks assigned to a paused agent wait until the person resumes it. > - Stranded-task recovery looks for assigned work that cannot continue and moves it to `blocked` for the board. > - A task created for a manually paused agent has no run yet. Recovery saw an assignee that is not invokable and moved the new task to `blocked` a few seconds after creation. > - The person paused the agent on purpose. The blocked task and its "the board must choose the next action" comment fight that decision, and the person must unblock the task by hand after resume. > - This pull request lets new `todo` work with no run wait while a manual pause lasts. The existing assigned-todo dispatch starts it when the agent is resumed. > - The benefit is that work queued for a paused agent starts on resume, as the pause promises. ## Linked Issues or Issue Description **What happened** A chat evaluation pauses an agent (Morgan) and then runs `paperclipai issue create ... --assignee-agent-id <Morgan> --status todo`. The CLI returns the task with status `todo`. Within about five seconds the task changes to `blocked`, with the system comment "Paperclip cannot safely continue automatic recovery because the original assignee is not invokable. The source assignment is unchanged and the board must choose the next action." The server log shows only the expected 409 "Agent is not invokable" for the assignment wake. **Expected behavior** The task stays `todo` while the person keeps the agent paused. When the person resumes the agent, the task starts once. A comment and a `blocked` status are not added. **Steps to reproduce** 1. Pause an agent from its page. 2. Create a `todo` task assigned to that agent. 3. Wait for the stranded-task recovery sweep. See that the task is `blocked` with the comment above. ## What Changed - `server/src/services/recovery/service.ts`: in `reconcileStrandedAssignedIssues`, skip a `todo` task with no run when its assignee is paused with a `manual` lifecycle hold. The existing rule above it already skips work whose run the pause cancelled. Other pause causes, such as a budget hold, keep the existing escalation. - `server/src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: a PostgreSQL test. The task stays `todo` with no comment and no run during two recovery sweeps. After resume it starts exactly once and finishes. A budget-held assignee still escalates. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 2 passed. Without the fix, the first test fails with `expected 'blocked' to be 'todo'`. - `npx vitest run src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-refused-assignment-wake.test.ts`: passed. - `src/__tests__/issue-tree-pause-resume-run.test.ts` had timing failures on this low-memory machine both with and without this change (different tests each time). Those tests do not pause agents, so this change does not reach them. CI runs them on this branch. ## Risks - Low risk. The skip needs all of these: the agent is paused, it has a `manual` hold, the task is `todo`, and the task has no run. Work that already ran keeps the current rules. - No persisted state changes shape. State saved by the previous release carries forward: a task that the previous release already moved to `blocked` stays `blocked`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20143 (a pause's stop never blocks the task behind an incident), #9998 (open: reject paused agents as assignment targets). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6db0bb9776 |
fix(chat): tell agents which CLI commands a person runs in their own terminal (#20267)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The chat guidance in `server/src/services/agent-conversations.ts` tells every chat agent how to answer. > - People often ask a chat agent for a terminal command: create a task, download a plan, inspect or stop a run, or point the CLI at a company. > - The guidance did not name the Paperclip CLI. Agents guessed: they did the work themselves, wrote `curl`, used placeholder company ids, or invented an API address. > - The person then has to fix the command, or gets a file in the agent's workspace instead of on their own machine. > - This pull request adds one guidance paragraph. It names the supported `npx paperclipai` commands, asks for real ids from the conversation, and asks for the person's own Paperclip URL as a placeholder. > - The benefit is a command the person can run as it is, with the right company and the right actor. ## Linked Issues or Issue Description **What happened** Three chat evaluations (cli-operations) asked for terminal commands. "Show me how to point my terminal at the Acorn company." got `context set` with `<acorn-company-id>` and an invented `http://localhost:3100`, although the agent can read its company id. "Download this task's plan from the terminal as a file." got a file saved in the agent's own workspace and an attachment link, with no terminal command. "Show me the command to inspect Morgan's last run and stop it if it is still running." got `curl` with environment variables that exist only in the agent's runtime. **Expected behavior** The agent gives `npx paperclipai` commands that the person runs in their own terminal. The commands use real ids that the agent can read in the conversation. The Paperclip address is the placeholder `<your Paperclip URL>`. The agent says which commands need a board user. **Steps to reproduce** 1. Open a chat with an agent. 2. Send "Download this task's plan from the terminal as a file." 3. See that the agent saves the file itself and gives no terminal command. ## What Changed - `server/src/services/agent-conversations.ts`: add a chat guidance paragraph for terminal and CLI requests. It names the supported commands for task creation, document download, run read and cancel, and context profiles. It asks for one command per line, real ids, and the `<your Paperclip URL>` placeholder. - `cli/src/__tests__/chat-cli-guidance.test.ts`: a new test reads the guidance and resolves every `npx paperclipai` command against the registered commander commands. It fails on an unknown command, flag or missing argument. - `server/src/__tests__/agent-conversations.test.ts`: check that the paragraph is in the directive and in the chat prompt. ## Verification - `cd cli && npx vitest run src/__tests__/chat-cli-guidance.test.ts`: 3 passed. - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts`: 80 passed. - The chat evaluations `cli-operations.choose-cli-context`, `export-document-cli` and `inspect-run-cli` run with this commit on an evaluation branch. Results are added in a comment. ## Risks - Low risk. The change adds prompt text and tests. Routes, permissions and responses do not change. - The chat prompt is about 300 words longer. - If a CLI command or flag changes, the new CLI test fails until the guidance changes too. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20029 (terminal sign-in guidance in the API catalog). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f7c85bca18 |
test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness boots a full temporary Paperclip instance per wave: embedded Postgres, workspaces and caches, often 5–13 GB > - At wave end it deleted that instance only when the wave passed with no harness error and no `--keep` > - Almost every chaos wave has at least one failing task, so almost every instance stayed on disk > - Over Nov 6–9, 359 `paperclip-runner-e2e-*` directories filled the machine's disk (ENOSPC), which stopped a running storm wave > - This pull request deletes the instance at the end of every wave, smoke and baseline run unless it is explicitly kept or its processes could not be stopped, and expires kept instances after 6 hours > - The benefit is that chaos waves can run back to back without filling the disk ## Linked Issues or Issue Description **What happened?** `main.ts` kept the temporary root whenever `keep || anyFailure || harnessError`. A failing task is the normal output of a chaos wave, so instances accumulated until the disk was full. The codex K13 storm w-20261109T113925-s9403 died after about a minute with every task cut off. **Steps to reproduce** 1. Run any wave in which at least one task fails. 2. The `paperclip-runner-e2e-*` directory under the OS temp dir stays behind, with no expiry. **Expected behavior** The instance is removed when the wave ends. Evidence is already in `results/`. Only an explicit `--keep`, or a stop failure that may leave processes running, keeps it, and kept instances expire. **Actual behavior** Every failing wave leaks its instance permanently. ## What Changed - New shared `tests/runner-e2e/instance-retention.ts`: - `releaseInstance` deletes the root unless `--keep` was passed or the processes could not be stopped. In those cases it writes a `.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets the directory to 0700. An optional `retain` list keeps small evidence files and removes everything else. - `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*` directories that carry the marker, are older than 6 hours, and have no live `postmaster.pid` anywhere inside them, and that no live process command line references. Each root is made removable first, and one failed delete does not stop the sweep. Unmarked directories are never touched, so existing directories from older runs stay until someone decides on them. `recovery_state` markers never expire. - `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets it. - Callers: - A wave whose Daytona cleanup is still pending (environment DELETE blocked, sandboxes still running after cleanup, or cleanup errors) keeps its instance as `recovery_state`, because its database owns that cleanup. - The wave releases its instance before writing the ledger and summary, so a failed report write cannot skip cleanup. - Restart chaos `main.ts` (crash-point suite included) releases the instance in its final cleanup, which also runs on failure and interrupts. The crash-point task-count check moved above supervisor creation so that error no longer leaks a fresh root. - `ttft-baseline.ts` releases after stopping the supervisor, including on SIGINT and SIGTERM. - The rolling-deployment and controller-replacement smokes now delete their roots, keeping only the evidence report, logs and screenshots. If cleanup is unconfirmed, the root is kept with a marker. - `launch.ts` marks its unconfirmed-remote-cleanup root as `recovery_state`, which never expires, and its `PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing deletion path is unchanged, and the file gets one line shorter. - README: the instance lifecycle and `--keep`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328 files, 6915 passed. The new retention tests cover: - deleting a finished instance, and retaining only listed evidence; - keeping it with a marker and mode 0700; - sweeping only marked, expired instances whose Postgres isn't running, while sparing fresh, live, recovery-state, unmarked and unrelated directories; - a stale `postmaster.pid`; - sweeping on `createRunnerE2ETemporaryRoot`. The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran the sweep. - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221 passed. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and `tests/restart-chaos`: clean. - No wave or smoke was run, because the disk is still nearly full. ## Risks - A failed wave's instance is no longer available for post-mortem DB inspection unless the wave ran with `--keep`. Evidence bundles in `results/` are unchanged. - The sweep relies on `postmaster.pid`. An instance whose Postgres was started outside the harness, without that file, could be removed after 6 hours. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Pavel-GuseynovandClaude Opus 5.5
|
4d297b998a |
fix(server): give remote MCP tool calls a 30 second default deadline and record stream failure evidence (#19768)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The tool gateway calls remote MCP servers over Streamable HTTP, and a server can answer `tools/call` with a `text/event-stream` body > - Such a server can send progress notifications before the response, and a slow tool can take more than 10 seconds > - The gateway aborts a remote MCP `tools/call` that uses the default deadline after 10 seconds, the general tool default > - When the call fails, the call evidence does not show how far the stream got, or whether the cause was the deadline or a broken transport > - This pull request gives remote MCP calls a default deadline of 30 seconds, the default response timeout of the remote HTTP client, and records the failure cause and the stream progress > - The benefit is that slow remote tools complete, and an operator can tell a deadline from a peer reset in the call evidence ## Linked Issues or Issue Description Refs #10147. That issue reports that a named MCP gateway aborts remote tools after 10 seconds and ignores the connection timeoutMs. This pull request raises the default deadline of a remote MCP `tools/call` to 30 seconds. It does not make the named gateway read the connection timeoutMs. Related: #11910 (open) keeps a connection callable after one timed-out tools/call. On current master a timeout and a JSON-RPC response error already keep the connection healthy, and this pull request keeps that policy. I searched pull requests and issues for "event stream tools/call", "SSE MCP tool call hang", "remote MCP timeout stream", "mcp text/event-stream" and "tool_timeout remote MCP". The only related open pull request is #11910. ## What Changed - `server/src/services/remote-http-fetch.ts`: the default response timeout is exported as `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` (30,000 ms, unchanged value). - `server/src/services/tool-gateway.ts`: - A remote MCP `tools/call` that uses the default deadline now gets `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` instead of the general tool default of 10 seconds. The Railway `run-command` budget and an explicit `timeoutMs` keep their values. - The call evidence records the response status, content type and request id before the body is read. A body that fails partway records the bytes received so far as its size. - A failed call records `failureKind` (`invocation_timeout`, `protocol_failure` or `transport_failure`), the deadline and whether it was aborted, the stream progress, and the error with its cause chain. Each string in the chain is redacted with `redactSensitiveText`, and a cyclic cause chain is cut with a "[Circular cause]" marker. - A deadline abort is a 504 `tool_timeout` and starts a fresh MCP session on the next call, as before. The connection health policy of master does not change. - A deliberate stop of a call (`elicitation_required`, `provider_interaction_required`, `oauth_insufficient_scope`, `oauth_refreshed_retry_required`) keeps its own reason when the deadline passes while it is recorded, as on master. An error that is already a gateway error also gets a failure kind: `tool_timeout` is `invocation_timeout`; a malformed, invalid or oversized response is `protocol_failure`; and `remote_http_*` (for example a DNS failure) and `mcp_remote_fetch_failed` are `transport_failure`. - A non-success response body is read as a stream, so a body that fails partway records the bytes that arrived. The reader stops at the size limit and cancels the body, as the `tools/call` reader does. #19785 makes the same reader change for the context path. - `server/src/services/mcp-http.ts`: `readMcpHttpResponse` takes an `onProgress` callback. It reports bytes, chunks and events received, whether the matching response arrived, and whether the stream ended. As on master, the read returns as soon as the response with the matching id arrives. - Tests: - `server/src/__tests__/mcp-http.test.ts`: a partial event followed by a peer error keeps the progress that was consumed, and releases the reader. - `server/src/__tests__/tool-gateway.test.ts`: six scenarios run against a remote server that streams one progress event first: slow success after 12 seconds, the default deadline, an explicit 1,500 ms deadline, a peer reset, a cyclic error cause, and a primitive error value. - `server/src/__tests__/tool-gateway.test.ts`: three regression tests: a tool question sent after the deadline passed keeps `elicitation_required`; a DNS failure records `transport_failure`; and a 503 body that fails partway records the bytes that arrived. All three fail oncanary/v2026.1109.0-canary.14 |
||
|
DottaandClaude Opus 5.5
|
3e8744fa91 |
perf(task-chat): stop a slow run transcript from hiding the chat (#20277)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat shows saved messages, run transcripts and the composer > - The chat hides the messages and the composer until every run transcript loads. This prevents layout movement > - On staging, one task's run-log read took 11–12 s. Saved messages and the composer stayed hidden for 13.9 s after a click, and for 14.6 s again on revisit > - This pull request limits how long a transcript can hold the first reveal after saved history is ready > - The benefit is that a slow run log no longer hides the conversation, and the layout contract stays in place ## Linked Issues or Issue Description **Describe the bug** When a task has a run whose transcript loads slowly, the whole chat stays behind the loading skeleton. The composer stays hidden too. On staging, a failed run with no saved reply needed an 11–12 s object-store log read. The chat appeared after 13.9 s. Saved messages that loaded in about 2 s were not visible during that time. **Steps to reproduce** 1. Open a task whose latest run failed before it saved a reply, and whose run log is large or slow to read. 2. Watch the chat area after the click. **Expected behavior** Saved messages and the composer appear when saved history is ready. Run output must not move the messages that are already visible. **Actual behavior** The chat stays hidden until the run log read finishes, or until the 15 s reveal timeout. Refs #20276 (the measurement harness and baseline plan). ## What Changed - `TaskChatThread` now waits at most `INITIAL_TRANSCRIPT_REVEAL_BUDGET_MS` (600 ms) for run transcripts. The time starts when saved history, the plan and chat metadata are ready. The budget is one deadline. A transcript that loads early does not restart it. - When the budget expires, the slow runs are latched for that task, so the decision does not change later: - **Live runs** stop holding the reveal. Their output continues to stream into the live tail at the bottom of the chat. - **Settled runs in compact history** stop loading. They use the existing fixed **More run output** control, so late output cannot insert rows above the reader. Selecting the control loads the full history, as it does today. - **Settled runs in other modes** (expanded history, conversation mode, a deep link to a comment) still wait, as before. In those modes, output would insert rows around saved replies. - **Live runs with a steering message or other timeline anchors** still wait, as before. Their output before the anchor goes into the main conversation above the anchored message. - A live run that finishes after the reveal keeps its visible transcript. Only runs that were already settled when the budget expired are deferred. ## Verification - `cd ui && npx vitest run src/components/TaskChatThread.test.tsx`. New tests for legacy and native runs: - A slow failed run in compact history: the chat reveals at the budget, the transcript is no longer requested, **More run output** appears, and expanding loads the run. - A slow live run: the chat reveals at the budget, the transcript keeps streaming, and the transcript stays after the run finishes. - A slow settled run without the compact control: the chat still waits. - The budget does not start while saved history is pending. - When one of two transcripts loads before the budget ends, the chat still reveals at the first deadline. - A slow live run with a steering message: the chat still waits. - A compact task with 300 slow failed runs: the chat reveals at the budget and defers every run. - `cd ui && npx vitest run src/components/task-chat/transcript-reveal-budget.test.ts`. The budget selection is one pass. A 5,000-run test counts anchor lookups and throws above one lookup for each live run. - Red check: with the budget timer disabled, the 6 new positive tests fail. With the change, they pass. - `npx vitest run src/components/TaskChatThread.test.tsx src/components/task-chat src/pages/IssueDetail`: 64 files and 1,609 tests pass. - `pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/task-loading-frame.spec.ts`: 7 of 7 pass. This covers a stable loading frame on desktop and mobile, composer geometry, the reading position, a stalled web font, and the Classic and mobile preference paths. - After deploy, staging is measured with the harness from #20276. The finish condition is that a slow run-log task opens in the same time as other tasks. ## Risks - After a slow transcript is deferred, a settled failed run in compact history shows its failure message and **More run output**, not its last transcript lines. The output is one click away. The server-side log read fix is a separate follow-up. - A live run's output can appear in the tail after the reveal. This is the normal streaming behaviour of the tail. - A 600 ms budget is a trade-off. Fast transcripts still arrive in the first frame, and slow ones no longer hide the chat. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ac0d0374f2 |
perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description **Problem or motivation** Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. **Proposed solution** Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. **Alternatives considered** Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. **Roadmap alignment** This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fb88a89f04 |
fix(chat): tell chat agents how to link a case page while Cases is enabled (#20265)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers questions about company records and links each record it cites > - Chat guidance gives link formats for tasks, approvals, runs and agent profiles > - Cases (an experiment) have their own page at `/<prefix>/cases/<identifier>`, but the guidance does not name it > - So an agent that answers a question about a case's fact history names the case but cannot link it > - This pull request adds a Cases directive to chat guidance, only while the Cases experiment is on > - The benefit is that a person can open the case, where its facts and audit history are, from the reply ## Linked Issues or Issue Description **What happened** In a chat eval with Cases enabled, a person asked who changed a case's diagnosis. The agent read the case audit and the task's document revisions and answered correctly. It wrote "case COM-C1" as plain text. It linked only the task. The guidance in the chat prompt has no link format for a case, and the agent does not guess routes. **Expected behavior** When a reply concerns a case (its facts, fact-change history or handoffs), the agent links the case page as a Markdown link, for example `[RUN-C1](/<prefix>/cases/RUN-C1)`, and also links the tasks it cites. When the Cases experiment is off, the prompt does not mention cases. **Steps to reproduce** 1. Enable the Cases experiment. 2. Create a case with a primary task, and change one of its facts. 3. In Agent Chat, ask "Who changed the diagnosis, and what did it say before?" 4. Before this change, the reply names the case identifier but has no link to the case page. ## What Changed - `server/src/services/agent-conversations.ts`: new `CASES_CHAT_DIRECTIVE`. It says what a case is, names the `/<prefix>/cases/<case identifier>` page, and asks for a Markdown link when a reply concerns a case. - `server/src/services/heartbeat/task-markdown.ts`: new `casesEnabled` input. A conversation prompt includes the directive under "Cases:" only when it is true. - `server/src/services/heartbeat/task-context.ts`: sets `casesEnabled` from the `enableCases` experimental setting, the same way as `statusCardsEnabled`. - `server/src/services/heartbeat/task-markdown.test.ts`: new test. The directive appears for a standing chat with Cases on. It does not appear with Cases off or for a non-chat task. The test also checks that the board route `cases/:caseIdentifier` exists in `ui/src/App.tsx`. ## Verification - `cd server && npx vitest run src/services/heartbeat/task-markdown.test.ts` passes (18 tests). - `npx tsc --noEmit -p server` passes. - The `cases.case-history` chat eval uses this guidance. Its rerun is on branch `chat-eval/case-unclear-9`. ## Risks - Low risk. The change adds prompt text only when the Cases experiment is on. It does not change any API or permission. - The prompt for chats on instances with Cases enabled is slightly longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M context, extended thinking, with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1004e55180 |
fix(recovery): stop stall wakes from failing setup on a held workspace (#20273)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, check for proof that its run would be refused. The proof is that the assignee's own latest run on the task failed setup with the unfinished-writer error, and the bound workspace hold still refuses a run. Then do not wake it; count it as deferred and re-check at most hourly. A different assignee (for example a controller-only agent that never uses the workspace), a later run, or a settled hold lets the wake through. Reservation already tried and failed the unchanged-writer release for that run, so the sweep does not repeat that proof. - `blocker-stall-wakeups.ts`: count failed runs that carry the stall (the keyed wake's run and keyless resource-wait retries that keep its context). Retry after 10, then 20 minutes, and stop after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `remote-workspace-writer-hold.ts` / `remote-workspaces.ts`: share the unfinished-writer error text as one constant. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 22/22. Added after review: a waiter with no refused run, or a reassigned waiter, is still woken; failed resource-wait retries count toward the limit. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. The parent's own earlier run failed setup on the hold, as on staging. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2dd7a98753 |
fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, read its bound remote workspace hold. When the hold refuses a run, do not wake it; count it as deferred and re-check at most hourly. A workspace with an unchanged-release baseline is left to reservation, which can release it. - `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried after 10, then 20 minutes, and not after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
610b54f551 |
test(chaos): close review-thread gaps in speed grading, S1 stops, readiness and stall reads (#20244)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades how real tasks survive control-plane restarts, and its grades decide which product fixes come next > - Reviewers left 46 unresolved threads on ten merged harness PRs (#20071, #20094, #20106, #20111, #20119, #20147, #20158, #20161, #20178, #20194) > - Several are real harness defects: false speed breaches from one-sample rows, a codex-only wave that cannot start, event paging that can loop forever, an S1 task failed for a Stop the harness never sent, and a readiness wait that aborts on a transient lifecycle error > - Wave S1 t05 (w-20261109T091247-s9806) hit the S1 defect: the run finished before the stop window, so neither the Stop nor the continue comment was sent, and the task was graded `done_too_late Infinity` > - This pull request fixes the valid findings and adds the missing tests > - The benefit is that a failed task points at the product, not at the harness ## Linked Issues or Issue Description Refs #20071, #20094, #20111, #20147, #20158, #20161, #20178, #20194 (review threads). The security finding from #20111 was fixed separately in #20228. **What happened?** Harness defects found in review and in waves: - Aggregate p50/p95 rows breached at n = 1–3, which caused most marginal speed failures. - `--adapters codex` with the default workloads failed at parse because M1 needs two adapters. - `runEvents` paged forever if a full page had no `seq`. - A timed-out task whose last sample was `done` was graded `done_too_late Infinity`, and could pass when a fault had no ready time. - S1 waited forever for a running run to stop; it also clicked Stop while another task's fault had the server down or recovering. - The readiness wait aborted on any `lifecycleError`, including the transient one lifecycle `defer` sets before a 60 s retry, and hid read errors as "not ready". Daytona used the 240 s local timeout. - One failed parent read could hide an M1 stall for the whole 10 s check window. ## What Changed - Speed: aggregate rows breach only from 5 samples (`insufficient sample (n < 5)` otherwise). Restart recoveries over the control-plane p95 budget now count in the per-run check, so a single slow recovery still fails. - Config: the default workloads skip M1 when fewer than two adapters are given; an explicit `--workloads M1` with one adapter still fails. - `runEvents` throws when a full page has no later `seq`; the run's speed read fails instead of hanging. - `gradeR1` fails a timed-out task as `no_rest_within_budget` before checking its status. - S1: the Stop waits for a ready server with no fault in progress. If the run already finished when the window opens, the harness records a missed Stop, skips the stop-triggered fault, still posts the continue comment, and the oracle checks only the continue. - Readiness: only a terminal lifecycle state (rejected, terminating, cleaning up, terminated) aborts at once. Any other lifecycle error is logged when it changes (`seed_agent_lifecycle_error`) and the wait continues until ready or the timeout, which names each agent's last lifecycle or read error. The agent API exposes neither `lifecycleOperation.retryRequired` nor a retry-claim timestamp, so a deferred retry cannot be told apart from a parked failure earlier. Daytona waits up to 10 min. - M1: a failed parent read retries on the next poll. - Child speed collection moved into `childSpeedTasksInTurn` so its one-at-a-time order is tested. - Tests for each change, plus the gaps reviewers named: the stop trigger's arming, stall-wake parsing, and `seedChaosLab`'s readiness wiring. - README: S1 missed Stop, R1 timeout rule, speed sample minimum. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 230 passed. Each new test failed before its fix, such as the paging test running out of memory in the loop and the stop-trigger test firing a fault after a missed Stop. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Wave evidence for the defects: S1 t05 in w-20261109T091247-s9806 (missed Stop); codex warm agent_boot p50 breach at n = 3 in w-20261109T104527-s9901. ## Risks - Aggregate rows with fewer than 5 samples no longer fail a wave. Per-run p95 checks still catch individual slow runs and recoveries, but a consistently slightly slow p50 in a small wave is now only reported. - A lifecycle error that persists without `retryRequired` now waits for the readiness timeout (4 min local, 10 min Daytona) instead of failing at once. The timeout message names the error. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1b09920a64 |
fix(issues): refuse agent status changes that detach another run's live execution (#20271)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that an agent run executes holds a checkout lock and an execution lock (`checkoutRunId`, `executionRunId`) > - A status change away from `in_progress` clears these locks. It does not stop the run that holds them > - Agents cannot stop another agent's run: run cancel and the task-tree pause are board-only > - But an agent that can edit a task (for example a lead with the manage-active-checkouts grant over its report) can change its status. The run then keeps running and is no longer bound to its task > - In chat evals, a lead "paused" a launch tree. It moved running subtasks to backlog. The runs kept running, detached > - This pull request refuses an agent's status change when a different queued or running run holds the task's lock. The denial is final, links the task, and names the person's controls > - The benefit is that agents cannot interrupt another agent's run through a status edit, and runs are not left orphaned ## Linked Issues or Issue Description Refs #20261 (chat guidance sends a task-tree pause to the person's Pause subtree control). Refs #19394 (resume restarts the work a subtree pause interrupted). **What happened** An agent changed the status of a task while another agent's run executed it. The request succeeded. The server cleared `checkoutRunId` and `executionRunId`. The run continued, but it was no longer bound to the task. In the chat evals `steering.stop-versus-pause` and `task-coordination.pause-tree`, a lead moved running subtasks to backlog to "pause" a launch. Their runs kept running, detached. **Expected behavior** An agent cannot get the effect of interrupting another agent's run. When a different live run holds the task's lock, the agent's status change is refused before any write. The refusal says it is final. It sends the person to the task and names the controls that really stop or hold the work: the run's Stop or Interrupt, or More task actions, then Pause subtree on the parent task. The lock-holding run, board users, and paths that stop the run themselves continue to work. **Steps to reproduce** 1. Create agent A and agent B, with B reporting to A (A gets `tasks:manage_active_checkouts` through the manager chain). 2. Give B an `in_progress` task. Set `checkoutRunId` and `executionRunId` to a `running` heartbeat run of B. 3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with `{ "status": "backlog" }`. 4. Before this change: `200`, the status is `backlog`, both lock fields are `null`, and B's run is still `running`. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it refuses a status change when `executionRunId` or `checkoutRunId` names a different run in the same company that is `queued` or `running`. It runs in `PATCH /issues/:id` after the final status and assignee are known (execution-policy transitions included) and before any run stop or write. It also runs in recovery resolution when the source status changes. - The guard applies to any status other than `in_progress`, also to the current status, because the service clears the locks for those too. - The write is conditional on the lock the guard checked. The route re-reads the lock and passes it to `svc.update` as `expectedExecutionLock`. The service compares it under the row lock and returns `409 issue_execution_lock_changed` if a checkout landed in between. - These cases stay allowed: the lock-holding run itself, board users, requests with no status, a change to `in_progress`, `cancelled` (the route cancels the run), reassignment (the route stops the run first), `done` through PATCH while a Runner goal is active (terminalization pauses the goal and stops the run; recovery resolution does not get this exception), and a lock holder that already finished. - `packages/shared/src/issue-write-denial.ts`: new denial code `issue_status_change_live_run` (409). The copy says the refusal is final, links the task as `[identifier](/prefix/issues/identifier)`, and names Stop or Interrupt and More task actions, then Pause subtree. A new optional `issuePath` context field carries the link. The UI notice renders it through the existing generic path. - `doc/SPEC-implementation.md`: documents the rule in the checkout contract section. - Paths checked with no change needed: there is no bulk issue update route; tree holds and run cancel are board-only; `POST /issues/:id/release` already lets only the assignee's checkout run release; API-catalog actions call the same PATCH route; MCP status changes already refuse with `MCP_ACTIVE_EXECUTION`. ## Verification - Red on master, green with this change: - `server/src/__tests__/issue-stale-execution-lock-routes.test.ts` (embedded Postgres, real issue service): a manager moves its report's running task to backlog. On master: `200`, status `backlog`, `checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409 issue_status_change_live_run`, row unchanged, run still `running`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued lock holder, and a repeated current status (red on master). Allowed cases: field edits, the lock-holding run, board, `cancelled` (run is cancelled), and finished lock holders; an allowed change carries `expectedExecutionLock`. - Same embedded-Postgres file: `svc.update` with an `expectedExecutionLock` that differs from the row returns `409 issue_execution_lock_changed` and leaves the row unchanged. - Also passed locally: `issue-closed-workspace-routes`, `issue-comment-reopen-routes`, `issue-recovery-actions`, `issue-scheduled-retry-routes`, `low-trust-red-team-routes`, `issue-update-comment-wakeup-routes`, `external-object-routes`, `packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`. - `tsc --noEmit` shows no errors in the changed files. Full typecheck, build and test are left to CI. ## Risks - Behavior change for agents: a lead or CEO that used status edits to take over a report's running task now gets a 409. The supported paths (comment, cancel, reassign, or the person's Stop or Pause subtree) still work. - A queued run that holds the lock also blocks the change. This is deliberate: the queued run would otherwise start later with no task bound to it. - Agents can still cancel another agent's run by cancelling its task. That is existing, documented behavior ("task cancellation cancels the intent"), and this PR does not change it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6459488984 |
fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, a remote (Daytona) run can resume by reattaching to
its surviving sandbox runner. If provider recovery fails and the
checkpoint allows it, the run opens a replacement provider session
instead.
> - Before a replacement, the executor archives the local controller
state and the remote session root as a continuity break, then opens a
new session.
> - On a reattach backend, the replacement still adopted the archived
runner and asked for its old authority, so it failed. The archive had
already moved the remote state, so the next retry could not verify the
runner and the issue waited 17 minutes on an unconfirmed stop.
> - Review showed that a reattach backend fixes many launch inputs once
at creation (runner binary, provider pack and companions, artifacts,
live-recovery mode, authority, spawn barrier), so resetting them after
the archive does not scale.
> - This pull request builds the replacement on a new backend for the
same run, created under a `bootstrap_incomplete` claim after the
archive, so every launch input resolves through the ordinary fresh path.
> - The benefit is that a failed reattach after a restart falls back to
a working replacement instead of a stuck issue.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1
(no rest within budget). A K2 drain suspended run a53dd948's sandbox
runner (pid 589) during startup. On restart the run claimed
`reattach_remote_runner`. The adopted runner authenticated and restored,
then its connection closed, and provider recovery failed. The runtime
opened a replacement: `provider.session.archive_before_replacement`
archived the local and remote state, then the replacement failed with
"PRP provider resume state is unavailable". The same-run retry at
10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the
remote `runner-state.json` had moved), and file preservation deferred on
`agent_stop_unconfirmed` until the wave ended.
**Expected behavior**
After the continuity archive, the replacement opens a new runner for
this run, as it does for every other claim kind.
**Steps to reproduce**
1. Run a remote (Daytona) native run and restart the server so the run
claims `reattach_remote_runner`.
2. Make provider recovery fail on the adopted runner with a checkpoint
that allows replacement.
3. See the replacement open on the adopted runner's authority and fail
after the archive.
**Paperclip version or commit**
master at
|