mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
59ded526be0368bf10c21ddec2d25e131d028cc6
3864
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude
|
59ded526be |
fix(native): ask once to re-check Paperclip before resting blocked on its own stall (#20779)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs report their result through a completion report. The server checks the report before it accepts it. > - When the Paperclip server pauses or restarts, the agent's Paperclip tool calls stay unanswered. They return later, all together. > - The agent can read that silence as an outage. It then reports a blocker owned by a "Paperclip ... service" with reasonCode `external_service_unavailable`. > - The server accepts that report. The task rests blocked with no Retry. Paperclip's own stall stops the task. Our rule says that must not happen. > - This pull request rejects that blocker report one time per run. The correction tells the agent that Paperclip is reachable now and to re-check. > - The benefit is that the agent continues in the same turn and reports the real outcome. ## Linked Issues or Issue Description **What happened?** In a restart-chaos test, the server was paused (SIGSTOP) for about 2 to 5 minutes while a Codex agent saved a task document. `write_document` and `read_document` did not return during the pause. After the pause, the save had succeeded, but the agent reported a blocker: owner "Paperclip document service", reasonCode `external_service_unavailable`. The run succeeded with disposition `blocked`. The task rested blocked with `canRetry=false` and no automatic recovery. A second test (two long server holds, then a same-run resume) gave the same result. **Expected behavior** A Paperclip stall or restart must not leave a task stuck. The agent must re-check Paperclip after it is reachable and finish the task. **Steps to reproduce** 1. Start a native Codex task that saves a task document. 2. Pause the server process with SIGSTOP for about 2 minutes during the save, then resume it. 3. The agent reports a blocker owned by a Paperclip service. The task rests blocked with no Retry. **Paperclip version or commit** master at |
||
|
DottaandClaude
|
eb75c9bb85 |
fix(native): resend a throttled harness backup upload instead of losing it (#20772)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a native run in a per-turn remote sandbox, the server saves a durable harness backup to object storage. The next run restores the provider session from it. > - On staging, two OpenCode runs had an accepted result, but `harness_state.backup.persist` failed. The run log for both says `remote checkpoint failed: Please reduce your request rate.` This is the S3 `SlowDown` (HTTP 503) answer. > - The backup and the OpenCode provider-artifact bundle were uploaded as one stream `PUT`. The S3 SDK cannot send a stream body again, and this path had no retry for a request-rate limit. One `SlowDown` lost the backup. > - The sandbox was then released. The follow-up run had no state to resume and failed with `PRP provider resume state is unavailable` (class P, see #20757). > - This pull request uploads both objects from their encrypted scratch file, as workspace snapshots already do. The existing throttled-retry path then sends them again. > - The benefit is that a short request-rate limit no longer loses the only copy of a provider session. ## Linked Issues or Issue Description Refs #20757 (the failure this causes on the next run). Refs #19917 (added the file-source upload path that this change uses). **What happened?** 1. An OpenCode run in a per-turn Daytona sandbox finished, and its result was accepted. 2. The server uploaded the encrypted harness backup with `storage.putFile` and a stream body. 3. Object storage answered `503 SlowDown` ("Please reduce your request rate."). Nothing sent the request again. 4. `harness_state.backup.persist` failed, and `native.post_result_failure` was recorded. 5. The next run for the same chat task started in a new sandbox and could not resume the provider session. **Expected behavior** A request-rate limit slows the upload down. It does not lose the backup. **Steps to reproduce** 1. Point the storage service at an S3 endpoint that answers the first `PUT` of each object with `503 SlowDown`. 2. Call `persistNativeHarnessBackup`. 3. On `master` it throws `SlowDown: Please reduce your request rate.` **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `server/src/services/native-runtime/native-harness-backup-storage.ts`: pass the encrypted scratch file as `sourceFile` to `putFile`. - `server/src/services/native-runtime/native-provider-artifacts.ts`: the same for the provider-artifact bundle. - `native-harness-backup-throttle.integration.test.ts` (new, focused file): a real S3 client against a local endpoint that throttles the first `PUT` of each object once. Cases for a Codex backup, and for an OpenCode backup with provider artifacts. Each case also throttles the first read-back of each object once. Each case checks that the backup is current and restores the provider session bytes. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-harness-backup-throttle.integration.test.ts`: both cases fail on `master` with `SlowDown: Please reduce your request rate.` and pass with this change. - `cd server && npx vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 68 pass (file unchanged). - `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify`: it selected the full server suite and failed 16 of 34,997 tests, all local-environment failures unrelated to object storage: missing `.claude/skills/*/SKILL.md` files in this worktree, `ENAMETOOLONG` under the macOS temp directory, and 30 s hook timeouts under load. Required CI is the authority for that suite. ## Risks - Low. A file source also sends a `Content-MD5` header, so the backend rejects any byte it did not receive intact. Workspace snapshots have used this path since #19917. - An upload above 16 MiB now uses the file multipart path. That path has the same verified-part retry. - The read-back check (`getObject`) needs no change: a GET has no body, so the S3 client resends it after `SlowDown`. The test now also throttles the first read of each object once and proves the backup still publishes. - The Rust Runner is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nicky Leach
|
4d141cdf64 | Carry plugin task refusals and batch project reads (#20762) | ||
|
DottaandClaude Opus 5.5
|
f125b823f4 |
fix(recovery): release a parent's Retry once a done child's recovered files are saved (#20728)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task waits for its done child tasks to save their workspace files before it runs again (the workspace-finalize barrier) > - "Recover saved files" saves the retained work of a done child and records a successful `workspace_finalize` > - That finalize row had no `execution_workspace_id`. The barrier reads finalize rows by the blocker's workspace, so it never saw the row > - The parent's saved Retry then waited forever with "Saving its files" on a child that had already saved its files > - This pull request lets the barrier accept the recovery's own finalize for the recovered task's finished operations of the same run. It does not make the recovery finalize a workspace-wide save > - The benefit is that recoveries work by themselves: the parent's dependency wait releases and its saved Retry starts on the next dependency pass, with no migration ## Linked Issues or Issue Description **What happened?** A parent task had three done children. One legacy child shared the parent's execution workspace. The operator pressed "Recover saved files" on that child. The recovery saved the files and marked the old run succeeded. The operator then pressed Retry on the parent. The server accepted the Retry, but the parent stayed on "Retry is waiting for a finished task this one depends on to save its files" for more than 15 minutes. `blockerAttention.pendingFinalizeBlockerIssueIds` named the recovered child. The child had no active recovery action and no execution blocker. The workspace was clean (`dirty: false`, `writerRunId: null`). The child's workspace operations showed the cause. The latest operation on the workspace was a succeeded `workspace_recovery`. The recovery's succeeded `workspace_finalize` existed, but with `execution_workspace_id = NULL`. Earlier finalize attempts of that run had failed. `pendingFinalizeBlockerCondition` only looks at operations on the blocker's workspace, so it found no successful finalize after the recovery. **Expected behavior** When "Recover saved files" saves a done child's work, the child must stop holding its dependents. The parent's saved Retry must start without another operator action. **Steps to reproduce** 1. Let a child task finish as `done` with a retained remote workspace whose finalize failed. 2. Run "Recover saved files" on the child, so recovery saves the remote workspace snapshot. 3. Block a parent task on the child. The parent can share the child's workspace. 4. Read the parent's dependency readiness. Before this fix it reports the child in `pendingFinalizeBlockerIssueIds` forever. **Paperclip version or commit** Seen on staging at `a46b9ac7`. Reproduced on `master` at `e039be9202`. ## What Changed - `server/src/services/issues.ts` (`pendingFinalizeBlockerCondition`): the blocker task's latest workspace operation is settled when a succeeded recovery finalize (`owningService: workspace_recovery`) from the same company, task, and run started at or after that operation finished. The operation must be finished (`succeeded`, `failed`, or `skipped`). The recovery finalize can have no workspace (all rows written so far) or this workspace. - The rule is scoped to the blocker task and its run. It does not change the workspace-wide `finalized_at`, so other tasks that share the workspace keep their own barrier. - `server/src/services/workspace-recovery-dependency-finalize.test.ts`: new regression tests that run the real remote "Recover saved files" path, make the child `done`, and block a parent that shares the child's workspace. - `doc/execution-semantics.md`: documents how the recovery finalize settles the dependency barrier. ## Verification - Red (unfixed `issues.ts`): `npx vitest run src/services/workspace-recovery-dependency-finalize.test.ts` fails 3 of 4 tests. The parent stays not ready with the recovered child in `pendingFinalizeBlockerIssueIds`. - Green (fixed): the same file passes 4 of 4 tests: - a real recovery releases the parent in `getDependencyReadiness` and in the issue list `blockerAttention`; - snapshot-fallback shape: the preservation receipt has no workspace and the child's latest operation is the run's failed finalize. The parent is released; - safety: a sibling task that started work on the shared workspace while the recovery waited for teardown still holds the parent; - safety: child work that starts after the recovery still holds the parent. - Focused suites pass: `src/__tests__/issues-service.test.ts`, `src/services/workspace-recovery-admission.test.ts`, `src/services/task-retry.test.ts`, `src/services/saved-bootstrap-continuation.test.ts`, `src/services/remote-workspace-recovery-receipt-fallback.test.ts`, `src/__tests__/issue-dependency-wakeups-routes.test.ts` (414 tests with the new file). - `pnpm verify` exits 2 (a doc change is a global CI input), so required CI supplies broad coverage. Server `tsc --noEmit` passes. ## Risks - Previous-release story: no persisted shape changes. Recovery finalize rows already saved (with no workspace) are read by the new rule. Affected parents become dependency-ready on the next read, with no migration or backfill. The deferred-receipt and dependency backstop passes then start the saved Retry. - The barrier keeps its protection. Unfinished operations, operations from another run, operations that start after the recovery finalize, and other tasks on a shared workspace still wait for their own save. - No schema change and no write-path change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution, read-only staging API inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
1ac8b6a711 |
fix(workspaces): stop re-saving a retained sandbox whose disk is still full (#20758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a native run on a remote sandbox, Paperclip saves the sandbox workspace back to durable storage. When that save fails, saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - A staging census of the last 48 hours (`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`, class F) found saves that failed because the sandbox disk was full. > - #20678 already ends finalization at the first full-disk copy-back and hands the work to saved-work recovery. Recovery then deletes inactive tool caches that it can prove safe, and saves again. > - When that save still found the disk full, recovery recorded a transient `export_interrupted` failure with cause `provider_error`. It then saved again on its automatic schedule (up to 5 retries in 30 minutes). Nothing frees space in a stopped, retained sandbox between those attempts, so each attempt failed the same way. The task showed "Recovering saved work…" and no Retry while this went on. > - This pull request ends the recovery generation at the first such failure with a named cause, `sandbox_disk_full`, and hands it to the board with a Retry. > - The benefit is that the task does not spend 30 minutes on saves that cannot succeed, the user sees why, and one Retry saves the files and starts one run. ## Linked Issues or Issue Description Refs #20678, #20695 (named sandbox disk-full failures for copy-back and restore) **What happened?** A native run on a remote sandbox finished, and its workspace copy-back hit a full sandbox disk. Finalization handed the task to saved-work recovery (#20678). Recovery's own save also hit the full disk. Recovery classified this as `export_interrupted` / `retry_transient` with cause `provider_error`, so it scheduled more automatic saves of the same sandbox. Each one failed the same way. The task showed "Recovering saved work…" with no Retry until the 30-minute window ran out. **Expected behavior** A save that still finds the disk full after the safe cache cleanup is not repeated automatically. The recovery ends at once with the cause `sandbox_disk_full`, and the board gets a Retry. The workspace keeps its previous saved version, and the run's unsaved files stay in the retained sandbox. One Retry saves the files once there is room and starts exactly one run. **Steps to reproduce** 1. Give a native run an accepted result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make every save from its retained sandbox fail with `remote_workspace_sandbox_disk_full`, with no inactive tool cache to delete. 3. Run saved-work recovery. Before this change, it records `export_interrupted` with cause `provider_error` and schedules another save. ## What Changed - `workspace-recovery.ts`: a save that fails with `remote_workspace_sandbox_disk_full` after a confirmed cache cleanup (or with nothing to clean up) now fails the generation with `export_interrupted`, retry class `user_action_required`, and cause `{ label: "sandbox_disk_full", code: "ENOSPC" }`. That class is not retried automatically, and it is not a permanent-source class, so recovery never falls back to the last save and never drops the run's unsaved files. The next sweep hands the hold to the board. - A cleanup whose outcome is unknown (lost acknowledgement, interrupted journal) keeps the existing scheduled retry, because its journal resumes on the next attempt and can still free space. - `retained-provider-cache.ts`: `reclaimInactiveNativeProviderCache` returns `{ changed, confirmed }` instead of a boolean, so the caller can tell "nothing to delete" from "outcome unknown". It has one caller. - `system-recovery-state.ts`: the board handoff text for this cause says that the sandbox disk is full, that Paperclip deletes only inactive tool caches it can prove safe, and that Retry saves the files again. - New `workspace-recovery-disk-full.test.ts` covers the whole path: one failed attempt, no further automatic saves, the board handoff text, the previous saved version unchanged, the agent's files still in the sandbox, no unfinished in-sandbox capture attempt, a Retry offered in the task presentation and blocker, `POST /issues/:id/retry` accepted (202), the files saved after Retry, and exactly one run started (also with a replayed dispatch). - `doc/execution-semantics.md`: documents the behavior. ## Verification - Red on master: `cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts` fails. The recovery state is `scheduled` with cause `provider_error` instead of `exhausted` with cause `sandbox_disk_full`. - Green on this branch: the same command passes. - Related suites pass: `workspace-recovery-retry-copyback.test.ts` (79 tests, including the 8 cache-cleanup resume tests that need the scheduled retry after an unconfirmed cleanup), `system-recovery-state.test.ts`, `workspace-recovery-capture-policy.test.ts`, `workspace-recovery-fallback.test.ts`, `workspace-recovery-stop-proof.test.ts`, `native-run-finalizer-telemetry.test.ts` (167 tests in all). - `npx tsc --noEmit -p server/tsconfig.json` is clean. - `pnpm verify`: exit 2 (broader verification required), because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change touches only the saved-work recovery path when its save fails with the exact full-disk message. - Upgrade-safe: the failure code (`export_interrupted`), the retry class (`user_action_required`) and the cause label all pass the earlier release's recovery schema. An earlier release that reads this state shows its generic "Retry to try again" text. - A full-disk failure no longer gets automatic retries in its 30-minute window. If space became free by itself in that window, the user must now press Retry. Nothing in a stopped, retained sandbox frees space by itself, so this is expected to be rare. - No Runner (Rust) change, so no new sandbox image is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
ea25af759e |
fix(native): name a resume that finds no saved Runner state, so Retry starts a fresh session (#20757)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can continue the provider session of the previous run on the same task. In a remote sandbox, the Runner keeps the session state inside the sandbox, and the server saves a harness backup after each run. > - On staging, five runs in 48 hours failed with `native_session_recovery_failed: Error: PRP provider resume state is unavailable` (class P in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`). > - In each case we read, the previous run had an accepted result, but its `harness_state.backup.persist` failed. Its per-turn sandbox was then destroyed. The next run started in a new sandbox with no Runner state for the session. > - The Runner transport threw an untyped error. The runtime could not tell it apart from a transport drop. A governed-wait or same-session checkpoint ended as a generic `native_session_interrupted` with an "ambiguous state" hold that said not to open a replacement session. > - This pull request types that refusal and lets the driver classify it as `provider_state_incompatible` when nothing is in flight. The existing recovery policy then decides. > - The benefit is that the run ends with the permanent named code `native_provider_state_incompatible`. Retry, or the next message, continues in a fresh provider session in the same workspace. No provider turn or tool call can be repeated, because the follow-up never started a turn. ## Linked Issues or Issue Description Refs #20712, #20723 (the same classification for a moved remote workspace root). Refs #20691 (fixes the `provider_identity_incomplete` part of class P). **What happened?** 1. Run A on a chat task finished. Its result was accepted. Its harness backup failed to save (`harness_state.backup.persist` failed, `native.post_result_failure`). 2. Run B started for the next chat message in a new sandbox. It rebound run A's checkpoint with `providerRecoveryPolicy: allow_replacement_after_governed_wait`. 3. The Runner found no control-plane state and no archived authority epoch. It threw `PRP provider resume state is unavailable` as a plain `Error`. 4. The runtime refused a replacement for the governed-wait policy and failed run B with the generic `native_session_recovery_failed`. The task stayed blocked with an "ambiguous state" hold. **Expected behavior** The run ends with a permanent named code and a working Retry, as it does for other deterministic refusals. **Steps to reproduce** 1. Create a Runner transport that resumes a saved provider session from an empty state directory. 2. Request `thread/read`. On `master` it rejects with a plain `Error`. 3. Recover a saved Codex, OpenCode or ACPX session through the driver when the transport fails that way. On `master` the result has no `failureClass`. **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `packages/paperclip-runner/src/contracts/harness-driver.ts`: add `NativeRunnerResumeStateUnavailableError` (same message as before) and `isRunnerResumeStateUnavailable`. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: throw the typed error at the three branches where the control-plane state is absent. The branch with no PRP identity and the quarantine branch are unchanged. - `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`: classify the typed error as `provider_state_incompatible` when the saved session has no turn or runtime request in flight. Extract the in-flight test that the moved-root check already used, without changing it. - `doc/execution-semantics.md`: describe the new classification (additive). - Tests: a transport test (local and external Runner state), driver tests for Codex, OpenCode and ACPX Claude, in-flight cases that stay unclassified, and a server case that shows the blocked task offers Retry and that one click starts exactly one fresh-session run. ## Verification - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts -t "no saved state"`: 5 pass. With the driver change reverted, the 3 "reports … as provider state incompatible" cases fail (no `failureClass`). - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-cold-restore.test.ts`: 5 pass. On `master` the error is a plain `Error`. - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts src/live/runnerd-codex-transport-cold-restore.test.ts src/native-session-runtime.test.ts`: 410 pass. - `cd server && npx vitest run src/services/workspace-recovery-retry-copyback.test.ts -t "does not continue automatically"`: 5 pass, including the new case. It checks `canRetry: true`, an accepted Retry, exactly one queued successor, and `forceFreshSession: true`. - `cd packages/paperclip-runner && npx tsc --noEmit -p .`: clean. - `pnpm verify`: exit 2 (broader verification required) because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change only adds a class to a failure that already failed. A checkpoint with `allow_replacement_after_resume_failure` already replaced the session for this failure, and still does. - For `same_session_only` and `allow_replacement_after_governed_wait`, the run now ends with `native_provider_state_incompatible` instead of a retryable `native_session_interrupted`. Same-run retries stop at once. They could not succeed, because the state is absent. - A session with a turn or runtime request in flight keeps the old unclassified behavior. This PR does not change it. - The server code path is unchanged. The Rust Runner is unchanged, so no new runner image is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
9574065a28 |
fix(workspaces): hand on a proven-unchanged preparation epoch instead of abandoning it (#20736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox first restores its workspace's saved version, publishes that root as its dirty epoch, and only then starts work > - A restart that kills the run in that window leaves the epoch dirty. The ownership settle (`settlePreparationOnlyRemoteWorkspaceEpoch`) always abandons it: the root is forgotten and the retry restores the saved version into a new directory > - The saved provider session still names the old directory, which is what #20712 and #20723 now have to handle > - In the common case the killed preparation only restored the saved version and changed nothing. A durable proof for exactly that already exists (`proveRemoteWorkspaceUnchanged`, used when the next admission finds an ended writer) > - This pull request makes the settle use that proof first, and keeps abandoning whenever the proof fails > - The benefit is a stable workspace root across a restart during preparation, with no change for anything unproven ## Linked Issues or Issue Description Refs #20712 **What happened?** In restart-chaos waves, a server kill during a follow-up's preparation (right after the workspace restore published) made the ownership settle abandon the epoch. The retry restored into `<workspace>-<epoch>-<lease>` instead of the published root, and the inherited session checkpoint named a root that no longer existed. **Expected behavior** When durable records prove the killed preparation only restored the saved version, the workspace is handed on at that saved version and keeps its root. Anything unproven is abandoned exactly as before. **Steps to reproduce** 1. Run a remote native task to a saved version, then start a follow-up or continuation run. 2. Kill the server after the follow-up's restore publishes (crash point `remote_workspace.prepare.after_publish_before_provision`). 3. Let ownership reconciliation settle the dirty epoch, then let the retry prepare. ## What Changed - `recovery/remote-workspace-unsaved-files-settlement.ts`: before it abandons a preparation-only epoch, the settle calls the existing `releaseUnchangedRemoteWorkspaceWriterById` (lock the workspace, check the writer, then `releaseUnchangedRemoteWorkspaceWriter`). That releases the writer at its saved version (dirty false, writer cleared, location and root kept, baseline marked released, run reference set to `saved`) only when `proveRemoteWorkspaceUnchanged` holds. The proof requires all of these durable records: - the unchanged-baseline marker for this exact epoch, run and saved version. It is written in the same transaction that publishes the restored root, so a kill mid-restore leaves no marker; - no recorded process, process group, process start or provider session; - no execution-start run event, no native result and no native runner checkpoint (the controller writes one before every runner command); - no workspace operation other than the file-neutral restore, finalize and recovery receipts (provisioning and saves record theirs before they touch files); - every workspace repository already in the saved version. - If the proof fails for any reason, the settle abandons the epoch exactly as before. The existing settle preconditions (run terminal, still `preparing`, no finalization, no work-started event, no save in flight) are unchanged and run first. - `packages/shared/src/crash-point-registry.ts` and `remote-workspaces.ts`: add the crash point `remote_workspace.prepare.after_publish_before_provision`, so the chaos harness can kill exactly at this boundary. It counts only a preparation that published an unchanged baseline (a restore of a saved version), and it is a no-op unless a test arms it. `tests/runner-e2e/source-sdk-public-inputs.json` is refreshed because the registry file is a pinned SDK input. - Test `ownership-reconciliation-report.test.ts`: a proven-clean epoch is handed on with its root kept, with no abandonment receipt, and the writer hold is free for the retry. These cases are still abandoned exactly as before: no baseline (an epoch saved by the previous release), a baseline for another epoch, an already-released baseline, a provisioning operation. A run with a recorded process is still neither handed on nor abandoned by this settle. Previous-release story: an epoch saved by a release that did not write the baseline marker has no proof, so it is abandoned as before. No persisted shape changes; the marker already exists on master. ## Verification - Red: with the release branch disabled, the new test fails on the clean case (the epoch is abandoned). Green with the change. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/remote-workspace-snapshot-reuse.test.ts src/__tests__/crash-point-call-sites.test.ts` passes. `packages/shared` `crash-points.test.ts` passes. `tsc --noEmit -p server` passes. - Live Daytona waves (claude, workload S1, the new crash point at the continuation's preparation (`:2` before the point was gated to saved restores; `:1` now), image built from master `e039be9202`, one task each): | | before (release disabled) | after | | --- | --- | --- | | Killed epoch 2 | abandoned (`workspace.remote_epoch_abandoned`) | handed on (`workspace.snapshot_skipped_unchanged`, epoch 2) | | Retry root | new root `…-4-ecf1c81e…` | same published root `…-1-33e93b90…` | | Retry `task.prepare` | 4598 ms | 4731 ms | | Retry restore / environment acquire | 1141 / 1381 ms | 1106 / 1478 ms | | Task result | done, no Retry | done, no Retry | | Stopped or orphan sandbox at rest | none | none | Preparation time does not change in this setup. The Chaos Lab environment uses ephemeral leases, so every run gets a fresh sandbox and a full restore with or without this change, and the killed run's sandbox is destroyed either way. The gain is the stable root, which keeps the saved session's working directory valid. ## Risks - Low. The release path is the same one admission already uses for an ended writer, under the same proof. The settle now tries it first. - A proof false positive would keep a root whose files changed. The proof refuses on every record that precedes a write, and on any missing or older-release evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
f25fee3532 |
fix(native): hash runner artifacts and run archive tar off the event loop (#20722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process prepares native runs, renews every controller lease on timers, and serves every request > - On a run path, the server hashed the 31 MB runner binary with `readFileSync` (more than once per run), hashed provider packs and harness backups synchronously, and ran `tar` with `execFileSync` (timeouts up to 60 s, one with no timeout) > - On a slow disk, each of these calls freezes the whole process. A 103 s stall on a hosted instance lapsed every lease. The last event before that stall was immediately before a synchronous read and hash of the runner binary (see #20694 and #20709) > - This pull request moves those reads, hashes and archive commands off the event loop. It computes the runner binary digest once per boot > - The benefit is that a slow disk delays only the run that waits for it. Turns also start faster, because the runner binary is no longer re-read for every run ## Linked Issues or Issue Description Refs #20694, #20709 **What happened?** On a hosted instance, the server process stopped for 103 s without a restart. Every controller lease lapsed, and two tasks needed a manual Retry. The last run event before the gap was immediately before `matchesControllerRunnerArtifact`, which does `createHash("sha256").update(readFileSync(controllerRunnerBinary))`. Four workspace restores were running at the same time, and file I/O was very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is not proven. But each synchronous call below blocks the event loop for as long as the disk is slow. **Expected behavior** File hashing and archive commands on a run path must not block the event loop. A slow disk must delay only the run that waits for it. **Steps to reproduce** 1. Make a remote runner binary slow to read (the test uses a FIFO whose writer waits), or make `tar` slow (the test puts a `tar` wrapper that sleeps first on `PATH`). 2. Call `resolveRemotePaperclipRunnerBinary`, `stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a 10 ms timer runs. 3. Before this change, the timer stops for the whole slow step (814 ms, 1,764 ms and 3,379 ms). After this change, it keeps firing. ## What Changed - `server/src/services/native-runtime/off-loop-file-work.ts` (new): - `sha256FileStreamed` - `cachedFileSha256`: a streamed digest, reused while the file's device, inode, size, mtime and ctime do not change - `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file only when the header cannot answer, so the result equals `runnerBinaryTarget(readFileSync(path))` - `sha256DirectoryTreeStreamed` - `execFileBuffered`: `execFile` with stdin input, `maxBuffer`, `timeout` and `env`. It rejects on a non-zero exit, a timeout or a `maxBuffer` overflow, like `execFileSync`. - `native-session-executor.ts`: - `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot reads the binary once, not once or twice for every run. - The remote Claude daemon target is read from the binary header. - The bundled remote runner binary is checked by `verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a cached digest. - Provider-pack upload verification uses the new `verifyRemoteProviderPackManifest`. It runs the same checks, in the same order, with the same errors. One generator drives both the sync and the streamed readers. - Harness backup build and verify on run paths use streamed digests (`buildNativeHarnessBackupManifestStreamed`, `verifyNativeHarnessBackupStreamed`). The digest format does not change. - `stageRemoteRunnerDirectory` (3 calls), `assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract use `execFileBuffered`, with the same arguments, timeouts and buffer limits. `stageRemoteRunnerFile` uses `readFile`. - `packages/paperclip-runner/src/live/runner-binary.ts`: the default daemon lookup (`resolvePackagedRunnerBinary`, used by the local and remote resolvers) classifies the daemon with `runnerBinaryFileTarget`. It reads the 64 KiB header, not the whole file. - `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`: the release-manifest checks now live in one helper, `bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and the server's streamed check both use it. - `native-codex-runner.ts`: - `resolveRemotePaperclipRunnerBinary` reads only the header. - The local `runnerDigest` uses `cachedFileSha256`. - New `verifiedBundledRemoteRunnerBinary`. **Ordering and atomicity.** I checked each converted call for code that relied on the sync call being atomic with respect to other runs: - **Checkpoint copy-back.** It extracts into a private `mkdtemp` directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`) stays synchronous, so no other run can see a half-installed checkpoint. This function already awaited a remote command before the extract, so this change adds no new interleaving with the target. - **Harness backup verify then restore.** The verify was sync, but the restore after it was already async. The window between them existed before. One owner serializes the turns of a session, so no other turn publishes a backup for that session while a turn verifies it. - **Staging and provider-pack hashing.** These read build-owned, immutable trees, or the private staging copies of a run. ## Verification - Red/green: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/off-loop-file-work.test.ts` - With `native-session-executor.ts` and `native-codex-runner.ts` from `origin/master`: 3 failed. The longest timer gaps were 814 ms (runner binary target read), 1,764 ms (directory staging tar) and 3,379 ms (checkpoint list, size check and extract). - With this change: 6 passed. The gaps stay under 400 ms. The same file also checks the digest cache, the header fallback, and the `execFileSync`-like errors (exit code, timeout kill, `ERR_CHILD_PROCESS_STDIO_MAXBUFFER`). - `native-session-executor.test.ts`: - The streamed provider-pack verifier returns the same manifest as the sync reader, and rejects a changed asset tree with the same error. - The streamed harness backup manifest equals the sync one. - The streamed verify agrees with the sync verify, for a valid backup and for a corrupt one. - Run together: `native-session-executor`, `off-loop-file-work`, `native-codex-runner`, `setup-readiness`, `antigravity-connection-probe` (1,653 passed). `run-completion.test.ts` passed all 153 tests. Its `afterAll` embedded-Postgres cleanup timed out locally, which is not related to this change; CI runs it. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - **Turn start.** This change adds no wait before `turn.start`; it removes work. The first run of a boot streams the runner binary once. Later runs reuse the digest after one `stat`. - **Cache staleness.** The cache keys on device, inode, size, mtime and ctime. An in-place rewrite changes mtime and ctime, and a replaced file changes its inode, so a changed binary is hashed again. - **stderr.** `execFileSync` wrote a child's stderr to the server's stderr. `execFileBuffered` captures it. Tar warnings no longer appear in server logs; errors and exit codes are unchanged. - **Persisted state.** No persisted shape changes. Old saved state and new code (and the reverse) read the same manifests and digests. - **Not converted in this PR:** small metadata syscalls (`existsSync`, `lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the native state machines. Some of these rely on running synchronously. Also not converted: the sync harness backup verify inside the sync restart-authority checks, and the sync stamp verify in `native-harness-backup-storage.ts`. Both are listed for a follow-up. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments carry the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
94c7c83ce8 |
fix(runner): refuse a moved-root same-session checkpoint by name (#20723)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox resume a task's provider session from a saved checkpoint > - #20712 made a checkpoint that names an abandoned remote workspace root decline recovery, so a replacement session opens in the new root > - A checkpoint that requires the same session (an unfinished goal, `same_session_only`) cannot be replaced. It then failed with the generic `native_session_recovery_failed`, which recovery treats as a retryable interruption > - A retry cannot move a saved session to another root, so the failure repeats. The upgrade-safe rule requires a permanent, named code for this case > - Also, the working-directory errors said "Codex" on Claude and other ACPX runs, because one app-server driver serves all of them > - This pull request reports the moved root as provider state that is incompatible, and makes the errors name the agent generically > - The benefit is a named, permanent refusal with a working board Retry, and error text that is correct for every agent ## Linked Issues or Issue Description Refs #20712 **What happened?** After #20712, a `same_session_only` checkpoint with a moved remote root ended with `native_session_recovery_failed: persisted session working directory is not the assigned remote workspace`. That is not a named permanent code, so recovery could retry a run that cannot succeed. On ACPX Claude runs, path errors read "Remote Codex working directory ...". **Expected behavior** The run ends with the permanent code `native_provider_state_incompatible`. Automatic retries stop. A board Retry continues the task in a fresh provider session in the same workspace. Errors do not name Codex on other agents. **Steps to reproduce** 1. Save a native checkpoint with `providerRecoveryPolicy: same_session_only` under one remote workspace root. 2. Assign the run a different sibling root (the epoch was abandoned). 3. Resume the run. ## What Changed - `codex-app-server-driver-impl.ts`: the moved-root decline from #20712 now carries `failureClass: provider_state_incompatible`. The runtime already maps that class: - checkpoint allows replacement (`allow_replacement_after_resume_failure`): a replacement session opens in the assigned root, as before; - checkpoint requires the same session: `NativeProviderStateIncompatibleError`, code `native_provider_state_incompatible`. - The server already treats `native_provider_state_incompatible` as a permanent failure (no automatic retry, board action text "Retry the task to continue in a fresh provider session with the same workspace"). `rebindNativeSessionCheckpoint` returns no checkpoint after that code, so a board Retry starts a fresh session in the same workspace. No server change is needed. - A moved-root checkpoint with an active turn or a pending runtime request now reports `provider_side_effect_pending`. The runtime never replaces it under any policy, and the run ends with the permanent code `native_provider_side_effect_pending` for an operator to reconcile. Before, it threw the working-directory refusal, which recovery recorded as the retryable `native_session_interrupted`. - `harness-driver.ts`: the `PROVIDER_STATE_INCOMPATIBLE` doc now names the moved-workspace case. - `codex-boundaries.ts`: working-directory errors say "Agent working directory", "Remote agent working directory" and "Provider thread response" instead of "Codex". Tests updated to the new text. - Tests: - `native-session-runtime.test.ts`: the real app-server backend with `remote_runner` authority and a moved root, for each recovery policy. Same-session policies end with `native_provider_state_incompatible`, the replacement policy opens a replacement in the assigned root, an in-flight turn ends with `native_provider_side_effect_pending` under every policy, and no transport starts. - `native-backend-factory.test.ts`: the decline carries the failure class, and an ACPX Claude refusal reads "Remote agent working directory does not match the assigned workspace". Previous-release story: no persisted shape changes. Saved checkpoints are classified at recovery time. A saved same-session checkpoint under an old root now ends with the named permanent code instead of a generic one. ## Verification - Red: without the failure classes, the same-session and in-flight runtime cases fail (generic `native_session_recovery_failed`). Before the rename, the ACPX Claude message test fails. - Green: `cd packages/paperclip-runner && npx vitest run src/backends src/drivers/codex src/live/runner-prepared-launch-workspace.test.ts src/native-session-runtime.test.ts` passes (784 tests). - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Existing server coverage for the code: `native-session-resume.test.ts` (no rebind after `native_provider_state_incompatible`) and `native-session-executor.test.ts` (source-code mapping). ## Risks - Low. Error message text changes. A search found no code that matches on the old text outside the updated tests. - Same-session checkpoints with a moved root now stop for a board Retry instead of a generic interruption. That is the intended behavior: a retry cannot succeed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1143abaf92 |
fix(recovery): give due exhausted holds their own fair lane in the recovery worker (#20725)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. It also re-arms exhausted holds under the rules of #20295, #20569, #20641 and #20648 > - After #20680 and #20661 were deployed to staging, the worker still re-armed none of 29 exhausted holds in 36 minutes. One hold's id sorts in the first percent of all candidates, so the restart starvation that #20680 fixed was not the only cause > - The worker skipped every obligation while any unfinished save was selectable. Discovery is instance-wide. A save that another server owns stays unfinished on every run, so obligations never ran. The obligation pass also put due holds behind hundreds of settled holds whose saved input waits, at two per run. A completed pass never revisited a hold that became due by time alone > - This pull request gives due exhausted holds their own lane, and it shares the worker's slots fairly between unfinished saves, due holds and the obligation pass > - The benefit is that every due hold is reached within a few runs of the worker, whatever its other load. The holds save their files and free their tasks without a person ## Linked Issues or Issue Description Refs #20680 Refs #20661 Refs #20648 Refs #20641 Refs #20569 Refs #20295 **What happened?** On staging, 36 minutes after a deploy that included #20680 and #20661, the workspace recovery worker had touched none of the 29 exhausted holds of the company (22 capture-policy holds and 7 stop-proof or fallback holds). They had no `stopProofRecovery` or `capturePolicyRecovery` marker, no re-arm activity, and `updatedAt` from Nov 2 to Nov 10. The worker did run manual "Recover saved files" requests: one stop-proof hold saved its files 57 seconds after the request. So the worker was alive and the stop proof was available. The holds were not reached. **What did you expect to happen?** The worker reaches each exhausted hold whose re-arm or fallback is due, re-arms it, and saves the files, within a bounded time. **Steps to reproduce** 1. Create a scheduled save whose run's finalization lock another database session holds (another server owns the run). 2. Create an exhausted stop-proof hold and an exhausted capture-policy hold, in the evidence shape that the Nov 4 and Nov 10 releases saved. 3. Start the real workspace recovery worker. 4. The worker retries the busy save on every run and never re-arms either hold. ## What Changed - `server/src/services/workspace-recovery.ts`: `reconcileRetainedWorkspaces` draws candidates from lanes. Each lane is a page that its own cursor advances: unfinished saves; for the worker, exhausted holds whose re-arm or fallback is due; and the obligation pass. Before, the obligation pass ran only when no unfinished save was selectable (the gate at `const unfinished = rows.length > 0; if (!rows.length) { ... }`). - Each lane with work takes one free slot before any lane takes a second. The lane that goes first rotates on each run, so with one free slot (a long local export holds the other) each lane still gets it in one run of three. Actions of a run that one lane took join that run's job without another slot. - The due lane wraps at its tail. A hold that becomes due by time alone (a lease expiry or a re-arm backoff) is reached after the obligation pass completed with no committed change. - `retainedWorkspaceCandidates` takes a `dueOnly` filter (due re-arm or fallback of an exhausted hold). The obligation filter uses the same condition. One `now` is used for the whole query. - The startup sweep (no worker sweep state) also shares its slots between unfinished saves and its general pass. - `doc/execution-semantics.md`: document the lanes and the fairness rule. - New `workspace-recovery-passive-fairness.test.ts` with four real-Postgres regressions. ## Verification - Red on `origin/master` ( |
||
|
DottaandClaude
|
16024ffa2a |
fix(native): a same-boot stall never leaves a run waiting on itself (#20721)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - A second gap: after back-to-back stalls mid-tool, the executor returns and nothing in this process ever looks at the run again. No terminal state, no recovery and no Retry. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - It also makes heartbeat schedule that recovery as soon as such an executor exits. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode, T1, TM1). Three stalls of about 90 s came back to back, with less than 1 s between them. Each run was in a tool call. After the second thaw, `native.session.execute` failed for all three runs (lease lost, executor detached). After that, no run had any event: no terminal state, no `native.recovery.transition` and no Retry, for 20 minutes. The in-sandbox turns kept going. A single 90 s stall (wave s9340) stays green, because a later recovery signal happens to find the run. With stacked stalls, that signal can land while the executor is still tearing down, so it skips the run and sets no deadline. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat.ts`: when the executor ends with `NativeControllerDetachedForRestartError`, there was no shutdown or hot-restart handoff, and this process's own lease lapsed (`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery for the run when the executor exits. Recovery then reattaches the remote runner whose turn is still running, restarts an unstarted bootstrap, or fails the run with a working Retry. - `claimNativeRestartRecoveries`: a self-owned lease that is still live, with no driver here, now returns a `retryAt` at its expiry, so the wait is bounded. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Heartbeat test: an executor detaches after its own lease lapsed, with no restart. The same process must take the run back. Red before (timed out, the run stayed `running`), green after. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - Daytona wave `w-20261111T050504-s9331` from this branch (same seed and shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about 1 s between them). All 3 tasks reached `done` with no Retry. Run `632f0564` (claude) shows the exact s9331 sequence: `native.session.execute` failed after 216 s, then `reattach_remote_runner` 0.4 s later (before the third stall), then a second reattach after it, and the turn completed. Two tasks are graded FAIL only on the T1 crossing check: their own fault landed after their tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their status is `done`, with no Retry. - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
08ea18085e |
fix(workspaces): restore a saved workspace into one copy of sandbox disk (#20695)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task on a remote (Daytona) sandbox continues from its saved workspace. Before the agent starts, the server uploads the saved archive into the sandbox and a sandbox script extracts it. > - The extraction wrote a full second copy of the workspace beside the uploaded archive. A 7.5 GB workspace on a fresh sandbox then needed about 15 GB, and the restore failed with `stage=extract code=ENOSPC disk=full`. > - The failure was classified as retryable, so every automatic retry and every Retry repeated it on a same-size sandbox. The task could not continue. > - On a retained sandbox it is worse: the old root, the archive, and the new tree all share the disk. > - This pull request makes the restore need about one copy of the workspace. The sandbox removes only data that control-plane storage already holds durably, and it never removes a root whose save the server did not prove. > - The benefit is that a workspace that fits once on the sandbox disk restores. A disk that is too small for one copy now gets a named failure that does not loop. ## Linked Issues or Issue Description **What happened?** A saved remote workspace of about 7.5 GB could not be restored on a new Daytona sandbox. The run failed with `remote_workspace_restore_failed: step=sandbox_restore; remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=extract code=ENOSPC disk=full exit=1`. The run marked the failure as retryable, and each retry failed the same way. **Steps to reproduce** 1. Save a remote workspace whose archive is larger than half of the free sandbox disk. 2. Start the next run on another sandbox, so that the server restores the saved archive. 3. The upload succeeds. The extraction writes the second copy and fails with ENOSPC. **Expected behavior** A workspace that fits once on the sandbox disk restores. A disk that cannot hold one copy gives a specific failure that is not retried automatically, and Retry works after the sandbox has more space. **What the disk holds during a restore (from the code)** - The uploaded archive (`<parent>/.paperclip-runtime/workspace-snapshots/<operation>/workspace.pcws`, about the workspace size, because the archive is not compressed). - The extracted tree (`<root>.restoring-*`, the workspace size). - On a retained sandbox only: the old root, which stayed until after the new root was published. - After a restore that died: its partial tree and its archive, which no later attempt removed. Refs #20678 (the save side of the same full-disk class). ## What Changed - `workspaceArchive` restore has a `consumeArchive` mode. It creates the directories and symlinks first, then writes the file bodies from the last to the first. After it verifies each file against the manifest hash, it truncates the archive behind that file. The archive and the tree together need the workspace size plus the largest file. The default mode, which the server uses for local restores, does not change. - The sandbox restore script uses that mode on a private rename of the uploaded archive. A later upload to the same path cannot change the file that is being consumed. The script removes the consumed file when it stops. - The helper's own staging is now inside the root-specific `<root>.restoring-*` folder, not in a shared `.workspace-restore-*` sibling. - While it holds the root's restore lock, and after the epoch fence, the script removes leftovers of an earlier attempt on the same root that died: `<root>.restoring-*` folders, `.consuming` archives in its own staging, and archive staging folders that a `restore-target.json` binds to the same root. Each restore writes that binding after the reclaim step. - When the server proved that a retained root is its own earlier, fully saved allocation (`replaceRetiredRoot` and a Paperclip receipt, as before), the script removes the root's files before the extraction, not after the publish. The receipt and `.paperclip-runtime` stay, so an interrupted restore proves the same root again and continues. - A full disk at any restore stage reports `reason=sandbox_disk_full`. The existing named-refusal rule makes it non-retryable, so it goes to the board recovery action and does not loop. - `doc/execution-semantics.md` describes the disk need, the reclaim, the early removal, and the new reason. ## Verification - `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-snapshot-archive.test.ts` (287 passed). - `cd server && npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-lifecycle.test.ts src/services/remote-workspace-archive-worker.test.ts src/services/remote-workspace-snapshot-archive-environments.test.ts` (122 passed, 16 skipped). - `cd server && npx tsc --noEmit -p tsconfig.json` (pass). - The new tests run the real sandbox scripts against a folder with a byte capacity. A `--require` preload makes each file write fail with ENOSPC when the folder would exceed the capacity. - Red on `origin/master`, green with this change: - Fresh sandbox, room for the archive plus one file: master fails with the exact production error `stage=extract code=ENOSPC disk=full exit=1`. The fix restores every byte and leaves only the restored files. - Retained root that the server proved, room for the old root, the archive, and one file: master fails with ENOSPC. The fix restores and keeps the runner state. - Too small for one copy: the failure is `stage=extract reason=sandbox_disk_full`, with no partial tree or archive left. A retry with more space then restores. - Both sandbox processes are killed mid-extraction (a restart): a retry with the same identity and a retry with a new identity and epoch both reclaim the leftovers and restore into the same capacity. - The command dies right after it removed a proven root's files: the next restore proves the root again from its kept receipt, restores, and keeps the runner state. - Integration test through `prepareRemoteWorkspace`: a full disk gives `retryable: false` with `reason=sandbox_disk_full`. On master the same case is `retryable: true`. - Guards (pass on master and with the fix): a retained root without the server proof, or without a Paperclip receipt, is refused with `reason=occupied_root`, and none of its files are removed. - Previous-release story: the archive format and the saved state do not change. A partial `<root>.restoring-*` tree from the previous release is reclaimed (test). Archive staging from the previous release has no binding, so it is kept (test). The previous release's shared `.workspace-restore-*` siblings cannot be bound to a root, so they are kept. ## Risks - The consumed archive cannot be read again. Each restore uploads a new copy from control-plane storage, and the early-return path for an already restored root does not read the archive. Every file is still checked against the manifest, and the manifest is checked against the saved digest. - The reclaim step removes data only under the root's restore lock and after the epoch fence. It removes only names that a restore of the same root creates. Staging that names another root, or no root, is kept. - The early removal of a retained root depends on the existing server proof (`isRetiredOwnAllocation`) and the receipt check. If a restore fails after the removal, the saved version in storage is the source of truth, and the next restore proves the same root again. - Files in a lower overlay layer of a resumed sandbox image do not free space when they are removed. Such a sandbox can still be full. - A workspace that is larger than the free disk after the upload, plus its largest file, still fails. It now fails with the named, non-retryable `reason=sandbox_disk_full`. - No change to `packages/paperclip-runner`, the database schema, or the API. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
855e241887 |
fix(server): record a long event-loop block with the deltas that name its cause (#20709)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process owns the controller leases of its runs, renews them on timers, and serves every request > - On a hosted instance, the process stopped for about 103 seconds without a restart. Every lease lapsed, and two runs needed a manual Retry (the symptom is fixed in #20694) > - Nobody can say what blocked the process. The server records no event-loop delay, and it has no CPU, GC or memory record for that minute > - This pull request logs one local line for each event-loop block of 2 s or more. The line holds the deltas that tell the possible causes apart, and a likely-cause class > - The benefit is that the next stall names its class (main-thread CPU, GC, paging, CPU throttling, host suspend, or a blocking sync call) and its exact time, so we can match it to the run timeline and fix the cause ## Linked Issues or Issue Description Refs #20694 **What happened?** On a hosted instance, the merged run-event timeline of all active runs had no events from 08:19:52.04Z to 08:21:35.37Z (103.3 s). Events from streaming runs arrived about every 0.5 s before and after the gap. The process did not restart (the run diagnostic shows an observer uptime of about 24 minutes when the reaper ran). The first event after the gap was a span with a 60 s timeout that ended after 103.5 s. Thus the timer callback ran about 43 s late. This agrees with a blocked event loop, or with a frozen process. Nothing in the process recorded which one. **Expected behavior** A long event-loop block is a product failure. It lapses every lease and freezes every user. The server must record each long block with enough data to identify its cause. **Steps to reproduce** 1. Start the server. 2. Block the main thread for more than 2 s, for example with a synchronous read on a slow disk or a busy loop. 3. Before this change: the logs show nothing. After this change: the logs show `event loop blocked for <n> ms (likely <cause>)` with the measured deltas. ## What Changed - `server/src/event-loop-stall-monitor.ts` (new): one unref'd 1.5 s timeout, re-armed after each check, measures how late it runs. (Server startup creates no interval timers, so it is not a `setInterval`.) When a check is 2 s or more late, it logs once with these fields: - stall duration, start and end time (wall clock and monotonic clock) - process CPU (user and system), GC pause time and count (`perf_hooks` GC entries) - major and minor page faults, file-system reads and writes, voluntary and involuntary context switches (`process.resourceUsage()`) - cgroup v2 `throttled_usec` delta (Linux; read asynchronously, best effort) - RSS, heap and external memory - `likelyCause`: `host_suspended`, `gc`, `memory_paging`, `cpu_throttled`, `main_thread_cpu`, or `blocked_off_cpu` (sync file I/O, a sync child process, or another off-CPU wait) - `server/src/index.ts`: starts the monitor when the listener binds. It stops the monitor at the start of shutdown, and when a failed startup closes the listener. It writes the line with the existing `logger.warn`. - This is a local log line only. It is not the Telemetry path and not the Observability (OpenTelemetry) path. No data leaves the process. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/event-loop-stall-monitor.test.ts src/__tests__/server-startup-feedback-export.test.ts` (131 passed; the monitor file passed 5 runs in a row): - an 800 ms busy loop is recorded as `main_thread_cpu`, with the stall and CPU time - an 800 ms `Atomics.wait` (blocks without CPU, like a sync read on a stalled disk) is recorded as `blocked_off_cpu` - nothing is logged while the loop is responsive, or after `stop()` (injected clock) - the file-read delta of a stall is kept while the cgroup read is pending - server startup still creates no interval timer - a 100 s stall with 90 s of cgroup throttling is recorded as `cpu_throttled`, with the stall start and end times - each cause class is selected from its deltas - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - CI supplies the broad server suites. ## Risks - Low risk. The timeout reads only process counters. Every 10 s it reads one small cgroup file asynchronously. It does not query the database or the network, so it does not keep an idle instance or its database awake. It is unref'd and stopped on shutdown. - `process.cpuUsage()` counts all threads of the process, not only the main thread. A busy libuv thread pool during a stall can make an off-CPU block look like `main_thread_cpu`. The raw deltas are in the line, so a reader can check this. - If a platform suspends and resumes the process, each resume can log one `host_suspended` line. This is expected and useful. - The monitor names the class of a block, not the call stack. If the class is not enough, the next step is a bounded CPU profile around the stall time. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the module comment carries the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
200c6e40e2 |
fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run's first provider start fails before any session opens, Paperclip retries the bootstrap on the same run. Durable controller evidence proves that no provider session or provider event exists. > - A remote sandbox keeps the Runner state inside the workspace. A replacement sandbox gets that state back when the saved workspace is restored. > - When the provider refused `session.open` (for example an Anthropic 429), the Runner still suspended cleanly. Its state says `suspended`, but the provider state has no session identity. > - The same-run restart handled a Runner that never suspended. It did not handle a Runner that suspended before the provider opened a session. It failed with `runner_harness_state_mismatch: provider_identity_incomplete (provider_not_suspended)`, and the task showed "Recovery needed" without Retry. > - This pull request lets that proven restart set the unopened session aside and start the provider again. > - The benefit is that a rate-limited provider start continues by itself after the limit resets. ## Linked Issues or Issue Description Refs #20627 and #20683 (other restore paths). This PR changes only the same-run bootstrap restart. **What happened?** Two ACPX Claude tasks on remote Daytona sandboxes, on staging: 1. The first attempt sent `session.open`. ACPX rejected it with `classification=provider_rate_limit` (Anthropic 429). The Runner exited with `lifecycle=suspended`. 2. Paperclip recorded `recoveryMode: bootstrap_retry`, `providerSessionEstablished: false`, `providerEventsExist: false`, and a retry after the limit reset. 3. One hour later the retry ran as `bootstrap_incomplete` on the same run, in a new sandbox with the saved workspace restored. 4. Host preparation inspected the restored Runner state. The Runner was `suspended`, but the provider had no session identity. The inspection threw `provider_identity_incomplete (provider_not_suspended)` before the bootstrap restart branch ran. 5. The run failed. The task stopped at "Recovery needed" with only inspect actions. **Expected behavior** The restart sets the unopened session aside and opens a new provider session, as it already does for a Runner that never suspended. **Steps to reproduce** 1. Restart a remote run as `bootstrap_incomplete` with a proven bootstrap restart. 2. Give the sandbox a Runner state for this run with `lifecycle: suspended` and a provider state with no session identity. 3. Prepare the host. On `master` it throws `runner_harness_state_mismatch: provider_identity_incomplete`. The new `interrupted_bootstrap_unopened` cases of `bootstraps only an untouched provider session in a $outcome workspace lease` run these steps for a created and a resumed lease. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - During host preparation, a Runner state that is suspended with no provider session identity returns the new incomplete reason `provider_unopened`, but only when all of these hold: - the run is restarting as `bootstrap_incomplete`; - the state belongs to this same run; - the controller already recorded the bootstrap restart proof (its durable journal shows no provider turn, and the old Runner is stopped or absent). - the provider state is the one saved before any session opened (`remoteProviderStateNeverOpened`): `prepared`, no session or identity, no turn in progress, no unconfirmed exit. Only ACPX and Codex-family states qualify. - The created-lease and resumed-lease restart branches share one check (`restartableBootstrapState`) and accept `provider_unopened` exactly like `not_suspended`. They move the old session aside and claim a new, untouched session root. - Every other case still throws `provider_identity_incomplete`: another run's state, a restart without the proof, a normal resume, and close checkpointing. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`: 47 passed. - Red on `master`: the `created` and `resumed` `interrupted_bootstrap_unopened` cases fail with `runner_harness_state_mismatch: provider_identity_incomplete (identity_binding)`. Green with this change. - Fail-closed cases stay red-proof: `interrupted_bootstrap_unopened_unproven` (no restart proof) and `interrupted_bootstrap_unopened_other_run` (another run's state) still throw. - Full `native-session-executor.test.ts` and `pnpm --filter @paperclipai/server typecheck` pass. ## Risks - Low risk. The new branch needs the same proof that already authorizes a fresh bootstrap for a Runner that never suspended. The proof comes from the controller's durable journal, not from the sandbox. - The normal path is unchanged and has no added wait. - Not proven live: I did not confirm on staging that the controller recorded the restart proof for these runs. If it did not, the run still fails closed as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
4baff83225 |
fix(chat): never batch messages of two chat conversations into one wake (#20697)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat channels let a linked user talk to an agent from GitHub, Slack and other platforms. Each external thread is a chat conversation. Many conversations can share one task. > - A reviewed chat run attests exactly one conversation. #17054 keeps each conversation in its own wake batch, but it compares only the conversation that a wake records. > - The work-intent outbox saves unanswered user messages as a plain comment wake. That wake records no conversation, even when its messages are chat messages. A wake with no recorded conversation matched every conversation, so a message from another conversation merged into it. > - The promoted run held messages of two conversations. The binding check correctly refused it (`reviewed_chat_execution_binding_not_authorized`, `binding_denied`), and the task needed Retry. > - This pull request compares the conversations of the queued messages themselves. Wakes of different conversations never merge. Each conversation gets its own run, in order. > - The benefit is that replies in several threads of one task each get an attested run, without a failed run or a Retry. ## Linked Issues or Issue Description Refs #17054 (the earlier per-conversation batching fix). Refs #20689 (a different shape of the same error). **What happened?** On staging, a GitHub pull-request review task had two runs fail with `setup_failed` / `reviewed_chat_execution_binding_not_authorized` (`reviewedChatBinding.denialReason: binding_denied`). Each failed run held the same two older saved messages plus one new message. The two runs recorded two different `chatConversationId` values. The work-intent outbox had saved the older messages as plain deferred wakes with no conversation. A new message from each conversation then merged into one of these saved wakes. **Steps to reproduce** 1. Bind two chat conversations to one `in_review` task. 2. Let the outbox save an unanswered message of conversation B as a deferred wake while a run is active. 3. As the linked user, send a message in conversation A while the run is still active. 4. See that the message of A merges into the saved wake of B. The promoted run fails the binding check. **Expected behavior** The saved message of B runs as its own wake. The message of A runs after it, in its own attested run. ## What Changed - New `server/src/modules/wake-queue/adapters/chat-conversation-batch.ts`: reads the inbound chat links (`chat_message_links`, direction `inbound`) of queued messages and finds messages of a conversation other than the expected one. - Deferred-wake lookup (`findExistingDeferredWake`): when the incoming wake has a conversation, a saved wake qualifies only if none of its queued messages is an inbound message of another conversation. This is one more predicate in the same query (no new round trip). It uses the existing `chat_message_links_company_comment_idx` index. - Coalescing into the active run (`coalesceIntoActiveExecutionRun`): returns no merge when the merged batch would hold messages of a conversation other than the one it records. The wake is then saved behind the run, which is the existing fallback. - Release drain (`findNextDeferredWake`), upgrade repair: a saved chat wake that older code already merged across two conversations is split. The other conversation's messages move to a plain saved wake (the outbox shape), ordered just before it. The original wake keeps its conversation and its own messages. The drain then reads the queue again. - `doc/execution-semantics.md`: describe the comparison and the upgrade repair. - New test `chat-wake-conversation-batch.test.ts` (embedded Postgres, real admission and release code, real `authorizeChatConversationForBoundRun`). ## Verification - Red on `origin/master` (test file only): `pnpm --dir server exec vitest run src/modules/wake-queue/adapters/chat-wake-conversation-batch.test.ts` gives 4 failed, 6 passed. The 4 failures are the bug: the new message merges into the other conversation's saved wake (`[[earlier, reply]]`), two conversations cross-merge, a message coalesces into a queued run of another conversation, and the old merged shape stays merged. The 6 passing cases pin unchanged batching. - Green with the fix: 10 of 10 pass. Each separated wake passes `authorizeChatConversationForBoundRun` for its own conversation. The old merged shape fails it with `paperclip_runner_chat_attachment_binding_denied` before the split. - Batching that stays the same (pinned, pass before and after): comment wakes with no conversation merge; messages of one conversation merge; an outbox-saved wake of the same conversation still absorbs the reply; a chat message and plain comments merge in both orders; same-conversation messages still coalesce into a queued run; two recorded conversations stay apart. - Related suites pass locally: `src/modules/wake-queue`, `src/modules/run-dispatch`, `chat-run-milestone-work`, `durable-chat-wakeup`, `heartbeat-comment-wake-batching`, `heartbeat-reviewed-chat-binding.integration`, `slack-conversation-lifecycle`, `queued-comment-consumption`, `issue-queued-comments-routes` (23 files, 1037 tests). `heartbeat-process-recovery`: see the comment below. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. No schema change and no migration. - Behavior change only for chat messages: a message no longer joins a saved wake or a run that holds another conversation's inbound messages. It gets its own wake behind them, and it runs after them. Before this change, that batch always failed the binding check. - Board comments have no inbound chat link, so they belong to no conversation. Wakes without a conversation, wakes of one conversation, and mixed board-comment and chat wakes batch exactly as before. - Ordering: the deferred queue still promotes one wake per release, oldest first. The second conversation waits for the issue execution lock. Single-assignee and atomic checkout are unchanged. - Upgrade: a saved wake that older code merged across conversations is split at promotion. No message is dropped or delivered twice: each message id moves to exactly one wake. A run that older code already promoted in the merged shape is not changed; it fails the binding check as before and keeps its existing Retry path. - No wait is added before a turn starts. Admission adds one predicate to an existing query. Coalescing adds one indexed read only when the merged batch has a chat conversation. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d79b7f29f2 |
fix(heartbeat): bind queued card-answer interrupts to the delivered answer (#20698)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can queue direction while a task's run is active, then use Interrupt to stop the run and start a successor that receives the direction > - The queued direction can be an answer to a card, for example an accepted confirmation of a saved plan > - `explicitOperatorRunIdentity` verifies the click from the coalesced queue receipt before the successor run can start > - That check required comment ids in the receipt, but a card answer carries its interaction and no comment ids > - So every claim of the successor failed with 403, and the direction never reached a run > - This pull request binds a card-answer receipt to the answer that the successor run delivers, and to the user who gave it > - The benefit is that Interrupt works for queued card answers, and the operator identity stays verified ## Linked Issues or Issue Description Refs #14043, #14510 **What happened** A person accepted a `request_confirmation` card on a task while its run was active, and then clicked Interrupt. The active run stopped ("Interrupted to send queued messages"). The successor run was then cancelled with "Cancelled because the queued run cannot be claimed: Queued-message interrupt authority is unavailable". The coalesced receipt had the interaction id and no `queuedCommentIds`. The successor run's context had the same `interactionId` and the receipt id in `queuedCommentSourceWakeIds`. The chat eval `steering.interrupt-with-direction` recorded this. **Expected behavior** The successor run starts, receives the accepted direction once, and runs with the clicking user as its responsible user. A receipt that the run does not deliver, or an answer that another user gave, still fails closed. **Steps to reproduce** 1. Start a run on a task. 2. While the run is active, accept a confirmation card on that task. The answer goes into the deferred queue. 3. Click Interrupt for the queued messages. 4. Before this change, the successor run is cancelled with `queued_run_claim_rejected`. ## What Changed - `server/src/services/run-identity.ts`: the shared receipt binding (consumed `coalesced` receipt for the same company, agent and run; same task; `user` request from the marker's actor) runs first, as before. Then: - A receipt with comment ids keeps the existing rule: all ids must be delivered to the run. - A receipt without comment ids must be a queued card answer (`queuedInteractionId`). The run context must deliver the same `interactionId`. The interaction must be on the same company and task, with the receipt's status, and `resolvedByUserId` must be the clicking user. Then the run gets the operator identity (`queued_comment_interrupt`). Otherwise the claim still fails with 403. - `server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: a successor of a card-answer interrupt is claimed, runs, and has the clicking user as responsible user. Two negative cases: the run delivers a different answer, and another user resolved the answer. Both stay rejected. Relation to #14510: that open PR fixes the same claim failure (#14043) by returning `null` for card-answer receipts, so the run starts without the operator identity. Its Risks section notes that keeping the operator identity needs a check that the actor resolved the interaction. This PR adds that check instead, so the successor keeps the identity that enqueue time records. If maintainers prefer #14510's approach, this PR's regression test still applies to it. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: the new positive test fails before the change (successor cancelled) and passes after it. The negative tests and the existing tests pass. - Suites that cover interrupt identity, each run separately: `run-identity` (37), `heartbeat-responsible-user-invariant` (18), `issue-queued-comments-routes` (129), `deferred-execution-receipt-work` (50), `task-interruption-query` (31), `explicit-native-continuation` (436), `run-cancellation` (8), `workspace-recovery-admission` (147), `wake-queue/adapters/postgres` (113): all pass. In `heartbeat-process-recovery`, the interrupt-named tests pass except `dispatches interrupted CLI input and cleans up early fixture exits`, which fails before any interrupt (the CLI child does not start: `controller_lease_lost`). It fails the same way on unchanged master on this loaded host (3 of 3 attempts failed at least one case), so it is not caused by this change. CI covers it. - `server` `tsc --noEmit`: no errors in the changed files. - Live chat eval `chat-features.runner-opencode.local.steering.interrupt-with-direction` at this branch (run 42938970466): passed. The old run stopped with "Interrupted to send queued messages". Exactly one successor run started, delivered the accepted card answer, ran with the clicking user as responsible user, and saved the approved single page and its delivery document. Before this change, the same case failed with the successor cancelled as `queued_run_claim_rejected`. ## Risks - Low risk. The change adds a verified path only for receipts that previously always failed. Comment receipts keep their checks. - If a different user answered the card than the user who clicked Interrupt, the claim still fails. The person who answered can click Interrupt. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
ce3881f1b3 |
fix(skills): release a skill cache lock whose holder's deployment was replaced (#20688)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Run preparation publishes skill revisions into a shared runtime skill cache. A file lock serializes each publication. A lock whose holder provably exited is released at once. > - That check used `controllerProcessExited`, a PID proof. It fails closed whenever the recorded process space (host, kernel boot, PID namespace) differs from this server's. > - Every Railway deploy starts the server in a new process space. A lock left by a server that a deploy stopped mid-publish can never be released. Each later preparation of that skill waits 60 s and fails with "Runtime skill cache publisher is busy … remove it", until an operator deletes the file. > - The rest of recovery already uses one shared prover (`proveOwnerGone`). Across process spaces it accepts a replaced platform deployment on the same exclusive volume, or a stopped or fenced server boot. > - This pull request makes the skill cache lock use that prover. It was the last caller of the separate PID-only proof. > - The benefit is one rule for "the controller from a previous deployment is provably gone", and no manual file removal after a deploy. ## Linked Issues or Issue Description **What happened?** A census of staging failures after deploys showed that `controllerProcessExited` always fails closed after a deploy, because the process-space hash changes. On `master`, the only remaining caller is the runtime skill cache lock check. The restart chaos harness restarts in the same process space, so the PID proof succeeded there and hid the gap. I did not find this failure in the staging run census. It is a latent dead end with the same cause as the census class. **Expected behavior** A lock whose holder ran in a deployment that the platform replaced on the same exclusive volume, or whose server boot stopped or was fenced, is released at once. Any other holder is still waited for. **Steps to reproduce** 1. Run the server on Railway with a volume. 2. Redeploy while a skill revision is being published (the lock file exists). 3. Prepare a run that needs the same skill revision on the new deployment. 4. Observe a 60 s wait and `Runtime skill cache publisher is busy; retry preparation.` on every attempt. **Paperclip version or commit** `master` at `c444da7d05` (the new test fails there). ## What Changed - `server/src/services/runtime-skill-cache.ts`: `runtimeSkillCacheLockHolderExited` builds a local owner stamp from the recorded controller (boot ID, PID, process space, start time, and the Railway deployment and volume when recorded). It returns true only for the `gone` verdict of `proveOwnerGone`. - Same-space PID rules do not change. A dead PID is gone, a live PID with no start time is unprovable, and this server's own boot is alive. - `server/src/__tests__/runtime-skill-cache.test.ts`: a new test for a holder in another process space whose deployment the platform replaced. - The proof is bounded by the lock wait (5 s, never past the deadline). A timed-out or failed proof keeps the lock held. A proof still running is joined by later waits on the same holder, so a stalled read adds at most one database query per holder. - `server/src/services/controller-process-identity.ts`: removes `controllerProcessExited`, which has no caller left; its test assertions now check `sameProcessSpace`. ## Verification - `npx vitest run src/__tests__/runtime-skill-cache.test.ts src/services/controller-process-space-identity.test.ts` (from `server/`): 31 passed. - New: "releases a lock whose holder ran in a deployment the platform replaced". It fails on `master` (`expected false to be true`) and passes with this change. - The same test keeps the lock held for: the same deployment ID, another volume, a non-exclusive volume, another service, and no recorded deployment. - The existing test "never treats age, another host, or a legacy holder as proof that a lock holder exited" still passes. Without a deployment or controller lease proof, another host stays held. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` selected 2504 of 3294 test files (the change reaches `owner-proof.ts` importers) and hit my 30 min local limit with no failure seen. Required CI supplies the broad coverage. - Live proof: not done (tests only). ## Risks - Across process spaces, a held lock can now trigger a read of `controller_instances` for the holder's boot. This happens at most once per second, and only while a lock is held. Uncontended preparation does no extra work. - The prover may fence a boot row whose lease lapsed past its grace, as it already does for every other recorded owner. - No persisted shape changes. Older lock files without a `controller` field stay held, as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
a46b9ac724 |
fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one
## Linked Issues or Issue Description
**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.
**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.
**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.
Refs #20666 (other recovery work in the same staging census; not changed
here)
## What Changed
- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.
Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.
## Verification
- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.
## Risks
- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
DottaandClaude
|
8c0dbffa70 |
fix(heartbeat): retry a legacy run whose lease this process lost before dispatch (#20694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat orphan reaper settles legacy (non-native) runs that lost their controller lease > - #16355 taught the reaper to retry a legacy run that a restart stopped while setup was still `preparing`, because revocation fences the old controller and `preparing` proves no adapter dispatch > - That proof was applied only when the lapsed lease belonged to a different boot > - A stall can make the current process lose its own lease while setup is still `preparing`, and its executor can end without settling the row > - The reaper then wrote "retrying once" but recorded no bootstrap evidence, so the retry was refused and the task was blocked for reconciliation > - This pull request applies the same revocation proof to a lease from the current boot > - The benefit is that these tasks recover by themselves instead of waiting for a person to press Retry ## Linked Issues or Issue Description Refs #16355 **What happened?** On a hosted instance, one server process stalled for about 100 seconds (lease renewals were 42–46 s late). Several legacy `paperclip_runner` runs on a remote sandbox were still in setup (`executionStage: "preparing"`, no process id). Their controller leases expired. Some executors settled their own runs as `controller_lease_lost`. Two runs had no executor left, so the orphan reaper of the same process settled them 4 minutes later. Each run got `error: "Process lost -- server may have restarted; retrying once"` and `processLossDiagnostic.retryEligible: true`, but no `executionRecovery` evidence. `enqueueProcessLossRetry` then saw a run that needs reconciliation and scheduled nothing. The next sweep recorded `legacy_execution_requires_reconciliation` with `preserve_without_replay_v1`, and both tasks stayed `blocked` until a person pressed Retry. **Expected behavior** Revocation commits the stage. Renewal requires the old boot id, so no controller can move a revoked run to `dispatching`. A revoked run that is still `preparing` with no process identity did no provider work, whichever boot owned the lease. The reaper should record the bootstrap evidence and use the bounded retry. **Steps to reproduce** 1. Claim a legacy run in this process: `controllerBootId` is the current boot, `executionStage: "preparing"`, no process id. 2. Let its controller lease expire with no local executor (for example after an event-loop or database stall). 3. Run the orphan reaper. 4. Before this change: `process_lost; retrying once`, no `executionRecovery`, no retry, and a reconciliation hold. After this change: bootstrap evidence and a `scheduled_retry` successor. ## What Changed - `server/src/services/heartbeat/recovery.ts`: `restartInterruptedBeforeDispatch` no longer requires the lease to come from another boot. The other clauses stay: legacy runtime, a stamped controller boot id, and `preparing` with no process id or group in the state committed by the revocation. Locally tracked runs and runs with a live lease are still skipped before this point. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: two cases in the hosted shape (assignment wake and due-monitor wake, `paperclip_runner`, prepared remote workspace, same-boot expired lease, `preparing`). They expect bootstrap evidence, a scheduled retry, no recovery hold after a second sweep, and no adapter call. A control case keeps a same-boot `dispatching` run held for reconciliation. ## Verification - Red on `origin/master` (source change reverted): both new cases fail. The monitor case shows the exact hosted shape: `retryEligible: true` with no `executionRecovery`. - Green with the change: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease this process lost|this process dispatched held|restart stopped before adapter dispatch|reached adapter dispatch held|lost monitor|older build without a controller stamp|failure budget"` (10 passed). - Related files, run together: `heartbeat-process-recovery`, `live-recovery-work`, `interrupted-preparation-hold-settle`, `resource-wait-stop`, `legacy-controller-lease`, `execution-recovery-dead-controller`, `legacy-process-settlement`, `run-failure-report` (876 passed; 2 timing failures under load in unrelated setup-failure tests, both pass when run alone). - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - Low risk. The proof is the one #16355 already relies on across boots: the revocation compare-and-set replaces the boot id, so the in-memory controller of this process also fails its next renewal and cannot dispatch. - A same-boot run reaches this code only when it is not in `activeRunExecutions` or `runningProcesses` and its lease expired. If an executor is stalled before it registers, it aborts at its first renewal. Setup work can run again in the retry; provider work cannot. - Upgrade safety: no persisted shape changes. The reaper reads the same fields (`executionStage`, `controllerBootId`, process ids) that older releases write. Runs that older code already parked keep their hold and their working Retry. - Runs that reached `dispatching`, recorded a process, or have no controller stamp keep the current behavior. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the code comment carries the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
a1ec05a1a4 |
fix(native): read settled harness state again after a dropped sandbox connection (#20690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one provider conversation across the per-turn runs of a task. After each turn, the controller reads the settled Runner and provider state out of the sandbox and saves it as the harness backup. > - A per-turn sandbox is released after the turn. The harness backup is then the only copy of the provider session. > - On staging, one dropped sandbox connection (`write ECONNRESET`) during that read lost the backup. The run itself succeeded, because its result was already accepted. > - The next run of the task could not resume the conversation: `PRP provider resume state is unavailable`. When the task was waiting for a reply, the run stopped at "Recovery needed" with no Retry. A person had to send a message to start a new conversation. > - This pull request reads the settled state again when the sandbox connection drops during that read. > - The benefit is that one network reset no longer loses a task's conversation. ## Linked Issues or Issue Description Refs #20627 (warm backup restore when the source sandbox is idle) and #20683 (relaunch of a settled runner). This PR changes a different step: the backup capture at the end of a turn. **What happened?** A per-turn OpenCode task ran on a remote Daytona sandbox with reuse disabled. 1. Run A accepted its result. The Runner suspended. 2. The controller read the settled state out of the sandbox. The read failed with `remote checkpoint failed: write ECONNRESET`. `harness_state.backup.persist` and `session.checkpoint.persist` failed. 3. Run A still finished as `succeeded`, because the result was durable. The sandbox was released. No harness backup of turn A existed. 4. Run B started in a new sandbox. It failed in 0.2 s with `native_session_recovery_failed: Error: PRP provider resume state is unavailable`. When the prior turn yielded for a reply, the checkpoint policy does not allow a replacement session, so the task showed "Recovery needed" with only inspect actions. On staging this happened on 4 tasks in 48 hours. In all 3 predecessor runs that I could inspect, the backup capture failed. 2 of them show the `ECONNRESET` in the run log. No deploy happened between run A and run B in one case, so a deploy is not the cause. **Expected behavior** A transient connection drop while reading settled state does not lose the provider conversation. The read is repeated, and the backup is saved. **Steps to reproduce** 1. Run the first durable turn of a remote OpenCode session. 2. Make the sandbox `syncOut` of the backup directories fail once with `ECONNRESET`. 3. Call the settled checkpoint. On `master` it throws `write ECONNRESET`, and no backup is saved. The new test `uses the bundled Linux image authority for remote opencode-new-durable-reset without controller overrides` runs these steps. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - The settled-state capture (inspect plus `syncOut` of each persistence directory into a pending directory) is now one step that can repeat. The step writes nothing durable, so it is safe to repeat. - A dropped, reset or refused sandbox connection repeats the step, at most 3 attempts, with 1 s and 2 s delays. Each repeat clears the pending directory and inspects the live state again, so the copy always matches the inspected identity. - Any other failure fails at once, as before: a state mismatch, a missing directory, an unsafe archive, or a TLS failure. - The durable backup upload and the local publish run only after a complete capture. They did not change. - `isTransientRemoteCheckpointReadFailure` classifies the error with the existing `classifyRemoteConnectionError`, plus a message match for SDK errors that lose their `code`. - The first completeness inspection and its visibility-window re-reads use the same bounded retry (`retryTransientRemoteCheckpointRead`). A dropped provider-state read keeps its `provider_state_unreadable` refusal for every other caller, but the checkpoint retry looks through it to the dropped read. Undici `fetch failed` and `no response returned` also count as drops. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts`: 1343 passed. - Red on `master`: the new `opencode-new-durable-reset` case fails with `Error: write ECONNRESET`. Green with this change. - The `isTransientRemoteCheckpointReadFailure` test checks that state mismatches, unsafe archives and TLS failures do not repeat. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The normal path has no added wait. A repeat adds at most 3 s, and only after a connection failure. - A repeated read inspects the live state again. If the state changed between attempts, the existing identity checks still refuse it. - A connection that stays down for all 3 attempts fails the same way as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
69c69e4c20 |
fix(recovery): finish the passive recovery pass before restarting it (#20680)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. One passive pass of the worker visits holds with obligations: saved input that waits, cleanup, and due re-arms and fallbacks of exhausted holds > - #17741 restarts that pass after each committed workspace-recovery change, so new work is found quickly > - On staging, changes commit continuously (stop receipts, new holds, recovery progress). Every pass restarted from the first candidates, which are settled holds whose saved input waits behind another hold, and it never reached later candidates > - After #20295, #20569 and #20661 were deployed, no due re-arm ran: five stop-proof holds and about twenty capture-policy holds kept their exhausted state, with no marker and no activity > - This pull request restarts the pass only after it reaches its tail, and it continues an unfinished pass on the next run > - The benefit is that every candidate is visited on a busy instance, so the exhausted holds re-arm, save their files and free their tasks without an administrator ## Linked Issues or Issue Description Refs #17741 Refs #20295 Refs #20569 Refs #20661 **What happened?** Exhausted workspace holds were not re-armed on staging, even after the fixes that add their re-arm were deployed. Their recovery actions stayed unchanged for days (`updatedAt` on Nov 5 or Nov 10). No `stopProofRecovery` or `capturePolicyRecovery` marker and no `issue.workspace_recovery_rearmed` activity was recorded. The worker did process new restore holds, which are selected by its unfinished pass. **What did you expect to happen?** The worker's passive pass reaches every hold with a due re-arm or fallback, re-arms it, and saves the files. **Steps to reproduce** 1. Create several settled holds with saved user input that waits, with action ids that sort before an exhausted stop-proof hold. 2. Start the real workspace recovery worker. 3. Commit a workspace-recovery notification every 50 ms, as stop receipts and new holds do on a busy instance. 4. The worker never re-arms the exhausted hold. Without the notifications, it re-arms the hold. ## What Changed - `workspace-recovery-work.ts`: a committed change marks the passive pass for restart (`restart`). It no longer resets the cursor. While a restart is pending, an unfinished pass that made progress continues on the next run. - `workspace-recovery.ts`: when the passive cursor passes the last candidate and a restart is pending, the same run starts a fresh pass from the first candidate, so a new change is not delayed by another run interval. Without a pending restart the pass records `complete` and does not wrap, so paused input does not keep the worker active. - `doc/execution-semantics.md`: document that a pass is never restarted midway. - `workspace-recovery-stop-proof.test.ts`: add a real-worker regression in the staging shape (six earlier settled holds with waiting user input, one exhausted stop-proof hold, continuous notifications). ## Verification - Red: on master, the new test "re-arms the hold on a busy instance whose passive sweep keeps selecting earlier settled holds" fails (`stopProofRecovery` stays undefined). Control: the same test without the notifications passes on master. This shows that the restarts cause the starvation. - Green: with the fix, the test passes. These suites pass: `workspace-recovery-stop-proof` (12), `execution-control-work` (27), which includes "visits a newly committed earlier Retry during a busy recovered-source pass, then becomes quiet", plus `workspace-recovery-capture-policy`, `workspace-recovery-preparation-dispatch`, `workspace-recovery-warm-handoff`, `workspace-recovery-preparation` and `server-startup-feedback-export` (279). The timing-sensitive `heartbeat-observed-workspace-preservation` suite passed 6 of 6 runs (an earlier revision that restarted only on the next run made it flaky). - Staging evidence (read-only GET requests): the 28 exhausted holds of the company have no re-arm marker and no fallback check. Many blocked tasks with settled holds have deferred wakes. ## Risks - A change committed during a pass is now picked up at the next pass, not at once. Unfinished physical work still uses its own pass first, so new recoveries are not delayed. Saved input in a passive obligation can wait up to one full pass. - Previous-release story: no persisted state changes. Holds saved by earlier releases are selected on the first full pass after deploy, with no migration. - Quiet behavior stays: a completed pass with no new change keeps the worker idle. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, browser GET reads) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a8f21a171c |
fix(cases): hold direct case status writes to terminal handoff authority (#20686)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The experimental Cases feature closes a case through a terminal
handoff to `done`
> - The terminal handoff lets an agent close a case only when the agent
owns the case's primary task
> - The direct case write routes (`PATCH /api/cases/:id` and the keyed
upsert on `POST /api/companies/:companyId/cases`) also write `status`
and `completedAt`
> - These routes did not apply the primary-task ownership rule to agent
status writes
> - This pull request applies the same ownership rule to those routes,
and sends `done` through the terminal handoff
> - The benefit is that a case closes only through the authority and
gates that the product describes
## Linked Issues or Issue Description
No public issue exists. This is the bug description.
**What happened**
An agent that did not own a case's primary task tried `handoff_case` to
`done` and got "Agents may complete only their own case work". The same
agent then sent `PATCH /api/cases/:id` with `status: "done"`. The route
returned 200 and closed the case. A chat eval (`cases.close-case`)
recorded this sequence.
**Expected behavior**
An agent changes a case with a primary task to or from
`done`/`cancelled` only with the authority that the terminal handoff
requires. `done` goes through the terminal handoff, so primary-task
completion keeps its normal gates (`doc/PRODUCT.md`, Cases section). The
dedicated `update_case` tool already excludes `status`.
**Steps to reproduce**
1. Create a case with a primary task that is assigned to agent A.
2. As agent B in the same company, send `PATCH /api/cases/:id` with
`{"status":"done"}`.
3. Before this change, the response is 200 and the case status is
`done`.
## What Changed
- `server/src/services/case-handoffs.ts`: add
`assertCaseStatusWriteAuthority`. For an agent actor, on a case that has
a primary task, a status change into or out of `done`/`cancelled` must
pass the same check as the terminal handoff (`authorizeTerminalTask`:
the agent is the primary task's assignee). A change to `done` then
returns 422 and names the terminal handoff. Board actors, non-terminal
status changes and fact edits do not change.
- `server/src/routes/cases.ts`: call the check inside the case write
transaction in `PATCH /api/cases/:id` and in the keyed upsert branch of
`POST /api/companies/:companyId/cases`.
- The terminal ownership denial now names the owning agent and the
primary task, and asks the caller to contact that agent on the task. The
message keeps its old first sentence.
- The keyed upsert falls back to the locked case row (not the pre-lock
read), so an upsert without `status` cannot undo a concurrent close.
- `server/src/routes/experimental-api-paths.ts`: add a catalog summary
and description for `PATCH /api/cases/{id}` that states the closure
rule.
- `server/src/__tests__/cases-routes.test.ts`: regression test for
non-owner `done`/`cancelled` through PATCH and upsert, the owner's
`done` through PATCH (422), owner cancel, non-owner reopen, and board
close.
## Verification
- `cd server && npx vitest run src/__tests__/cases-routes.test.ts -t
"terminal handoff's primary-task"`: fails before the route change
(`expected 403, got 200`) and passes after it.
- `cd server && npx vitest run src/__tests__/cases-routes.test.ts
src/__tests__/case-mutation-fence.integration.test.ts
src/__tests__/case-handoff-native.integration.test.ts
src/__tests__/case-handoff-legacy.integration.test.ts
src/services/case-handoffs.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/heartbeat-case-terminal-handoff.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/case-history-api-discovery.test.ts
src/services/native-runtime/runner-api.test.ts`: all pass. (One local
run of the legacy integration suite failed in fixture scratch
preparation under load; two reruns with the change pass.)
- `server` `tsc --noEmit`: no errors in the changed files. The fresh
worktree had unrelated errors from unbuilt workspace packages; CI covers
the full typecheck.
- Live chat eval
(`chat-features.runner-opencode.local.cases.close-case`, run
42936492869, at the first commit): the non-owner agent's terminal
handoff was denied with the owner named, and the agent did not change
case status through another route. The eval did not pass: the first chat
turn did not finish inside the scenario's 180-second observation window.
That is a separate latency issue in the eval.
## Risks
- Behavior change for agents: an agent can no longer set `done` through
PATCH on a case with a primary task, and a non-owner agent can no longer
cancel or reopen it. The agent skill reference already says "Do not
change case status separately".
- Cases without a primary task keep the old behavior, because they have
no owner to check and the terminal handoff does not apply to them.
- The check reads the primary task without a row lock, to keep the task
-> run -> case lock order. A concurrent reassignment can race one
request. The terminal handoff itself still locks and rechecks.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude
|
b93df4ca38 |
fix(chat): consume a saved chat wake whose message a completed run answered (#20689)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - External chat (for example GitHub review requests) wakes the
assigned agent with a saved chat message
> - When the agent is already running, the chat wake is saved behind
that run and released after it finishes
> - The running agent often reads the new message as part of its current
turn, so the message is already answered when the saved wake is released
> - The release kept that chat wake anyway, removed its now-delivered
message ids, and started a reviewed chat run with no message
> - A reviewed chat run with no message cannot be attested, so it failed
with `reviewed_chat_execution_binding_not_authorized` and the task
became blocked
> - This pull request treats a chat message wake like a comment wake:
when a completed run already received all its messages, the saved wake
is consumed
> - The benefit is that a message that arrives during an active run no
longer blocks the task after it was answered
## Linked Issues or Issue Description
**What happened?**
A GitHub chat message arrived for a task while the assigned agent's run
was active. The wake was saved behind that run. The run read the message
and answered it. When the run finished, the saved chat wake was still
promoted. Its message ids were removed because a completed run already
received them. The new run had source `chat:<provider>` and no message,
so reviewed chat attestation denied it with `context_mismatch`. The run
failed with `reviewed_chat_execution_binding_not_authorized`.
Review-path recovery inherited the same context and failed too, and the
task was blocked.
**Steps to reproduce**
1. Assign a task in review to an agent that is running.
2. Deliver an external chat message for the task. The wake is saved
behind the active run.
3. Let the active run receive the message as wake input and succeed.
4. Release the task's execution lock.
5. See that the saved chat wake is promoted with no message, instead of
being consumed.
**Expected behavior**
The saved chat wake is consumed with "Queued messages were already
delivered to a completed run". No new run starts. The chat does not get
a "not started" notice for a message that was answered.
**Paperclip version or commit**
`origin/master` at
|
||
|
DottaandClaude
|
06af074452 |
fix(native): deliver the retry after a failed native turn, once per chain (#20681)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime decides a task's next status after each run, through the status arbiter and the status-decision committer > - When a native turn fails without a classified provider error, the arbiter keeps the task's status and asks for one retry (`schedule_retry`) > - The native-status wake dispatcher refused that retry wake, because its provenance check accepted only `enqueue_continuation` effects > - So the task stayed `in_progress` with no live run, no queued wake and no visible cause, until a person noticed it > - This pull request lets the dispatcher accept the committed retry effect, keeps the retry on its chat route, and limits the automatic continuation to one per chain > - The benefit is that a failed turn always ends with either queued work or a blocked task with a working Retry ## Linked Issues or Issue Description **What happened?** A native run that ends with a failed turn and no classified provider error leaves its task stranded. Examples: an OpenCode server that the sandbox killed with SIGKILL during a turn (`opencode_server_exited`), or an OpenCode turn that ended without an answer (`opencode_empty_response`). The run's accepted result is finalized as `run_failed_partial_evidence_preserved`. The committer queues a `schedule_retry` wake. The dispatcher then skips the wake with `chat_control_recovery_proof_unresolved`. The task stays `in_progress` with no run, no wake and no blocker. **Steps to reproduce** 1. Run a native task. Make the provider turn fail with `opencode_server_exited` or `opencode_empty_response` and an accepted `needs_review` result. 2. Finalize the run. 3. Run the native-status wake dispatcher. 4. See that the retry wake and its dispatch receipt are `skipped`, and that no run starts. **Expected behavior** The task gets its automatic continuation. If that continuation fails the same way, the task is blocked with a clear board blocker and a Retry that starts a new run. **Paperclip version or commit** `origin/master` at |
||
|
DottaandClaude
|
7435abcc63 |
fix(native): wait out a run-event append before saving provider-pack admission (#20682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run in a remote sandbox records a provider-pack admission receipt on its run row after the provider session opens. Artifact retention then requires that receipt. > - `recordNativeProviderPackAdmission` locked the run row with `FOR UPDATE NOWAIT`. > - A run-event append locks the same run row for a few milliseconds (`FOR NO KEY UPDATE` since #20670, `FOR UPDATE` before). Both conflict with `FOR UPDATE`. Startup appends many events at the same moment. > - When the two met, the admission write failed with `55P03`. The admission state stayed `pending`. Artifact retention then failed the run with `native_provider_artifacts_admission_unverified`. The task stopped as "Recovery needed" with no Retry. > - This pull request makes the admission write wait for the run-row lock within the existing 1000 ms `lock_timeout`. The coordinator claim keeps its `NOWAIT` fence. > - The benefit is that an ordinary concurrent event append no longer fails a healthy run before its first turn. ## Linked Issues or Issue Description Refs #20670 (changed the append lock mode; the conflict with `FOR UPDATE NOWAIT` remains). **What happened?** On a hosted staging instance, three OpenCode runs on remote sandboxes failed in the first minute with `native_provider_artifacts_admission_unverified`. Each run log shows `native provider-pack admission receipt could not be saved` about 0.2–0.7 s after the runner authenticated. The run profile kept `nativeProviderPackAdmissionState.state = "pending"`. The tasks stayed blocked. **Steps to reproduce** 1. Start a native run and hold its `heartbeat_runs` row lock in another transaction, as a run-event append does. 2. Call `recordNativeProviderPackAdmission` for that run. 3. Observe `could not obtain lock on row in relation "heartbeat_runs"` (`55P03`). **Expected behavior** The admission waits a short, bounded time for the append to commit, then saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` sets `lock_timeout = 1000ms` for every call, not only `waitForClaim` calls. - The run-row lock is now `FOR UPDATE` without `NOWAIT`, so it waits for a concurrent append within that bound. - The coordinator (`native_run_finalizations`) lock keeps `NOWAIT` when the caller does not wait for the claim. The existing fence tests still pass. - New PostgreSQL test: `waits for a concurrent run-event append's run-row lock instead of losing the admission` (both `waitForClaim` modes). ## Verification - RED on `origin/master` (`c444da7d05`): the new test fails with `55P03` in both modes. - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts` (37 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Low risk. The normal path has no contention and does not wait. Under contention the admission waits at most 1000 ms, before any provider turn. - A lock wait between the coordinator row and the run row can now meet a deadlock. PostgreSQL detects it and raises an error. That error takes the same caught path as the earlier `55P03`, so the result is no worse than before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
6e524d8a69 |
fix(workspaces): stop retrying a copy-back that hit a full sandbox disk (#20678)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a native run, Paperclip copies the agent's sandbox workspace back to durable storage. This is finalization. > - The copy-back builds an archive inside the sandbox. When the sandbox disk is full, the copy-back fails with `remote_workspace_sandbox_disk_full`. > - Finalization treated that failure as transient. It retried three times. Each attempt took about 15 minutes for a large workspace, and nothing freed space between attempts. > - Saved-work recovery already reclaims the retained sandbox's inactive provider cache when a save hits a full disk. It runs only after finalization ends. > - This pull request ends finalization on the first full-disk copy-back failure, with the accepted result kept, so saved-work recovery starts at once. > - The benefit is that the task gets its files saved sooner, and the user sees what happened instead of "failed after 3 attempts". ## Linked Issues or Issue Description Refs #19619 (related: records why the read-only capture fallback failed; this PR does not change that path) **What happened?** On staging, a native run finished its turn and its result was accepted. The workspace copy-back then failed three times with `remote_workspace_sandbox_disk_full` (about 7.5 GB of workspace, about 6 GB free). The three attempts spanned about one hour across deploys. Finalization then ended as `native_workspace_sync_out_retry_exhausted`. Saved-work recovery reclaimed the inactive provider cache (about 3 GB freed) and saved the files. **Expected behavior** A full sandbox disk is recognised on the first failure. Paperclip does not spend the finalization budget on attempts that cannot succeed. The agent's result is kept. The user sees that the disk is full and that Paperclip saves the files again without another provider turn. **Steps to reproduce** 1. Run a native task on a remote sandbox whose workspace is larger than the free sandbox space. 2. Let the provider turn finish. 3. Observe three copy-back attempts that each fail with `remote_workspace_sandbox_disk_full`. ## What Changed - `recordRetryableFailure` treats a workspace failure whose cause starts with `remote_workspace_sandbox_disk_full:` as an exhausted budget on the first attempt. The code stays `native_workspace_sync_out_retry_exhausted`, so the existing export hold, saved-work recovery, and export Retry apply unchanged. - The run error for that case says that the sandbox disk is full and that Paperclip saves the files again without repeating the agent's work. It no longer says "after 3 attempts". - Other copy-back failures keep their bounded retry. - `captureRemoteWorkspaceArchive` rethrows the read-only fallback's own failure when that fallback only lost its connection (`transport_closed`, `provider_unavailable`, `provider_rate_limited`, `sandbox_state_transition`). That failure is not disk-full, so finalization keeps its bounded retry for it. Other fallback failures keep the disk-full code. - `remoteWorkspaceFailureCause` records the server-built read-only capture failure, so a failed save names where the fallback stopped. - The transport exports `REMOTE_WORKSPACE_SANDBOX_DISK_FULL_CODE`. The finalizer and tests use it. - `doc/execution-semantics.md` describes the new rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-finalizer-telemetry.test.ts src/services/remote-workspace-transport.test.ts` (195 pass). - New test "hands a full sandbox disk to saved-work recovery on the first copy-back failure" fails on `origin/master` (the failure stays `retryable_failure`) and passes with this change. It completes an accepted result first and checks that the result stays linked and that no agent wakeup is created. - New transport test "keeps a full disk only when the read-only fallback did not just lose its connection": the two dropped-connection cases fail on `origin/master` and pass with this change. - New test "keeps other copy-back failures on their bounded retry" passes before and after. - `cd server && npx tsc --noEmit` passes. `pnpm verify` selected 2504 test files because the finalizer is widely imported; its first selected batch passed, and required CI supplies the full run. ## Risks - Low risk. The terminal state is the same state that three failed attempts produce today. Only the timing changes. - A sandbox whose disk frees space by itself between attempts now waits for saved-work recovery instead of the next finalizer attempt. Saved-work recovery repeats the save with its own bounded attempts, so this does not lose work. - Upgrade safety: a run saved by an older release in `retryable_failure` with this cause ends on its next full-disk failure. No migration is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. Extended reasoning was not configured. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4a61a2501e |
fix(storage): never promise a storage retry no storage wait owns (#20676)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run that hits a file-storage outage gets a bounded storage wait: a retry that starts again after storage is back > - The run error says "Paperclip will retry automatically when storage is back" > - On staging, a server restart reattached a remote native runner while its turn was still running. A storage outage then failed the reattachment > - Restart recovery kept the workspace under a no-replay hold, so the run took the code `native_continuation_requires_reconciliation`. Only a `storage_unavailable` run gets a storage wait, so no retry was scheduled. The run still showed the storage retry promise, and the task stayed blocked for about 15 hours > - This pull request makes the storage text follow the run's final code and the real storage wait outcome, and reads old rows truthfully > - The benefit is that every storage message is true: it promises a retry only when one is scheduled, and it offers a working Retry when nothing retries the run ## Linked Issues or Issue Description **What happened?** A native run on a remote sandbox was reattached after a server restart. Its provider turn was still in flight. File storage failed during the reattachment. Restart recovery settled the run as `native_continuation_requires_reconciliation`, kept its files, and did not replay the unfinished turn (this is the existing policy for a turn with an unknown outcome). The run error said "Paperclip will retry automatically when storage is back". No storage wait was scheduled, so nothing retried. The task stayed blocked until an operator clicked Retry. **Root cause** `server/src/services/heartbeat/run-completion.ts` chose the message and the error code separately. The message used `isStorageUnavailableError(err)`. The code used a chain where `nativeTerminalFailureCode` wins over `storage_unavailable`. The storage wait is scheduled only when the code is `storage_unavailable` (and no native terminal failure is recorded). So the message promised a wait that the code made impossible. The same false promise stayed when the storage wait was refused (`not_scheduled`) or its write failed. **Expected behavior** A storage failure promises an automatic retry only when the storage wait is scheduled. Otherwise it offers Retry. Rows saved by earlier releases read the same way after deploy, with no migration. **Steps to reproduce** Fail a native run with a storage-unavailable error after its native coordinator is settled as `terminal_failure` / `native_continuation_requires_reconciliation`. See the new regression in `run-completion.test.ts`. ## What Changed - `storage-unavailable.ts`: add the held messages (outage and rejected credentials). They say Paperclip is not retrying automatically and offer Retry or a message. Add `storageUnavailableHeldMessage()` and `presentStorageFailureMessage()`. - `run-completion.ts` (`failRun`): compute the error code first. Use the storage retry promise only when the code is `storage_unavailable`. Under any other code (for example a native recovery hold), use the held message. No storage wait is created for such a run, because it could only be refused. - `run-completion.ts` (`failRun` and `failRunSetup`): replace `markStorageWaitExhausted` with `settleStorageWaitMessage`. It keeps the promise only when a retry is scheduled. It writes the exhausted text when the budget is spent, and the held text when the wait was refused, its write failed, or a case handoff took over. - `run-completion.ts`: when the promise is withdrawn, the failure's `error` event (written before the retry decision) gets the same final text, so the transcript does not keep the promise after reload. - `run-completion.ts` (`failRunSetup`): a lost controller lease keeps its own code and text, not the storage promise. - `run-lifecycle.ts`: `decorateHeartbeatRunRuntimeStatus` presents a saved storage retry promise under a code other than `storage_unavailable` with the held text. This covers run detail and issue run presence. `listRuns` in `run-state.ts` applies the same projection, so the company run list and the Inbox failure read agree. Rows from earlier releases read truthfully with no migration. - `doc/run-log-events.md`: document which runs get a storage wait and what each message means. Previous-release story (upgrade-safe run state): this change adds no persisted shape. Rows saved by the previous release carry forward. A row with the storage promise under another code reads with the held text. A `storage_unavailable` row keeps its text. The native hold policy is unchanged: an unfinished turn with an unknown outcome is not replayed automatically, and the existing Retry continues it. ## Verification - Red before the fix (assertion failures, not setup errors): the new `run-completion.test.ts` cases received "Paperclip will retry automatically when storage is back" under `native_continuation_requires_reconciliation`, and the `run-lifecycle.test.ts` cases read the saved promise unchanged. - Green after the fix: - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts src/services/heartbeat/run-lifecycle.test.ts -t storage` (13 passed) - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts src/services/heartbeat/run-lifecycle.test.ts src/storage/storage-unavailable.test.ts` (213 passed) - `cd server && npx tsc --noEmit -p tsconfig.json` (clean) - Existing storage cases still pass: a scheduled storage wait keeps its promise and suppresses rapid recovery; an exhausted wait writes the exhausted text. - `pnpm verify` exited 2 (a doc is a global CI input), so required CI supplies the broad coverage. - The full `run-completion.test.ts` file passed every test locally. Its `afterAll` hook timed out under machine load (load average about 17), also with the new tests skipped. ## Risks - Low risk. The change affects only the text of failed runs and one read projection. It does not change retry scheduling, recovery ownership, or replay policy. - The read projection matches this repository's own two promise constants under a code other than `storage_unavailable`. A future change to those constants must keep the projection in sync. - Error events already written by earlier releases keep their original text in the run log. Live events already published before a withdrawn promise are not resent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser reads for GET-only diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
2eede8b318 |
fix(native): give a relaunched settled runner its admitted provider pack (#20683)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a server restart or a same-run retry, a remote native run reattaches to its runner in the sandbox (`reattach_remote_runner`). > - When the controller saved the runner artifact, the executor prepares that run as an adoption. It skips provider-pack selection, because a live adopted runner keeps its launched provider. > - The reattach probe can instead find the runner settled and exited (`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the transport relaunches the runner from its durable state. That relaunch must start the provider again, so it needs the provider-pack launch inputs. > - The executor did not give them. OpenCode failed with "OpenCode executable is missing from the provider pack". ACPX failed with "ACPX launch profile omitted its provider-pack authority or verified artifact digests". Both ended the run. > - This pull request selects the saved (admitted) provider-pack authority on that settled-exit relaunch and verifies it as an exact installed identity, as an ordinary reattach does. > - The benefit is that a restart or a same-run retry can resume an exited runner with the same provider pack, instead of failing. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, two OpenCode runs and one ACPX (Claude) run were interrupted by a deploy (SIGTERM) or ended a turn without a result. The server scheduled a same-run retry that reattached to the existing remote runner. The runner had exited cleanly (`runner_remote_process_exited lifecycle=suspended`). The retry then failed within three seconds with `runner_remote_provider_artifact_incompatible`. **Steps to reproduce** 1. Prepare a remote OpenCode or ACPX run with a persisted runner artifact (`controllerPersistence.runnerArtifact`). 2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`, where the runner process is not running and its state is `suspended` and settled. 3. Observe that the transport options have no `adoptExistingRunner`, no `providerPackAuthorityDigest` and no `opencodeCommand` / `acpxSidecarSha256`. **Expected behavior** The relaunch gets the admitted provider pack's launch inputs, or it fails with a clear pack-identity error. ## What Changed - `createRunnerdBackend` (native session executor): when a persisted-runner reattachment finds no live runner to adopt, select the saved provider-pack authority (`reattach: true`), so `expectedProviderPackManifest` is the admitted pack. - `prepareRemoteRunnerAssets` no longer returns early for that relaunch. It verifies the installed pack as an exact identity and sets the active pack root, the same path as a reattach without a persisted artifact. - A live adopted runner is unchanged: it still skips pack selection. - The test `uses the bundled Linux image authority for remote opencode|claude-x64` now also covers the settled-exit relaunch for OpenCode and ACPX. ## Verification - RED on `origin/master` (`c444da7d05`): the extended test fails for `opencode` and `claude-x64` (the relaunch options have no provider-pack inputs). - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "uses the bundled Linux image authority"` (14 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Upgrade safety: the relaunch reads the saved authority, so a newer controller does not select a newer pack for an older session. It requires an exact installed identity. If the sandbox no longer has that pack, the relaunch fails with a pack-identity error, and the existing Retry path applies, as before. - No change to a live adoption, to fresh runs or to the normal path before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
e33ede9a8a |
fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
cec903c6f0 |
fix(native): retire a finished remote owner whose gone Runner left a delivery unacknowledged (#20677)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native follow-up run on a remote sandbox first proves that the previous per-turn owner finished. Then it retires that owner's controller journal and starts a new Runner. > - The proof requires a settled journal. A command that the Runner did not acknowledge holds the owner. > - On staging, a per-turn owner committed its result and its Daytona sandbox was destroyed after the turn (`paperclip_runner_destroy_after_turn`, cleanup `success`). Its journal still held a `run.attach` or a `semantic_tool.result` that the Runner never acknowledged. > - The next run failed with `prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(run.attach.pending)` (or `semantic_tool.result.pending`). A Retry or a new message gets the same journal, so it fails the same way. The task stays blocked. > - Retirement already excuses the Runner's own lifecycle commands when the prior sandbox lease ended on another sandbox. This pull request applies the same rule to deliveries that only that gone Runner could receive. > - The benefit is that these follow-ups start by themselves. Commands that carry user intent still hold the owner. ## Linked Issues or Issue Description **What happened?** A census of staging failures (48 h, Nov 9–10) found 6 failed runs with `prior_owner_terminal_state_indeterminate` and `retirement_refused=journal_commands_unsettled(run.attach.pending | semantic_tool.result.pending)`. The last one was at 18:56 UTC on Nov 10. In two examples that I checked: - The previous run succeeded. Its result was committed and its coordinator lease was released. - Its lease policy was `ephemeral`. The lease is `expired`, with `failureReason=paperclip_runner_destroy_after_turn` and `cleanupStatus=success`. - The follow-up got a new sandbox (a different provider lease) and refused in about 11 s. It started no Runner. The task became `blocked`, and the next action was "Send a message to continue". **Expected behavior** When the record proves that the previous Runner is gone, an undelivered attachment, tool answer or read cannot run. The follow-up retires the finished owner and starts a new Runner. Retirement replays no command. **Steps to reproduce** 1. Finish a per-turn remote native run whose controller journal keeps a `run.attach` or `semantic_tool.result` command in `pending`. 2. Let the sandbox lease of that run end (`expired` or `released`) on its own sandbox. 3. Start a follow-up run that gets a new sandbox. 4. Observe `retirement_refused=journal_commands_unsettled(run.attach.pending)`. **Paperclip version or commit** Staging builds of Nov 10. The refusal still reproduces on `master` at `48c6e996da` (the new tests fail there). ## What Changed - `native-session-executor.ts` (`readSettledRunnerdProjection`): when the prior run's sandbox leases ended on another sandbox (`remoteRunnerGone`, the existing proof), unfinished `run.attach`, `semantic_tool.result`, `session.snapshot` and `session.goal.get` commands no longer hold the retirement. These join the lifecycle commands that the rule already excused. - In the same case, a journaled `semantic_tool.result` answers its `semantic_tool.input`. The controller journals the answer after the tool ran. Only its delivery was outstanding. - No other proof changes. The prior run must still be terminal, with a committed result, a released coordinator lease, no recovery state, a stopped or fenced controller, a delivered `run.terminal`, and no live work after it. Quarantine copies must pass the same checks. - `doc/execution-semantics.md`: describe the rule. ## Verification - `server/src/services/native-runtime/native-recovered-owner-followup.test.ts` (real Postgres): - New: "retires a finished remote owner with an unacknowledged pending-attach / pending-tool-result when its sandbox lease ended on another sandbox". It fails on `master` with the exact staging errors `journal_commands_unsettled(run.attach.pending)` and `journal_commands_unsettled(semantic_tool.result.pending)`. It passes with this change, and the log shows the retirement. - Held cases stay held: a lease that is still retained, a lease on this run's sandbox, no lease record, and a tool input with no journaled answer (`journal_tool_input_unanswered`). - New: "keeps a stopped local owner held by an unacknowledged run.attach". A local Runner is not gone by record. - Whole file: 95 passed. - Neighbouring suites pass: `explicit-native-continuation.test.ts` and `native-controller-recovery.integration.test.ts` (460 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2: a global CI input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Live proof: not done. The redeploy chaos scenario (K15) is not on `master` yet. Tests only. ## Risks - The rule applies only when every sandbox lease of the prior run is `released` or `expired`, has a release time, and names a different provider lease than this run's. A reusable sandbox that this run resumed keeps the existing hold. - A command that carries user intent (`turn.start`, `turn.steer`, `request.resolve`, `interaction.receipt`, `external_provider.operation`, `session.goal.set`/`clear`, `session.close`, `run.cancel` and others) still holds the owner. - Retirement archives the old journal under `continuity-breaks` and replays nothing, as before. - No persisted shape changes. Old saved journals are read by the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ad0543ae90 |
test(recovery): re-arm a capture-policy save whose finished coordinator names a lapsed owner (#20672)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Workspace recovery re-arms a save that an older capture policy exhausted (#20569), so the files are saved without a person > - On staging, one save of this kind was not re-armed after the deploy. Its export hold also carries a workspace finalization block, and it has a resolved no-replay sibling for the same run > - #20661 found that discovery skipped a hold when the run's finished coordinator still named a lease owner whose lease had lapsed. The same discovery clause gates the capture-policy re-arm > - This pull request adds a regression test for that exact capture-policy shape. It shows that #20661 alone makes the worker re-arm and save it > - The benefit is that the capture-policy path keeps this guarantee in later changes ## Linked Issues or Issue Description Refs #20661 Refs #20569 **What happened?** An exhausted capture-policy save (`export_interrupted`, cause `provider_error`, 6 of 6 attempts, phase `preserve_files`) was not re-armed on the worker's first pass after #20569 was deployed. **What did you expect to happen?** The worker re-arms the save once under the current capture policy, saves the files, and returns the hold to the system. **Steps to reproduce** 1. Exhaust a save under capture policy 1 with `export_interrupted`. Add a workspace finalization block and a resolved no-replay sibling for the same run. 2. Leave a lapsed `native-finalizer:` lease owner on the run's `terminal_failure` coordinator. 3. Run the workspace recovery worker. Before #20661, `nextWorkspaceRecoveryAt` returns null and the hold is never selected. ## What Changed - `workspace-recovery-capture-policy.test.ts`: add "re-arms a save whose finished coordinator still names a lapsed finalizer owner". The test uses the staging record: export hold with `historicalWorkspaceSource` and `workspaceFinalizationBlock`, a resolved no-replay sibling, waiting user input, and a lapsed finalizer owner. It drives the worker's own discovery loop until the worker is idle. - No production code changes. ## Verification - Red: with the source files from before #20661 (`git checkout 686893636d^ -- <the four source files>`), the new test fails. `nextWorkspaceRecoveryAt` returns null, so the worker is never woken. - Green: on current master, `pnpm --filter @paperclipai/server exec vitest run src/services/workspace-recovery-capture-policy.test.ts` passes (5 tests). The file passed 4 runs in a row. - The test asserts that the save completes under policy 2, the hold returns to the system, `releasedLapsedOwner` is recorded, and the lapsed owner is released. ## Risks - Low risk. This change adds a test only. - Previous-release story: the test uses rows in the shape an older release saved, with no capture-policy marker. No migration is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, browser GET reads) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e9db5e0d03 |
perf(github): read the inherited review waits of an issue list in one pass (#20673)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can review a GitHub pull request. After the agent publishes the review, the task can wait in review for the next GitHub reply. A later run of the same task can inherit this wait. > - Issue lists and inbox attention show these waits. For each waiting task, the server reads the inherited GitHub review wait: the source comment, the latest review, and the review publication. > - The server did these reads one time for each waiting task. A list request with N waiting tasks sent N copies of the prepared review statement and N copies of the publication read. > - On a hosted database, the prepared review statement ran about 270,000 times in about 2.5 hours (up to 87 calls per second). The run identity read before it ran the same number of times. The list read that calls it ran only about 1,400 times, so each list read started about 190 review reads on average. No other caller reads the waits of many runs in a loop. > - This pull request reads the reviews and the publications of all waits of one list request with one statement each. Each wait gets the same result as before. > - The benefit is about 200 times fewer review statements and fewer pool connections for each issue list. ## Linked Issues or Issue Description Refs #20400 (shares review-wait reads between the callers of one request). Refs #20551 (authorizes the waits of one issue page with one read; this change does the same for the inherited GitHub review reads of that page). **What happened** `listExternalReviewWaits` called `readInheritedGitHubReviewWait` one time for each candidate wait, inside `Promise.all`. Each call sent a run identity read, a source-message read, the prepared `inherited_github_review_wait` statement, and a publication read. A list with N waits sent each statement N times. **Expected behavior** One list request sends one review statement and one publication read for each company. The result of each wait does not change. **Steps to reproduce** 1. Put many tasks of one company in review with an inherited GitHub review wait. 2. Open the issue list or the inbox. 3. Count the statements of the database. The review statement runs one time for each waiting task. ## What Changed - `server/src/services/chat-channels/github/review-wait.ts`: - Split `readInheritedGitHubReviewWait` into shared steps: the source comment read, the review checks, the publication key, and the published result. The single-run read uses the same prepared statement and the same checks as before. - Add `readInheritedGitHubReviewWaits(db, runs)`. It reads the source comment of each run with the same lineage check as before. Then, for each company, one `DISTINCT ON` statement reads the latest review for each (source comment, task, agent) key, with the same joins, filters and order as the prepared statement. One read finds the processed publications. Each run gets the same result as a separate call: `null`, `revoked`, or `published`. - `server/src/services/native-runtime/external-review-wait.ts`: `listExternalReviewWaits` uses the batch read. The rest of the list and its authorization do not change. - `server/src/__tests__/github-review-wait.test.ts`: a new test reads 18 runs in one batch and compares each result with a separate single-run read. The runs include each revocation case (connection disabled or inactive, review state, review agent, repository, repository id, thread, publication, identity, company), a newer unreviewed head, no identity, no task, an endpoint of another agent, a run in another company and a duplicate run. The test also makes sure that the batch sends one review statement and one publication read for each company. The existing revocation tests also check the batch read. - `server/src/__tests__/helpers/github-review-wait.ts`: the seed helper can add a wait to an existing company and agent. ## Verification - `pnpm vitest run server/src/__tests__/github-review-wait.test.ts server/src/__tests__/github-review-check-state.test.ts server/src/__tests__/issue-list-review-wait-authorization.integration.test.ts server/src/__tests__/blocked-inbox-review-wait-scope.test.ts` - Focused local run of every test that imports the changed modules or the seed helper (7 files, 335 tests passed), including `external-chat-wait.integration.test.ts`, `external-review-wait.test.ts` and `legacy-execution-recovery.test.ts`. The change to the shared seed helper makes `pnpm verify` select the full server suite, so CI supplies that coverage. - Local measurement on real PostgreSQL (embedded PostgreSQL, `ANALYZE` after seeding). One company and agent, each wait has 5 reviews and 10 outbound messages. "Before" is the previous list code: one `readInheritedGitHubReviewWait` call for each run in `Promise.all`. "After" is `readInheritedGitHubReviewWaits`. The runs alternate order in each pair. Both return equal results in every sample. | Waits | Version | Statements | Review statements | Median ms | Min–max ms | Samples | | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 800 | 200 | 106.8 | 98.0–117.5 | 10 | | 200 | after | 402 | 1 | 44.3 | 41.6–50.9 | 10 | | 20 | before | 80 | 20 | 9.8 | 8.6–14.7 | 12 | | 20 | after | 42 | 1 | 6.3 | 5.3–7.0 | 12 | - `EXPLAIN (ANALYZE, BUFFERS)` of the batch review statement for 200 waits: index scans on each join, 6.7 ms execution. The single statement: 0.05 ms execution and about 2.7 ms planning for each call. - The run identity read and the source-message read still run one time for each wait. They are cheap on the hosted database (about 0 ms each), and they keep the lineage check for each run. ## Risks - Low risk. The batch statement uses the same joins, filters and order as the prepared statement, and `DISTINCT ON` keeps the first row for each key. The array filters can also match a mix of two keys. The code uses only rows of exact requested keys. - The single-run read and its prepared statement do not change. The finalizer, the status committer, and the attachment reuse paths still use it. - Errors do not change: a 403 from the lineage check gives `revoked`, and other errors are thrown. - No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c444da7d05 |
perf(run-events): stop blocking checkpoint FK checks during event appends (#20670)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run sends a stream of runner events. The server appends each event to the run log. The runner controller then saves a checkpoint for recovery. > - Each checkpoint save inserts the new `event:N` row and updates the `cursor` row in `native_runner_checkpoints`. The new row has a foreign key to the run row in `heartbeat_runs`. > - The event append locks the same run row `FOR UPDATE`. That lock blocks the `FOR KEY SHARE` lock that the foreign key check takes. So each checkpoint save waits for the event append that is in progress. > - On a hosted database, the two-row checkpoint upsert had the largest total SQL time. The one-row (cursor-only) upsert, which has no foreign key check, was much cheaper. > - This pull request changes the append lock to `FOR NO KEY UPDATE`. Appenders still serialize. Child foreign key checks no longer wait. > - The benefit is shorter checkpoint saves, shorter waits before each runner ACK, and less lock wait on the database. ## Linked Issues or Issue Description **What existing behavior does this improve?** The controller checkpoint save for native runs (`nativeRunnerCheckpointWriter`), and the native run-event append (`appendHeartbeatRunEvent`). **Subsystem affected** Server run-event log and native runtime checkpoint persistence. **Current behavior** On one hosted database, Neon query performance showed this statement as the largest total: `insert into native_runner_checkpoints ... values (...), (...) on conflict (run_id, part) do update`. It ran 6,582 times at 62.6 ms (412 s) in the first 38 minutes after a compute restart, and 9,435 times at 31.8 ms (300 s) about two hours later. An earlier day showed 100,650 calls at 102 ms. The one-row form of the same upsert ran 965 times at 13.4 ms. The runner already writes only changed records (`captureCheckpoint` compares plaintext), so unchanged material is not rewritten. The two-row form is a new event row plus the cursor. Only the new row fires the foreign key check. `appendHeartbeatRunEvent` locks the run row `FOR UPDATE` for each native event, in a transaction with several statements. `FOR UPDATE` conflicts with `FOR KEY SHARE`, so the check waits for that transaction. **Proposed behavior** `appendHeartbeatRunEvent` locks the run row `FOR NO KEY UPDATE`. This mode still conflicts with other appenders, `FOR SHARE`, `FOR UPDATE`, deletes and updates. It does not conflict with `FOR KEY SHARE`. The append never changes key columns: `runUpdate` is limited to process fields, and the native and retry-exhaustion paths reject it. The fast path already uses a plain `UPDATE`, which takes the same lock mode. The execution-admission adapter already locks this row `FOR NO KEY UPDATE`. **Reason and benefit** The wait goes in both directions. A checkpoint save waits for an append. An append also waits for an open checkpoint transaction that holds `FOR KEY SHARE`. Both waits stop. The controller sends each ACK only after its checkpoint commits, so shorter saves also shorten the event loop. ## What Changed - `server/src/services/heartbeat-run-events.ts`: lock the run row `FOR NO KEY UPDATE` instead of `FOR UPDATE` in the transactional append path. Add a comment that names the contract. - `heartbeat-run-event-atomic-append.test.ts`: add a regression test. An append transaction stays open. A checkpoint row insert for the same run must finish under a 2 s `lock_timeout`. A second appender must still wait. After release, the sequence is 1, 2 with no gap and no duplicate. The existing test with 40 concurrent appenders on two connections still proves contiguous sequence numbers. Update the existing lock-statement assertion to the new clause. - Add `scripts/benchmark-run-event-checkpoint-lock.ts`. It replays a native event stream with the real append and the real checkpoint writer on separate connections. It times each checkpoint save for a baseline append module and the current one. ## Verification - Red: with the old `FOR UPDATE` clause, the new test fails with `canceling statement due to lock timeout`. With the change it passes. - `npx vitest run src/__tests__/heartbeat-run-event-atomic-append.test.ts src/__tests__/heartbeat-run-event-sequencing.test.ts src/services/native-runtime/native-runner-checkpoint-authority.integration.test.ts src/__tests__/native-recovery-notice-work.test.ts` (server): 4 files, 42 tests passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm verify` exits 2, because a new script under `scripts/` is a global CI input. Required CI supplies the broad coverage. Local benchmark. Embedded PostgreSQL 18.1 on a laptop. 400 events per pass, 2 KiB event payload, 6 KiB header, real encryption. Order: warm-up, baseline, current, baseline, current. `HOLD_MS` keeps each append transaction open longer. It stands in for hosted round trips. | Hold | Pass | Median save | p90 | Max | Saves > 2 ms | Total save time | | --- | --- | --- | --- | --- | --- | --- | | 5 ms | Baseline (1) | 8.16 ms | 9.50 ms | 129.48 ms | 396 / 400 | 3,551 ms | | 5 ms | Current (2) | 1.44 ms | 2.08 ms | 12.66 ms | 44 / 400 | 677 ms | | 5 ms | Baseline (3) | 8.12 ms | 9.64 ms | 22.08 ms | 398 / 400 | 3,453 ms | | 5 ms | Current (4) | 1.50 ms | 2.37 ms | 10.54 ms | 80 / 400 | 700 ms | | 0 ms | Baseline (1) | 1.71 ms | 2.50 ms | 11.05 ms | 105 / 400 | 780 ms | | 0 ms | Current (2) | 0.97 ms | 1.42 ms | 10.28 ms | 22 / 400 | 458 ms | | 0 ms | Baseline (3) | 1.49 ms | 2.02 ms | 6.45 ms | 41 / 400 | 667 ms | | 0 ms | Current (4) | 0.91 ms | 1.24 ms | 9.80 ms | 19 / 400 | 420 ms | - Event order: the controller frame chain awaits each event's append and its local commit, but not its checkpoint save. The ACK goes through `#sendJson`, which queues the save and does not block the next frame. Only semantic inputs and command results await the save. So the save for event N overlaps the append of event N+1. Streaming events (`item.delta`) also go through the same locked append, with no checkpoint save, so a save can also overlap them. The benchmark uses this order. - Control without overlap (`SERIAL=1`, each save finishes before the next append, 5 ms hold): baseline median 1.15 / 1.07 ms, current 1.03 / 1.00 ms. There is no material difference. The gain comes only from the removed lock wait. - Results are the same in every pass. The script asserts that all event rows, the header and the cursor are stored, and that the run-log sequence is contiguous. - Command: see the script header. It needs a baseline copy of `heartbeat-run-events.ts` from the base revision. ## Risks - Low risk. The SQL change is one lock clause. Appenders still serialize with each other, with `FOR UPDATE` and `FOR SHARE` lockers, and with updates and deletes of the run row. Run deletion and key changes take `FOR UPDATE`, so they still wait for an append, and an append still waits for them. - The only new concurrency is with `FOR KEY SHARE` lockers. In this code base those are foreign key checks of child inserts. The one explicit `FOR KEY SHARE` read is on `issues`, not on `heartbeat_runs`. A foreign key check only needs the run row and its key to exist. The append does not delete the row or change its key. - The hosted cause is inferred. Neon shows calls and mean time only. We did not read lock waits on the hosted database. The local proof shows the mechanism and the size of the effect. A new query-performance window after deployment must confirm the result. - Persisted run state: no saved run, session, event or checkpoint shape changes. State saved by the previous release carries forward unchanged. The lock mode applies only inside one transaction, so a deploy during a run needs no special case. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, local PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
81c52e3666 |
perf(chat): find terminal GitHub responses by issue run indexes (#20671)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel posts a terminal response when an agent run for a GitHub request stops. A background sweep finds acknowledged response comments (`processed`, not final) whose issue is blocked or whose latest run is terminal > - For each candidate, the sweep reads the status of the latest run of the assigned agent for the issue. It matches runs with `coalesce(native_issue_id::text, context_snapshot->>'issueId')` > - No index has that expression. So PostgreSQL reads every run of the agent for each candidate, and it decompresses the large legacy `context_snapshot` of each run > - On a hosted test database, this statement ran 13 times at a mean of 11 s in a 38-minute window (149 s in total), with single runs of 16 s and 41 s > - This pull request splits the run match into its two disjoint branches. Each branch uses an existing issue index > - The benefit is that each candidate reads a few index entries instead of the agent's full run history. The returned rows do not change, and no migration is necessary ## Linked Issues or Issue Description **What existing behavior does this improve?** The discovery page query in `registerGitHubTerminalResponseWork` in `server/src/services/chat-channels/github/terminal-response-work.ts`. The query runs when the sweep reconstructs its candidates, and once or twice for each candidate check. **Subsystem affected** GitHub chat channel terminal-response recovery. **Current behavior** The `WHERE` clause has `issues.status = 'blocked' or (select r.status from heartbeat_runs r where r.company_id = ... and r.agent_id = ... and coalesce(r.native_issue_id::text, r.context_snapshot->>'issueId') = <issue id> order by r.created_at desc, r.id desc limit 1) in (<terminal statuses>)`. The planner can only use `heartbeat_runs_company_agent_started_idx`. It reads all runs of the agent, then evaluates the `coalesce` on each row. That decompresses the context of each legacy run. The cost is the number of candidates times the number of agent runs. **Proposed behavior** The subquery takes the latest run from two branches and then the latest of the two: - runs with `native_issue_id = <issue id>`, through `heartbeat_runs_company_native_issue_contract_id_uq` - runs with `native_issue_id is null and context_snapshot->>'issueId' = <issue id>`, through `heartbeat_runs_company_ctx_issue_created_idx` The two branches are disjoint, and together they are exactly the rows where the `coalesce` is equal to the issue ID. Both branches and the outer query use the same order (`created_at desc, id desc`). So the selected run is the same, including ties. **Reason and benefit** This statement was the largest single-statement cost in the hosted window after the earlier database fixes. A local reproduction is about 490x faster on the default fixture, and the gain grows with the agent's run history. **Breaking changes** None. The API, the candidate rules and the results do not change. ## What Changed - Rewrite the latest-run status subquery in the GitHub terminal-response discovery page as two index-backed branches (native binding, and legacy context without a native binding). - Add two integration tests for the sweep. In the first, newer runs of the same agent that do not bind the issue (one with a native binding to another issue, but a legacy context that names this issue) must not hide the stopped run. In the second, a newer active run with a native binding to the issue keeps the request waiting. - Add `scripts/benchmark-github-terminal-response-sweep.ts`. It captures the SQL that the real worker generates at a baseline commit and at `HEAD`, compares all rows, and measures both. ## Verification Cause and thesis. The cost comes from the `coalesce` run match, which has no index and makes each candidate scan the agent's run history. Thesis: if each precedence branch uses its own issue index, the cost of the page no longer depends on the agent's run count, and the rows stay identical. The benchmark confirms the thesis. Local SQL benchmark: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-terminal-response-sweep.ts --baseline-ref=7b33384446 ``` Fixture (synthetic): two companies, each with one GitHub endpoint and one assigned agent. Each issue has one GitHub response comment. The first N comments are acknowledged and not final; the others are final. Every tenth issue is blocked. Runs alternate between native binding and legacy context, and each run has a 4 KiB incompressible context. The fixture also has these edge cases: newer active runs with a native binding to another issue but a legacy context that names this issue, active runs that tie on `created_at` with the latest run (between native and legacy runs), and newer active runs of another agent. The benchmark compares every column of every candidate row (the page limit is lifted for this check). Embedded PostgreSQL 18.1 on an Apple laptop. The hosted database is Neon with different compute, so absolute times are local SQL measurements only. Each size has 2 warmups and then alternates the baseline and the treatment. | Size | Runs (both companies) | Candidate rows equal | Baseline median (range) | Treatment median (range) | Speedup | | --- | --- | --- | --- | --- | --- | | Small: 200 issues, 3 runs each, 20 candidates, 7 samples | 1,372 | 26 of 26 | 18.7 ms (17.7–20.8) | 3.8 ms (3.3–4.3) | 4.9x | | Default: 1,500 issues, 8 runs each, 150 candidates, 7 samples | 25,300 (156 MB) | 200 of 200 | 3,697 ms (2,875–4,032) | 7.6 ms (6.6–9.6) | 490x | | Large: 3,000 issues, 10 runs each, 400 candidates, 5 samples | 62,600 (382 MB) | 538 of 538 | 57,055 ms (48,343–66,079) | 31 ms (16–79) | 1,800x | Sizes are set with `BENCH_ISSUES`, `BENCH_RUNS_PER_ISSUE`, `BENCH_CANDIDATES` and `BENCH_ITERATIONS`. The timed query is the real first page (limit 100). Plan (default size): the baseline runs `SubPlan` 270 times. Each run is a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that removes about 12,000 rows by filter, for about 8.2 million buffer reads in total. The treatment runs an `Append` of two `Limit` index scans (`heartbeat_runs_company_native_issue_contract_id_uq` and `heartbeat_runs_company_ctx_issue_created_idx`), for about 3,800 buffer reads in total. Tests: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "terminal GitHub"`: 13 passed, including the 2 new tests. - Mutation check: when the legacy branch does not exclude runs with a native binding, the first new test fails. It passes with the fix. - `cd server && npx tsc --noEmit`: passed. ## Risks - Low risk. The change is one SQL subquery. The match is the same set of runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id = <uuid>`, and a null `native_issue_id` makes the `coalesce` use the legacy context. - The candidate set itself is not bounded by this change. Acknowledged responses that never become final stay in the candidate set. Each candidate is now cheap, but the count still grows with that history. - `readGitHubResponseRun` in `response-run-state.ts` uses the same `coalesce` match. The sweep calls it once for each candidate row, and other GitHub paths also call it. Each call is a separate, smaller statement. This pull request does not change it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
c14f2fe45d |
fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20645 lets a local same-run scheduled resume send the one same-conversation continuation for a turn lost on restore. It applies only when durable records prove three facts: the previous runner stopped, the attempt was admitted within the budget, and it is the same checkpointed conversation. > - It read the third proof from the coordinator's saved `failure_detail` while the phase was `retryable_failure`. > - In production, the resume reconciler (`claimNativeSessionResumptions`) claims the run first. It moves the coordinator to `observed` and clears `failure_detail`. So the executor never saw that evidence, and #20645 refused every real scheduled resume. > - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the exact path (`runner.drain` timeout, scheduled resume, `provider_turn_lost_on_restore`), sent no continuation, and needed Retry. > - This pull request makes the reconciler record that evidence durably when it claims the run, and makes the executor read it from there. > - The benefit is that #20645 works on the real path. The proof rule is unchanged. ## Linked Issues or Issue Description Refs #20645 **What happened?** Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus #20645), task codex B5 TM2: - `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`; - `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`, `recoverable`, `providerSessionEstablished` and `checkpointExists` all set; - `native.local_process_stopped`; - attempt 2 started; - `turn.failed` with `provider_turn_lost_on_restore` for the same turn ID that the checkpoint was running. No continuation was sent. The run failed with `native_provider_terminal_failed` and was preserved with `uncertain_provider_action`. **Expected behavior** The proven scheduled resume sends the single continuation. **Steps to reproduce** The new integration test below. The live path is timing-dependent (the L7 command in #20645). ## What Changed - `native-finalization-reconciler.ts`: when it claims a same-run resume, the reconciler appends one `same_run_resume` entry to `recovery_history`. The append is bounded like other history writes, and it runs in the same update that clears the failure. The entry copies `recoveryMode`, `recoverable`, `providerSessionEstablished`, `checkpointExists`, `successorRunId` and `replacementDenied` from the saved failure, with the controller boot and provider attempt. - `scheduled-resume-continuation.ts`: the evidence for the third proof now comes from either of two durable records: - the saved failure, when the executor claims a `retryable_failure` coordinator directly (unchanged); or - a reconciler claim: an `observed` coordinator whose last history entry is a `same_run_resume` entry naming the executor's own controller boot. The two other proofs are unchanged: the stop receipt, and an admitted attempt from 1 to 3. - `native-session-executor.ts`: passes the coordinator's last history entry and the controller boot to the decision. - `doc/execution-semantics.md`: describes the recorded evidence. Readers of `recovery_history` that look at the last entry do so only for restart claims, Stop recovery and shutdown suspension. Restart recovery's previous-claim lookup skips non-claim dispositions, so a `same_run_resume` entry does not change those paths. **Still refused** (tested): - every #20645 case; - the last history entry is not the resume claim, or there is no history; - the resume claim was recorded by another boot; - the claimed evidence is a bootstrap retry, has no checkpoint, or records a replacement; - a reconciler-claimed resume without a stop receipt. ## Verification - `native-scheduled-resume-continuation.integration.test.ts` now also runs through the real reconciler claim (`claimNativeSessionResumptions`) before `executePaperclipNativeSession`. On master the reconciler-claimed allowed case fails (`resumeInterruptedTurn` is false). With this change it is true, and the claim admits attempt 2. A reconciler-claimed resume without a stop receipt still passes false. - `scheduled-resume-continuation.test.ts`: decision cases for the reconciler-claim entry. - `native-session-resumption.test.ts` passes. 54 tests pass across the three files. - `cd server && npx tsc --noEmit` passes. - Live: not yet. The failing wave above is the reproduction. A rerun is timing-dependent. ## Risks - One more `recovery_history` entry per scheduled resume, inside the existing 20-entry bound. - The proof rule itself is unchanged. This only reads the same evidence from the record where it now survives. - No schema change. A run claimed by an earlier build has no `same_run_resume` entry, so it keeps the existing failure: refused, with Retry offered as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
48c6e996da |
fix(native): continue a task whose quarantined completed turn's holds settle (#20662)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a remote run starts, workspace preparation attaches the verified saved work of earlier runs of the same task. > - Preparation failed with `remote_workspace_recovery_receipt_invalid` when any earlier run had a recovery receipt that it could not verify. > - A refusal cannot change the saved state, so every Retry failed in the same way, and the task stayed blocked. Recoveries must fall back automatically. > - This pull request skips an unverifiable source. It attaches nothing unverified and starts from the workspace's newest verified save. > - The benefit is that Retry and continuations start again, while strict receipt checks still decide what is attached, and no saved work is changed. ## Linked Issues or Issue Description Refs #20631 **What happened?** On staging, after the deploy that included #20631, a user pressed Retry on a task whose previous run failed with `native_controller_recovery_missing_evidence:runner_not_settled`. The new run failed in setup in under a minute with `setup_failed` / `remote_workspace_recovery_receipt_invalid` (`executionRecovery: bootstrap`, `providerWorkStarted: false`). The task stayed blocked. Another Retry would fail the same way. #20631 accepted one shape of this refusal (a source that saved only the snapshot its own restore loaded). This task had a different shape. **Expected behavior** A Retry starts. A source whose receipt cannot be verified is not attached, and its saved state stays intact for workspace recovery. **Steps to reproduce** 1. Let a remote native run fail before provider work and leave a recovery receipt or restore hold that does not verify (for example a receipt on an unsaved reference, a receipt from another epoch, or a saved reference without a receipt). 2. Press Retry on the task. 3. See the new run fail in setup with `remote_workspace_recovery_receipt_invalid`. **Paperclip version or commit** Staging deploy that includes `e3bc4ca798`. The gap remains on `master` at `a3b34a9d88`. ## What Changed - `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an unverifiable source. Each of the five refusal shapes now skips the source and logs `remote_workspace_recovery_receipt_invalid` with a reason: `receipt_without_reference`, `reference_unreadable`, `receipt_on_unsaved_reference`, `receipt_unverified`, and `receipt_snapshot_missing` (was `remote_workspace_recovery_snapshot_missing`). - `retainRemoteRecoveryAttachments` no longer throws for a retained recovered copy that it cannot verify. It does not publish that copy again. The files stay in the workspace. - The writer check (`remote_workspace_recovery_writer_changed`), the workspace access check, and snapshot read and digest failures still refuse. - `doc/execution-semantics.md` describes the fallback rule. ## Verification - Red on unfixed code: `remote-workspace-recovery-receipt-fallback.test.ts` (moved out of `workspace-recovery-preparation.test.ts` to keep that file under 1,000 lines) "starts a Retry from the newest verified save when a predecessor has %s" (6 shapes: `receipt_without_reference`, `unreadable_reference_with_hold`, `receipt_on_prepared_reference`, `superseded_snapshot_with_hold`, `receipt_from_other_epoch`, `receipt_for_other_run`). All 6 failed with `remote_workspace_recovery_receipt_invalid`. The source is a native run that failed with `runner_not_settled`, and the writer is a Retry (`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry query. - Green: the same 6 tests pass. They also assert that nothing is attached, and that the source run row, its workspace operations, and its snapshot row are unchanged. - Updated safety tests: the canonical-attachment fault matrix now expects no attachment for receipt faults, and still expects a refusal for a missing or corrupt archive and for a wrong company or task. The retained-history test expects an unverifiable retained copy to be left out of the published list, while verified copies stay published. - `npx vitest run src/services/remote-workspace-recovery-receipt-fallback.test.ts src/services/workspace-recovery-preparation.test.ts`: 91 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Not verified live on Daytona. I could not read the staging rows for this task: the browser had no free tab. The tests cover every refusal shape of this function, so the exact staging shape is one of them. ## Risks - Previous-release state: the rule reads the same rows as before (run reference, result hold, workspace operations, recovery actions, snapshots). Rows that earlier releases saved carry forward. No migration. A blocked task recovers on its next Retry or wake. - A Retry can now start without files from an unverifiable source. Those files are not lost: the source's snapshot, restore hold, receipts and recovery action stay unchanged, and workspace recovery still owns them. The skip is logged with a named reason. - A retained copy that fails verification is dropped from the published list. Its directory stays in the workspace and is no longer retired automatically. - A missing or corrupt snapshot archive still refuses. A permanently lost archive therefore still blocks attach. This PR does not change that. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8f6ca25976 |
fix(workspaces): skip unverifiable recovery sources instead of failing Retry (#20666)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a remote run starts, workspace preparation attaches the verified saved work of earlier runs of the same task. > - Preparation failed with `remote_workspace_recovery_receipt_invalid` when any earlier run had a recovery receipt that it could not verify. > - A refusal cannot change the saved state, so every Retry failed in the same way, and the task stayed blocked. Recoveries must fall back automatically. > - This pull request skips an unverifiable source. It attaches nothing unverified and starts from the workspace's newest verified save. > - The benefit is that Retry and continuations start again, while strict receipt checks still decide what is attached, and no saved work is changed. ## Linked Issues or Issue Description Refs #20631 **What happened?** On staging, after the deploy that included #20631, a user pressed Retry on a task whose previous run failed with `native_controller_recovery_missing_evidence:runner_not_settled`. The new run failed in setup in under a minute with `setup_failed` / `remote_workspace_recovery_receipt_invalid` (`executionRecovery: bootstrap`, `providerWorkStarted: false`). The task stayed blocked. Another Retry would fail the same way. #20631 accepted one shape of this refusal (a source that saved only the snapshot its own restore loaded). This task had a different shape. **Expected behavior** A Retry starts. A source whose receipt cannot be verified is not attached, and its saved state stays intact for workspace recovery. **Steps to reproduce** 1. Let a remote native run fail before provider work and leave a recovery receipt or restore hold that does not verify (for example a receipt on an unsaved reference, a receipt from another epoch, or a saved reference without a receipt). 2. Press Retry on the task. 3. See the new run fail in setup with `remote_workspace_recovery_receipt_invalid`. **Paperclip version or commit** Staging deploy that includes `e3bc4ca798`. The gap remains on `master` at `a3b34a9d88`. ## What Changed - `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an unverifiable source. Each of the five refusal shapes now skips the source and logs `remote_workspace_recovery_receipt_invalid` with a reason: `receipt_without_reference`, `reference_unreadable`, `receipt_on_unsaved_reference`, `receipt_unverified`, and `receipt_snapshot_missing` (was `remote_workspace_recovery_snapshot_missing`). - `retainRemoteRecoveryAttachments` no longer throws for a retained recovered copy that it cannot verify. It does not publish that copy again. The files stay in the workspace. - The writer check (`remote_workspace_recovery_writer_changed`), the workspace access check, and snapshot read and digest failures still refuse. - `doc/execution-semantics.md` describes the fallback rule. ## Verification - Red on unfixed code: `remote-workspace-recovery-receipt-fallback.test.ts` (moved out of `workspace-recovery-preparation.test.ts` to keep that file under 1,000 lines) "starts a Retry from the newest verified save when a predecessor has %s" (6 shapes: `receipt_without_reference`, `unreadable_reference_with_hold`, `receipt_on_prepared_reference`, `superseded_snapshot_with_hold`, `receipt_from_other_epoch`, `receipt_for_other_run`). All 6 failed with `remote_workspace_recovery_receipt_invalid`. The source is a native run that failed with `runner_not_settled`, and the writer is a Retry (`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry query. - Green: the same 6 tests pass. They also assert that nothing is attached, and that the source run row, its workspace operations, and its snapshot row are unchanged. - Updated safety tests: the canonical-attachment fault matrix now expects no attachment for receipt faults, and still expects a refusal for a missing or corrupt archive and for a wrong company or task. The retained-history test expects an unverifiable retained copy to be left out of the published list, while verified copies stay published. - `npx vitest run src/services/remote-workspace-recovery-receipt-fallback.test.ts src/services/workspace-recovery-preparation.test.ts`: 91 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Not verified live on Daytona. I could not read the staging rows for this task: the browser had no free tab. The tests cover every refusal shape of this function, so the exact staging shape is one of them. ## Risks - Previous-release state: the rule reads the same rows as before (run reference, result hold, workspace operations, recovery actions, snapshots). Rows that earlier releases saved carry forward. No migration. A blocked task recovers on its next Retry or wake. - A Retry can now start without files from an unverifiable source. Those files are not lost: the source's snapshot, restore hold, receipts and recovery action stay unchanged, and workspace recovery still owns them. The skip is logged with a named reason. - A retained copy that fails verification is dropped from the published list. Its directory stays in the workspace and is no longer retired automatically. - A missing or corrupt snapshot archive still refuses. A permanently lost archive therefore still blocks attach. This PR does not change that. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
9296c74e8d |
fix(native): let a restart claim take back its lapsed lease at budget admission (#20650)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart, restart recovery claims an interrupted native run (`resume_dead_runner`) and grants a 20-minute lease. Budget admission then re-checks that the claim still owns the run before the resumed run may start provider work. > - A host sleep, or the restart-chaos fault TM2, can make that stored lease look expired while its owner is alive. #20625 lets the owner take back such a lease when the row still shows exactly its own claim. That rule did not cover budget admission. > - So a claim whose lease lapsed before dispatch was refused with `Native recovery no longer owns provider dispatch`. The run failed with `adapter_failed`, recovery held it with `native_continuation_requires_reconciliation`, and the user had to click Retry. > - This pull request applies the #20625 rule in budget admission: the claim takes its lease back through the existing re-acquire condition. > - The benefit is that a run resumed after a restart survives a time jump before dispatch with no Retry. Every fence and every budget check stays the same. ## Linked Issues or Issue Description Refs #20625, #20474 **What happened?** Restart-chaos wave L7 (seed 9107, local) on a branch with #20625, task codex B5 TM2. A TM3 restart claimed the run (`resume_dead_runner`, generation 2). TM2 then aged every database time by one hour before the resume reached dispatch. Budget admission (`lockRunBudgetRecoveryOwnerInTransaction`) required a live lease and threw `Native recovery no longer owns provider dispatch`. The run failed with `adapter_failed`, recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`, and the task needed Retry. **Expected behavior** The claim's own lapsed lease is taken back, and the resume is admitted when the budget allows it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` (timing-dependent). ## What Changed - `native-execution-lease.ts`: two new helpers. Both use the #20625 own-claim rule. Neither takes a lock out of order. - `readOwnNativeRecoveryClaimLease` (read-only) returns `live`, `lapsed_own_claim` or `lost`. It compares the lease against the database clock. A lapsed lease counts as `lapsed_own_claim` only when all of these hold: - the same lease owner; - this process's controller boot, pid and process start; - a non-null `recovery_state`; - `nativeExecutionLeaseHeldCondition({ remote: false, controllerGeneration })`, which applies the #20625 own-claim rule and every #20474 untouched check; - no non-local environment lease on the run. - `extendOwnNativeRecoveryClaimInTransaction` extends such a lease by the local execution TTL, with one conditional UPDATE on the same predicate. - `budget-reservations.ts`: - `lockRunBudgetRecoveryOwnerInTransaction` still refuses another owner, and refuses a lapsed lease unless it is the run's own untouched claim. It no longer locks the run row, so the documented coordinator → issue → run order holds. - The lease is extended only at the end of an admitted reservation (`reserveRunBudget`) or an admitted dispatch recheck (`recheckRunBudgetReservationInTransaction`). At those points the caller already holds the run row lock. A refused admission, for example one over budget, leaves the lease lapsed. - `heartbeat-native-restart-suspension.test.ts`: the test "still settles a recovery whose claim merely expired" encoded "expired means not the owner". #20474 and #20625 changed that rule for an untouched own claim. The test now has two cases: - an expired claim that another actor recorded on (`awaiting_evidence`) still settles with no provider session; - an untouched expired claim is dispatched. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new user of the rule. ## Verification - New `server/src/__tests__/native-recovery-dispatch-lease-reacquire.test.ts`: - `reserveRunBudget` takes back the claim's own lapsed lease and admits the resume. This fails before the change. - An over-budget resume is still refused, and its lease stays lapsed. - Fenced variants each throw `Native recovery no longer owns provider dispatch` and leave the row unchanged: a graceful-shutdown suspension, an appended failure settlement, `awaiting_evidence`, `blocked`, a claim from an earlier generation, another boot, a remote sandbox lease, and another lease owner. - At the dispatch gate (`recheckRunBudgetReservationInTransaction`), an admitted own claim gets its lease extended, and a fenced claim is refused with the row unchanged. - Related suites pass (308 tests): `native-execution-lease-reacquire.test.ts`, `closed-native-receipt-accounting.test.ts`, `heartbeat-native-predispatch-retry.test.ts`, `run-dispatch/adapters/postgres.test.ts`, `heartbeat-native-restart-suspension.test.ts` and `heartbeat-restart-claim-lapse.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: one L7 wave (seed 9107) ran on the first commit of this branch (`w-20261110T212307-s9107`). It logged 0 `Native recovery no longer owns provider dispatch` refusals, against 1 in the previous wave. Codex B5 TM2 got past dispatch, but it still needed Retry for another reason: after a `runner.drain` timeout, the scheduled same-run resume did not send the continuation of #20645. That case is tracked separately. The other five tasks passed. ## Risks - This widens when a restart claim can take back its own lapsed lease, and only at budget admission. It is local only. It needs the same owner, boot and generation, and the row must show exactly that claim. A claim that any other actor touched still fails as before. - Budget hard-stop behaviour is unchanged. The lease is extended only after budget admission passes. - No schema change and no migration. Rows saved by earlier releases carry the same history shape. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
de7ceac0de |
fix: preserve compatible historical Codex WEB evidence (#20665)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native session migration preserves local evidence in encrypted archives. > - Each WEB receipt binds the exact provider directory and exclusion policy. > - The Codex credential refresh change added two excluded files and changed that policy hash. > - A new reader then rejected old receipts even when their facts, files, skills, and authority still matched. > - This pull request accepts the exact prior Codex policy while checking archive paths against the current credential policy. > - Operators can keep valid saved evidence across the upgrade without copying it again. ## Linked Issues or Issue Description Refs: #19814, #20646. **What happened?** A reader upgrade rejected saved Codex WEB evidence because the profile hash changed when `auth-refresh.json` and `auth-refresh.json.tmp` became excluded credentials. The other receipt bindings still matched. This blocked the native harness migration gate. **Expected behavior** The exact known prior profile remains readable when the current archive path checks accept every saved entry. An archive that includes either newly excluded credential file must still fail. Unknown profiles and changed authority must fail. **Steps to reproduce** 1. Preserve WEB evidence with the frozen prior Codex profile and no credential refresh files. 2. Read its unchanged receipt with the current profile. 3. Observe that the old reader returns false solely because the profile hash changed. **Paperclip version or commit** Reproduced against `a3b34a9d88`; final branch is based on `e9343d8501`. ## What Changed - Accept the exact prior Codex profile hash only when the complete current profile has the reviewed successor hash and the provider is Codex. - Keep the current entry and excluded-path checks. Keep facts, controller, archive, skill, and ownership checks. - Add a frozen prior policy fixture with its literal hash. Test old receipt reads, both credential files, unknown policies, changed authority, and current writes. ## Verification - Real PostgreSQL baseline: the frozen old-policy receipt test failed because the reader returned false. The unchanged fixture passes with this fix. - All 18 selected PostgreSQL cases pass on final head after the master rebase: six new cases and twelve existing WEB archive and authority guards (14.93 seconds). - Full layered server typecheck passes on final head after the rebase. The module boundary check passes. - A single read-only native-only candidate check on the deployed dependency graph validated all 54 retained scopes. The 29 prior failures passed, and the other 25 stayed valid. The check made no storage calls or data writes. It did not replace final deployed enrollment acceptance. - Final head `14112d0e3abb6e318a86fe9799d95bddd2dbf76b` passes all [required CI checks](https://github.com/paperclipai/paperclip/actions/runs/42919867108): 55 successful checks, four expected skips, and one neutral review. The legacy security status also passes. Exact-head Greptile review is 5/5 with no open review threads. ## Risks The compatibility branch accepts only the two reviewed policy hashes. A future policy change can require another explicit compatibility decision. The reader still rejects saved credential refresh files. It does not rewrite receipts or change execution authority. No stored schema or operator command changes. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window capacity are not exposed in this session. Used reasoning, source editing, shell execution, and tool calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aa2a7be8b4 |
fix: recognize provider-owned Runner workspace storage (#20663)
## Thinking Path > - Paperclip manages agent work and preserves its execution state. > - A server replacement must retain every workspace that depends on the old server disk. > - Runner task environments can own their workspace outside the Core server. > - These environments save an adapter-managed workspace with a runnerTaskWorkspaceOwner binding. > - Readiness and maintenance treated that provider path as a Core filesystem dependency. > - This change validates the saved owner and shares that classification between both paths. > - Local, malformed, and foreign workspace bindings still block enrollment. ## Linked Issues or Issue Description Related enrollment safeguards: #18835. A GitHub search found no duplicate fix for this provider-owned class. **What happened?** Volume-free readiness returned workspaces_require_migration for an adapter-managed Runner workspace owned by a plugin task environment. Maintenance also selected it and failed with workspace_provider_unsupported. The provider owned the path, which did not exist on the Core server. **Expected behavior** A valid provider-owned workspace does not require migration from the Core server disk. Ordinary local workspaces and invalid ownership bindings remain blockers. **Steps to reproduce** 1. Run a paperclip_runner agent in a plugin task environment. 2. Retain its adapter-managed workspace and complete runnerTaskWorkspaceOwner binding. 3. Read startup eligibility for volume-free enrollment. 4. Observe the workspace blocker before this fix. 5. Run workspace maintenance and observe workspace_provider_unsupported for the same row. **Paperclip version or commit** Reproduced on |
||
|
Nicky Leach
|
0d4b1d8c89 | Gate agent plugin tabs by execution environment (#20657) | ||
|
Devin FoleyandPaperclip
|
e9343d8501 |
fix(heartbeat): release task Retries held by a drain as soon as it ends (#20640)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person clicks Retry on a stopped task, Paperclip saves the
request as a held wake and dispatches it right away, unless admission is
held. Admission is held during a task drain, for example while a
deployment restarts the server
> - A Retry held by a drain is released later by the deferred-receipt
worker. That worker shares one queue with every other held wake: comment
waits, execution waits, and interrupted queues
> - On a busy instance that queue holds hundreds of passive waits that
come due every 30 seconds. A Retry is checked in due-time order behind
all of them. Ending a drain does not wake the worker either
> - In a real case, a Retry clicked during a deployment drain was still
held 40 minutes later, after the drain expired and the server restarted.
It ran only when the person clicked Retry again
> - This pull request checks operator Retries ahead of passive waits,
and wakes the worker as soon as a drain stops or expires
> - The benefit is that a Retry held by a drain starts within one worker
pass after the drain ends
## Linked Issues or Issue Description
**What happened?**
A person clicked Retry on a blocked task 14 seconds after a deployment's
task drain started. The Retry was saved as `deferred_issue_execution`,
and the immediate dispatch returned early because of the drain. The
drain had a 20-minute TTL, and the deployment restarted the server about
6 minutes in. The held Retry still did not start, and its `updated_at`
never changed. Forty minutes later the person clicked Retry again, and
the new request started a run and merged with the held one.
That instance had 390 held wakes: 271 execution waits and 116 others,
against 6 task Retries. The execution waits come due every 30 seconds.
Measured on the instance, the deferred-receipt worker took about 7
minutes to get through that backlog once.
**Expected behavior**
A Retry that a drain held starts soon after the drain stops or expires,
or after the server restarts, whatever the size of the passive backlog.
**Steps to reproduce**
1. Start a non-idle task drain (`POST /instance/task-drain`).
2. Click Retry on a stopped task. The receipt is
`deferred_issue_execution`.
3. Stop the drain. On `master`, the deferred-receipt worker only notices
on its next 30-second re-check. It then checks the Retry in due-time
order behind every passive wait that came due before it.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
686893636d |
fix(recovery): release a lapsed owner of a finished coordinator so stop-proof holds re-arm (#20661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Workspace recovery saves the files of a run whose copy-back failed, and it holds the shared workspace until the save is done > - #20295 re-arms a save that was exhausted with `agent_stop_unconfirmed`, so the run's stop is proven again and the files are saved without a person > - On staging, four holds of this shape were never re-armed. Their runs ended six days ago, their sandbox leases are released with a provider stop receipt, and the servers that ran them were replaced many times. Their tasks stay blocked, and so do the parent tasks that wait for them > - The finished coordinator of each run still names a lease owner whose lease lapsed. Discovery treats that owner as "waiting for the abandoned-owner settlement", but that settlement handles only observed (unfinished) coordinators. Nothing ever releases the owner, so the worker never selects the hold, the re-arm refuses it, and the attempt's stop check refuses it too > - This pull request makes a lapsed owner of a finished coordinator count as stopped. The re-arm releases it in the same transaction, and then the new attempt proves the sandbox stop and saves the files as #20295 designed > - The benefit is that these holds settle on the worker's next pass after deploy, without a migration and without an administrator ## Linked Issues or Issue Description Refs #20295 **What happened?** An exhausted workspace save (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`) stayed blocked for six days. The task showed "Paperclip tries again automatically", but no re-arm was ever recorded (no `stopProofRecovery` marker, no `issue.workspace_recovery_rearmed` activity), and the shared workspace stayed held by the failed run. **What did you expect to happen?** The recovery worker re-arms the save once the run is provably over, saves the dirty files, and frees the workspace so the task and its parent can continue. **Steps to reproduce** 1. Exhaust a workspace save with `agent_stop_unconfirmed` (one attempt, window spent). 2. Leave a lease owner on the run's `terminal_failure` coordinator with an expired lease (or no expiry), as a finalizer pass of a replaced server does. 3. Run the workspace recovery worker. `nextWorkspaceRecoveryAt` returns null, discovery does not select the hold, and `rearmExhaustedStopProof` refuses it. ## What Changed - `workspace-recovery-stop-proof.ts`: add `lapsedFinishedCoordinatorOwner` and `releaseLapsedCoordinatorOwner`. A lease owner on a finished coordinator (committed, terminal failure, applied) whose lease lapsed, or that has no expiry, does not block `sourceRunProvablyOver`. The stop-proof re-arm clears that owner with a compare-and-swap in its locked transaction, and records `releasedLapsedOwner` in the marker and the activity. - `workspace-recovery-capture-policy.ts`: the capture-policy re-arm uses the same rule, because it shares the same check and discovery clause. - `workspace-recovery-fallback-discovery.ts`: the discovery clause still waits for the abandoned-owner settlement for a stale owner of an unfinished coordinator, but it no longer excludes a hold whose finished coordinator has a lapsed owner. A live lease still defers the re-arm to its expiry. - `system-recovery-state.ts`: add `FINISHED_COORDINATOR_PHASES`. - `doc/execution-semantics.md`: document the rule. - Tests: two red-then-green regressions in the staging shape (an expired finalizer lease, and an owner without an expiry saved by an older release), one strictness case (a live lease on a finished coordinator keeps the hold and is not released), and the existing stale-owner case now uses an observed coordinator, which still waits for the abandoned-owner settlement. ## Verification - Red before the fix: `pnpm --filter @paperclipai/server exec vitest run src/services/workspace-recovery-stop-proof.test.ts` failed the two new staging-shape cases (`nextWorkspaceRecoveryAt` returned null: the worker was never woken for the hold). - Green after the fix: `workspace-recovery-stop-proof.test.ts` and `workspace-recovery-capture-policy.test.ts` pass (15 tests). The staging-shape cases assert that the hold is re-armed, the save completes, the dirty files are saved, the workspace hold is free, and the lapsed owner is released. - `pnpm verify`: exit 2 (broader verification required, because `doc/execution-semantics.md` is a global CI input). Focused suites pass: `workspace-recovery-stop-proof`, `workspace-recovery-capture-policy`, `workspace-recovery-fallback`, `workspace-recovery-retry-copyback` (113 tests). Required CI supplies the broad coverage. - Staging evidence (read-only GET requests): the run is `failed`, finished six days ago, `controllerLeaseExpiresAt` is null, its only lease is `released` with `cleanupStatus: success`, a `stopped` termination receipt and a stop-and-retain receipt. The action's evidence has no re-arm marker, and four holds of the same shape were not touched after #20295 was deployed. ## Risks - Previous-release story: rows saved by earlier releases carry the lapsed owner and no marker. They are selected and re-armed on the worker's first pass after deploy. No migration. - Strictness stays where an owner could be live: a live lease on any coordinator defers the re-arm to its expiry, and a stale owner of an unfinished (observed) coordinator still waits for the abandoned-owner settlement, which proves its controller stopped. The new attempt still requires a provider stop receipt for the allocation before it reads files. - Clearing a lapsed owner matches the existing token-lease contract: an expired finalizer or cleanup lease is free for the next claim, and a holder's later write fails its token compare-and-swap. - Low risk otherwise: the change is limited to the two exhausted-hold re-arms and their discovery clause. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, browser GET reads) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
a3b34a9d88 |
fix(ui): allow queued message deletion before queue acknowledgement (#20553)
## Thinking Path Paperclip lets a user queue a message while an agent works. The user must be able to remove that message before delivery starts. A new local message can clear the displayed queue ID until the server accepts it. This also disables trash for older saved messages. The page rejects the action before it sends a request. This change keeps trash available when the server says the user can discard the message. When the queue ID is absent, the page uses the existing comment cancellation route. The server finds the queue and checks the author and dispatch state in the same transaction. ## Linked Issues or Issue Description **What happened?** Trash stays disabled for a saved queued message while another message waits for queue acknowledgement. The page handler also throws before it sends a cancellation request. **Expected behavior** Trash removes the selected queued message before dispatch. A message that has started dispatch stays visible with an error. Other queued messages stay queued. **Steps to reproduce** 1. Send a follow-up while an agent works. 2. Send another follow-up before the queue read includes it. 3. Try to discard the first saved message. **Paperclip version or commit** Reproduced from source at `0e3fdebea825`. **Deployment mode** Source checkout on Linux. UI tests use jsdom. Route tests use isolated embedded PostgreSQL. Related work: #15865 changes mixed-queue interrupt handling. #15939 changes recovery after a queued message is discarded. #15465 changes discard permissions for agent-authored entries. This change fixes trash for a user's own saved messages while queue acknowledgement is pending. ## What Changed - Keep trash enabled for a saved message with discard permission when the displayed queue ID is absent. - Use the existing author-only cancellation route when queue identity is pending. Keep the revision-checked route when queue identity is present. - Keep failed cancellations visible. Refresh the queue after cancellation and remove successful deletions from comment caches. - Cover deletion, dispatch conflicts, permission failures, network failures, history reloads, final-message cleanup, and concurrent dispatch. ## Verification - Red: three regression cases failed on the original production code. The trash button was disabled. Both page cases made no cancellation request. - Green: five complete UI suites pass, with 767 tests and one existing skipped test. These cover the page, queue component, thread, queue merge, and optimistic comments. - Green: 11 focused PostgreSQL route cases pass. They cover both routes, queue promotion, final-message cleanup, authorship, dispatch conflicts, and a concurrent dispatch claim. - `pnpm check:token-gates` and `git diff --check` pass. - `pnpm --filter @paperclipai/ui typecheck` passes. - The GitHub tree is `d4f0d61cbef2620c1fd62e6988ad80ccb1ba6173`. It is identical to the tested local tree after merging master `902f3767960d`. - `pnpm verify` selected more than 1,500 server test files. That broad local run was stopped without a pass claim. CI will supply broad build and test coverage. The preflight helper could not build Rust because Cargo is absent. - [CI Build](https://github.com/paperclipai/paperclip/actions/runs/42835584447/job/129897150252) and [browser E2E](https://github.com/paperclipai/paperclip/actions/runs/42835584447/job/129901676175) pass for head `44dea22faea51a6664a6f45852a42f62f5c72bbd`. - [Greptile review](https://github.com/paperclipai/paperclip/pull/20553#issuecomment-6770202777) gives the current head 5/5 with zero comments and no unresolved review threads. - [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42835584447) passes for head `44dea22faea51a6664a6f45852a42f62f5c72bbd`: 55 check runs passed and 4 were skipped. No failures or pending checks remain. Typecheck, Build, all general test shards, serialized suites, Runner checks, and browser E2E passed. Snyk also passed. ## Risks The fallback has no client queue revision. The existing server transaction finds the queue for the selected comment and checks ownership and dispatch state. Tests cover rejection after dispatch and when another user owns the message. No API contract or schema changes are required. No visual tokens change. Existing user documentation remains accurate. ## Model Used OpenAI GPT-6 in Codex. The runtime does not expose a more specific provider revision or context window size. Used reasoning, repository tools, code edits, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
f30ac226d6 |
Keep project environment mappings independent of shared defaults (#20651)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A project supplies work context and repository storage to task runs. > - Creation selected a storage owner but did not retain the selected environment on the project. > - Configuration stored an environment inside checkout policy, but policy reads and run selection dropped it. > - Projects therefore continued to follow shared agent and instance defaults. > - This pull request saves placement on each project and uses it for new runs. > - The benefit is an independent Project to Environment association with the existing storage, trust, and recovery safeguards. ## Linked Issues or Issue Description Refs #20459 and #11644. Follow-up to #20587. The reported journey is Project > Configuration: changing one project's environment must not change another project's mapping or shared defaults. Related open PR #18176 repairs placement through the legacy workspace-policy field. This follow-up introduces a separate project association, saves creation defaults, and migrates saved legacy references. It overlaps that PR's placement, permission, and recovery safeguards. The changes should be reviewed together. #9588 concerns environment-variable response privacy and does not address placement. ROADMAP.md has no separate project-association work planned. ## What Changed - Add nullable `projects.environment_id` and expose `Project.environmentId`. Save the effective creation default once on the project. Keep environment placement separate from checkout/isolation policy. - Read and write Environment directly in Project Configuration. Keep the control available with isolated workspaces disabled. Each form reads its project's value, even when environment choices share a query cache. - Offer environments compatible with the current storage owner. Change the target and storage together when the existing storage selector attaches a core project to a plugin. Preserve retained-data restrictions and validate the observed storage owner again on save. - Resolve new task runs from their organizational project's target before agent and instance defaults. File sources from another project do not change placement. Explicit low-trust sandbox choices and enforced sandbox policies retain precedence. External provider placement remains separate. - Restore admitted native runs from their owned environment lease. Later project edits apply to new admissions. Require workspace-write permission for environment changes and clears. Check company bindings before runtime admission and during lease creation. - Migrate valid saved policy references into the project column. Normalize legacy API writes. Count and clear matching project references when an environment is deleted. Update API and behavior documentation. - Refresh the maintained SDK source registry for the two changed shared files. Runtime pins are unchanged. ## Verification - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/42908045146) is green on `8b3b7d5b5aed51fc4c8e9c2e3506508bbaf9e00c`: **60 successful checks**, three skipped gates, no failures, and `ci / verify` passed. GitHub reports no merge conflicts. Standard Greptile scored the same commit **5/5** with both review threads resolved. - Source checkout on macOS arm64, Node 26.12.0, with fresh isolated PostgreSQL. No production data or live provider calls. - The two-project component test changes the first form with isolation disabled and a shared query cache. The second form keeps its target. Storage changes update only the selected project. A saved target remains selected while the list loads, and an explicit clear remains possible. - API integration creates two projects from the same default, patches one, reads both, and checks that agent and instance defaults remain unchanged. A later instance-default change does not rewrite either saved project target. Tests cover unavailable/incompatible choices, anonymous and cross-company denials, and storage preconditions. - The migration test runs the actual SQL against an existing project row and verifies its environment and disabled checkout policy survive. Deletion accounting and clearing affect only matching project references. - The final guard set passes **53 tests across four files**: `project-environments.test.ts`, `environment-selection-route-guards.test.ts`, `heartbeat/environment-selection.test.ts`, and `ProjectProperties.environment.test.tsx`. All **64 environment-run-orchestrator tests** pass, including direct lease acquisition without an extra caller argument. The broader affected set passed **109 tests across eight files** before the additional admission guards. These sets overlap. - UI typecheck, migration numbering/safety, token gates, module boundaries, query-error rendering, and diff checks pass. `pnpm verify --explain` returns exit 2 and requires broad CI because shared schema/runtime paths and behavior documentation changed. Broad coverage is delegated to current-head CI. - Early iterations failed due to a legacy parser that drops the environment field, a fixture without real company membership, and incorrect expectations for privacy-preserving 404 denials. The service now reads legacy references before normalization, the API fixture has real membership, and denial assertions match the existing privacy contract. The orchestrator mock also needed the reused company-binding guard in its service contract. The focused suites above pass after those corrections. - Local server typecheck remains blocked by installed ACPX, Slack, AI SDK, and Runner type mismatches in unchanged dependency-facing code. It reports no errors in the changed production paths. This is not a full local typecheck or suite pass. Hosted CI supplies full current-head coverage. - SDK source-proof and privacy qualification passes **56 tests across three files**, including the real built SDK. The maintained source registry matches the committed revision; runtime pins are unchanged. - Review reproduced two failures: an incoming policy without a storage owner could be validated before the existing owner was restored, and uppercase legacy UUIDs missed migration matching. Preserve the owner before validation and compare normalized UUID text. Both regressions now pass. The post-fix project/storage/orchestrator batch passes **107 tests**: `node node_modules/vitest/vitest.mjs run server/src/__tests__/project-environments.test.ts server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/environment-run-orchestrator.test.ts --silent`. - The first hosted run failed build, typecheck, and canary at the same `run-completion.test.ts` call because lease acquisition required a new argument. The lease method now enforces binding internally. Its existing caller compiles without a new field, and the orchestrator tests prove the binding flag reaches the runtime. The full server check no longer reports that error; the local dependency errors above remain. ## Risks - Apply the normal database migration before serving the new revision. The migration adds a nullable foreign key and copies valid legacy project selections. It does not move repository data or resize provider storage. The five newest schema snapshots are retained. - New projects retain their creation target when shared defaults change. Existing projects without a saved selection continue to inherit defaults. A null target explicitly opts into inheritance. - Existing saved project selections now affect placement. Missing, archived, incompatible, and foreign-company targets fail validation; they do not silently choose another storage owner. A running admitted process keeps its original owned lease. - Local dependency limitations are listed above. No live-provider, deployed-browser, or production rollout qualification is claimed. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
358977454b |
test(native): cover unknown-outcome recovery and hold settlement without a recorded PID space (#20649)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #20628 settles a host-tool callback that a server restart orphaned,
and settles the no-replay holds that earlier releases left for a
completed turn
> - Review of #20628 asked for two more cases. That PR merged before the
commit with those cases reached it
> - This pull request adds only those tests
> - The benefit is that the two edge rules of #20628 stay covered
## Linked Issues or Issue Description
Refs #20628
**What happened?**
The review of #20628 named two untested rules. First, a recovery
authority that throws `semantic_tool_outcome_unknown` must settle only
for a callback from an earlier controller, and must rethrow for this
controller's own callback. Second, a coordinator row saved before
`controller_host_hash` existed must still settle its holds through the
stopped controller boot.
**Expected behavior**
Both rules have regression tests.
**Steps to reproduce**
1. Read `recoverToolCallback` and `native_completed_turn_cleanup_hold`
on master.
2. Search the tests for a thrown `semantic_tool_outcome_unknown`, or for
a row with a null `controllerHostHash`. Neither exists.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude
|
30403781b9 |
fix(native): keep crediting proven unstarted restarts past the generation limit (#20647)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run survives a control-plane restart through restart recovery, which resumes the interrupted provider turn > - Restart recovery charges a provider attempt only when the provider did work that could be lost, and it stops after three charged attempts > - From controller generation 12 on, every restart charged an attempt before that work rule was checked > - In a busy restart wave, one long task sees 12 to 15 restarts, most caused by other tasks. A restart that hit before the provider accepted a turn was charged, the budget ran out, and a person had to click Retry > - This pull request keeps the work rule past generation 12 for restarts that a durable record proves did not start the provider, and bounds a crash loop with a count of consecutive restarts without provider progress > - The benefit is that a long task survives many unrelated restarts without a manual Retry, while a run that never makes progress still stops ## Linked Issues or Issue Description Refs #20625 (same restart recovery path). No public issue exists for this case. **What happened?** `restartResumeKeepsAttempt` returned "charge" for every restart once the run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller generations. It did this before it read the work-based outcome. In a local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1) the run reached generation 13. The last restart interrupted a resume whose runner started but whose provider had not accepted a turn. That restart was charged at the last attempt, so the claim failed with `execution_recovery_budget_exhausted` and `native_restart_recovery_blocked`. The task needed a manual Retry. **Expected behavior** A restart that a durable record proves did not start provider work does not spend an attempt, also past generation 12. A crash loop that never makes provider progress still stops automatically. When it stops, the task shows a Retry that works. **Steps to reproduce** Run a native task past controller generation 12. Make the last restart hit after `native.process_start_requested` but before `turn.accepted`, with the provider attempt at 3. The new integration test `does not charge a restart past the generation limit whose resume never started the provider` reproduces the failure before the fix. **Paperclip version or commit** `master` at |
||
|
DottaandClaude Opus 5.5
|
4f33bb3578 |
fix(workspaces): release the Blocked status of a sibling-settled no-replay hold (#20648)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a remote workspace save fails, recovery actions hold the task until the run's files are saved > - #20641 lets a resolved no-replay hold settle when a sibling action of the same run saved the files > - Review found that the settlement did not release the task's Blocked status, and that a settled hold with a historical source stayed selectable > - A new test with the waiting user comment also showed that the worker skipped the second hold of the same run in every pass, so it never settled > - #20641 was merged before these fixes were pushed > - This pull request completes #20641 with those fixes > - The benefit is that the task leaves Blocked and continues without a person ## Linked Issues or Issue Description Refs #20641, Refs #20569 **What happened?** #20641 merged at its first head. The review follow-ups for it were pushed later and are not on master: - A settled no-replay hold that owned the task's Blocked status (`nativeFailureBlock`) left the task `blocked`. - A hold settled by a sibling stayed selectable through `workspaceRecoverySourceEligible` when it named a `historicalWorkspaceSource`. - With saved user input waiting, the export hold stayed selectable and came first in each worker pass. `reconcileRetainedWorkspaces` skipped the other hold of the same run, so the no-replay hold never settled. **Expected behavior** After the save, the no-replay hold settles in the same or next pass. The task returns to `todo` when that hold owns its Blocked status. The settled hold is never selected again. **Steps to reproduce** 1. Use the staging-shape test in `workspace-recovery-capture-policy.test.ts`, with a waiting "try again" comment. 2. Run the worker until nothing is due. 3. On master, the worker never goes idle and the no-replay hold stays `resolved`/`blocked`. ## What Changed - `workspace-recovery-fallback.ts`: new shared `holdOwnsBlockedStatus`, used by `continueFromLastSavedWorkspace` and `settleSiblingRecoveredHold`. A settled no-replay hold that owns the Blocked status returns the task to `todo`. A newer decision's Blocked status is kept. An export hold's status stays with its own result finalization. - `workspace-recovery-source.ts`: `workspaceRecoverySourceEligible` excludes a resolved hold that carries `siblingWorkspaceRecovery`. - `workspace-recovery.ts`: `reconcileRetainedWorkspaces` runs several holds of one run one after another in that run's job, instead of skipping the later ones. - `workspace-recovery-capture-policy.test.ts`: the staging test now has the waiting comment, and a variant with `nativeFailureBlock` and `historicalWorkspaceSource`. It checks `todo`, no re-selection, and no repeat handoff in a later pass. - `doc/execution-semantics.md`: documents these rules. ## Verification - Red, with each rule disabled: the status stayed `blocked`; the settled hold was selected again; the worker never went idle (starvation). - Green: `npx vitest run --root server` on 14 workspace recovery suites (including `workspace-recovery-capture-policy`, `-fallback`, `-admission`, `-source`, `execution-blocker-read`, `preserved-native-response-wait.integration`): all passed. - `npx tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Several holds of one run now run in sequence in one job. Concurrency per run is unchanged (one job per run). - A no-replay hold releases the Blocked status only when it still owns it. Otherwise the status is kept. - Previous-release story: no new persisted state. Holds saved by earlier releases are handled on the first worker pass after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
805531cc71 |
fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both
## Linked Issues or Issue Description
Refs #20572, #20578
**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.
**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.
**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.
**Paperclip version or commit**
master at
|