mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
fff70fe49ecd79a14bc6831d19571376db72f305
6863
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
fff70fe49e |
fix(runner): relaunch a provider whose runner died before admitting it (#19784)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. When the control plane restarts, recovery starts a new runnerd process, and that process restores the provider (for example the Codex app server) from durable state. > - Each provider launch writes a durable startup fence before it spawns the process. Admission clears the fence. The fence stops a second runner from launching a provider while an unadmitted one may still run. > - A restart chaos wave sent SIGTERM and then SIGKILL to the control plane in quick succession. The second kill landed after the restored Codex provider was spawned and before it was admitted. > - The fence stayed in state with phase `spawned`. Every later runner refused to restore. The recovery probe's `runner.drain` failed with "provider startup ownership remains unadmitted", and the task needed a manual Retry. > - This pull request lets a new runner retire that fence when the recorded provider process and its process group have provably exited. It keeps every other fence closed. > - The benefit is that a double restart during provider relaunch continues automatically. ## Linked Issues or Issue Description **What happened?** A Codex run was active. A graceful restart (K2) suspended it, and recovery started a new runnerd that began restoring the provider. That runnerd spawned the Codex app server (pid recorded, phase `spawned`). A hard restart (K1) 9 s later killed it before admission. At the next boot, recovery chose `resume_dead_runner` and queued its `runner.drain` probe. The new runnerd failed the probe with `command_execution_failed: provider startup ownership remains unadmitted`. The run ended `native_session_recovery_failed`, and the task needed a Retry. The kept provider state had `lifecycle: "turn_active"` and a `startupAttempt` with `phase: "spawned"`, `trigger: "restore"`, `processId` = `processGroupId`, and no recorded exit. That process no longer existed. **Steps to reproduce** 1. Start a Codex task on a local runner. 2. Restart the control plane gracefully during the turn. 3. Kill the control plane again while recovery relaunches the provider, before the provider is admitted. **Expected behavior** Recovery relaunches the provider and continues the run. **Actual behavior** Every later runner refuses to restore, and the run fails. ## What Changed - `provider_backend.rs`: after a runner loads durable provider state, `retire_orphaned_startup_attempt` clears the startup fence only when all of these hold: - the attempt is in phase `spawned` (neither admitted nor failed); - the session is still restorable (`session_open`, `turn_active` or `provider_exited`); - signal 0 returns `ESRCH` for both the recorded process and its process group. It records a `provider_startup_retired` diagnostic event with the attempt, persists the state, and lets the normal restore path relaunch the provider. - Intent-phase fences (OpenCode). Three OpenCode runs after a K6 crash loop, and other runs on master, stopped at phase `intent` with no recorded process. Under load, the runner spends seconds verifying launch artifacts after it records intent. An intent alone could not show whether a process was started. - A new intent now carries `beforeSpawn: true`. - A new `SpawnStarting` observation clears the mark immediately before every provider spawn site in `start_with_tools_observed`. That clear is fsynced, but it emits no extra diagnostic event. - An intent still marked `beforeSpawn` (in a restorable session) proves that no process was started, so the fence is retired. - An intent that reached the spawn, or one written before this mark existed, stays fenced. - A runner that predates the field rejects the state as unknown (`deny_unknown_fields`), which fails closed. - `validate()` only allows the mark on an intent or on a spawn-stage failure. - The fence stays closed for: - an intent that reached the spawn, or one without the before-spawn mark; - a failed startup (initialization or spawn); - a session that recovery closed; - a process or group that may still run; - a permission error or an invalid id. - The existing test `failed_authenticated_identity_commit_keeps_the_durable_startup_fence` now uses a live process group instead of the fixed pid 123. That pid might not exist on a CI host, so the test would no longer prove a running provider keeps the fence. ## Verification - Red before the fix: the new integration test `restart_relaunches_a_provider_whose_runner_died_before_admitting_it` (fake Codex app server; turn active; a `spawned` restore fence with an exited process group; then the recovery `runner.drain` probe) failed with `provider startup ownership remains unadmitted`, the incident error. After the fix it passes. It shows one more `thread/resume`, the retired diagnostic, and a cleared fence. - Unit tests (unix-only module): retirement for each restorable lifecycle, and for an intent still before the spawn. A `SpawnStarting` outside its intent is rejected. The fence stays closed for: - an intent that reached the spawn; - an intent without the before-spawn mark; - a real child that survives its killed leader (process-group probe, no stubs); - a failed spawn; - a failed admission; - a closed session; - a live provider; - a surviving process group. - `cargo test -p paperclip-runner-core --test codex_provider`: 119 passed. The startup event counts are unchanged, because the spawn-start mark emits no event. The existing fence tests (`failed_provider_startup_is_persistently_fenced_before_another_process_can_resume`, `durable_recovery_closes_a_provider_that_reopens_a_settled_turn`) still pass. - `cargo test -p paperclip-runner-core --lib`: 570 passed. `cargo check --workspace --tests` passes. - `cargo fmt --check` passes. ## Risks - Process ids can be reused. A reused id only makes a gone process look alive, so the fence stays closed. Retirement needs `ESRCH` for both the pid and the group. - A provider process that moved to a new process group is outside this check. Before admission the runner has sent it no turn, so it cannot repeat provider work. - Intent fences written by older runners, including the kept incident states, have no before-spawn mark. They stay fenced and still need the Retry. New launches carry the mark. - A runner-side change takes effect on local runners after the runner build. Daytona needs a new image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9cc3760552 |
test(runner-e2e): allow controller outages up to ten minutes (#19797)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness and the runner-e2e recover-controller cells stop the control plane with `replaceController` (`tests/runner-e2e/controller-outage.ts`), hold it down, then start it again > - `parseControllerOutage` refuses holds longer than 300000 ms > - Restart chaos needs longer holds. One example is proving that a Daytona sandbox survives an outage longer than its old 5-minute auto-stop but inside the 7-minute runner reconnect grace (RC-54) > - With the cap, such a fault errors before the controller stops, so the scenario cannot run > - This pull request raises the test-harness cap to 600000 ms > - The benefit is that chaos runs can now inject 5–10 minute outages; production code and qualification contracts are unchanged ## Linked Issues or Issue Description **Bug description** A restart chaos replay with a 360000 ms K4 hold failed with "Controller outage requires SIGTERM/SIGKILL and 0..300000 integer milliseconds". The controller was never stopped. **Expected behavior** The test harness accepts outages of up to 10 minutes. The recover-controller receipt check keeps its own readiness limit (`readyAfterMs < 420000`). Refs #19544 ## What Changed - `tests/runner-e2e/controller-outage.ts`: adds `MAX_CONTROLLER_OUTAGE_MS = 600_000` and uses it in `parseControllerOutage` and its error message. `validControllerOutageReceipt` is unchanged and still requires readiness inside the 7-minute grace. - `tests/runner-e2e/controller-outage.test.ts`: the rejection cases now use 600001. A new test accepts 360000 (SIGKILL) and 600000 (SIGTERM). The existing five-minute contract test is unchanged. ## Verification - `vitest run tests/runner-e2e/controller-outage.test.ts tests/runner-e2e/server-restart-request.test.ts`: 52/52 pass. - `server-restart-request.test.ts` now uses 600001 as its out-of-range `controllerOutage`. The separate restart-request `controllerDowntimeMs` limit (five minutes) is unchanged. ## Risks - This change is test-only. No product or server code changes. - A suite that sets `PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS` above 300000 now runs instead of being refused. The recover-controller receipt check still fails any receipt whose readiness is outside the grace. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1107.0-canary.9 |
||
|
DottaandClaude Opus 5.5
|
35d41f11bb |
fix(runner): keep Codex thread usage reported before a run's first turn as history (#19783)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A new Codex run can attach to a Codex thread that earlier runs used > - In restart storm 9, a comment-wake run attached to that thread after a restart. Before its first turn, it received the thread's usage from the earlier run > - The runnerd transport gave that usage an empty turn owner. The driver rejected it as an unowned authoritative notification and failed the run > - The task then stopped with `native_event_replay_conflict` and needed the next run to finish > - This pull request keeps usage that arrives before a run's first turn as history > - The benefit is that a run is not lost to earlier work's usage, and it is not billed for that usage ## Linked Issues or Issue Description Refs #19718, #19722 **What happened?** Restart storm 9 (seed 7111), task CHA-5 (Codex). Run A succeeded. A K2 restart fired. The comment-wake run B then started on a fresh runnerd and attached to the same durable Codex session. Before run B started a turn, runnerd committed a `usage.reported` event for the thread's earlier usage. The transport rehydrated it as `thread/tokenUsage/updated` with `turnId: ""`, because no turn was bound yet. `classifyCodexNotification` classifies an empty id as `invalid_authority`. The driver failed the run with "Provider authoritative notification thread/tokenUsage/updated did not name its execution owner." Recovery then stopped with `native_event_replay_conflict`. **Expected behavior** Usage that arrives before a run's first turn does not fail the run. If the session has no usage baseline yet, that usage is history, so the run is not billed for it. **Steps to reproduce** 1. Use fake Codex with `--usage-before-first-turn`. It reports thread usage right after `thread/start`. 2. Open a driver session over the real runnerd transport. Wait until runnerd commits the usage, then start a turn. 3. Before this change, the run fails with "did not name its execution owner". ## What Changed - `runnerd-codex-transport.ts` (`rehydrateRunnerdUsageNotification`): the transport omits `turnId` when no turn is bound. It no longer sends an empty owner. - `codex-session-notifications.ts`: usage reported before the session's first turn, with no baseline yet, sets the baseline (history). Usage during or after a turn keeps its existing accounting. - Fake Codex: new `--usage-before-first-turn` flag. - New test `src/live/runnerd-codex-pre-turn-usage.test.ts` with real runnerd, the real Codex driver and fake Codex. ## Verification - Red, run twice before the fix: the test failed with `NativeProviderTerminalFailure: Provider authoritative notification thread/tokenUsage/updated did not name its execution owner.` This is the live message. - Green, run three times after the fix: the turn completes. The run's usage `runDelta` is 7/2 (thread total 12/3 minus the 5/1 reported before the turn). - `npx vitest run src/drivers/codex src/live/runnerd-codex-transport src/native-session-runtime`: 991 of 992 tests pass. One settlement test timed out under parallel load. It passes alone (19/19), and this change does not touch it. - `pnpm run typecheck` in `packages/paperclip-runner` passes. `cargo fmt` is clean. - Not run: the live restart-chaos replay. It needs provider credentials, which this session does not load. ## Risks - Low risk. The accounting change applies only when no turn is active, no turn is pending, the session has no settled turns and there is no baseline. In that state, the usage cannot belong to this run's work. - The test reproduces the failing shape (usage committed before the first turn) with a fresh session. It does not reproduce the exact restart and run-attach sequence from the storm. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6adc8b1049 |
fix(recovery): keep restarted Retry attempts as task directories and settle failed re-preparation (#19759)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery lets a user click Retry after a run fails, and a native run restarts its own preparation after a control-plane restart > - A Retry for a task without a project runs in a fresh task directory under `isolated-workspace-attempts` > - When a restart hit that Retry during bootstrap, preparing it again treated the task directory as the base of a new git worktree and failed > - The failed run then kept its native recovery claim open, so the task showed Retry but every Retry click created no run > - This pull request keeps the restarted attempt in its task directory and settles a failed re-preparation > - The benefit is that a Retry finishes after a restart, and a failed one can be retried again ## Linked Issues or Issue Description Refs #19666 **What happened?** A task without a project failed after a restart (`native_session_interrupted`). The user clicked Retry. The Retry run started in a fresh task directory. A restart hit it during bootstrap. Restart recovery prepared the same run again and failed with `workspace_validation_failed`: "requested isolated_workspace with git_worktree, but base workspace `.../isolated-workspace-attempts/<company>/<issue>/<run>` is not a git checkout". The task rested blocked with Retry shown. A later Retry click created no run: the receipt stayed deferred with "New work must wait while a native run recovers its unfinished accounting." A second storm wave showed the same failure on a Claude C1 task. **Expected behavior** Preparing a run again after a restart reuses the task directory that run already bound. A git worktree base is only the project's checkout or the task's own worktree. A run whose re-preparation fails leaves a working Retry. **Paperclip version or commit** `master` at `51eb4ff705` and `131b566ca4` (both include #19666). Restart-chaos waves on 2026-11-07. **Steps to reproduce** 1. Use a task with no project and isolated workspaces enabled. Let a native run fail after a restart. 2. Click Retry. The run starts in `isolated-workspace-attempts/<company>/<issue>/<run>`. 3. Restart the control plane while the run bootstraps. 4. The re-preparation fails with `git_worktree_base_not_git_checkout`. Click Retry again: no run starts. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `readFreshWorkspaceReadiness` returns `null` for a native run that already has admitted input and a bound workspace (from #19693). This is correct: the restart must resume the admitted workspace. - `server/src/services/heartbeat/environment-selection.ts` (`boundTaskDirectory`) excluded every native recovery binding from the task-directory path. With readiness no longer fresh, `useIsolatedTaskDirectory` was false. The default `isolated_workspace` strategy (`git_worktree`) applied, with the task directory as its base. - The restart recovery claim set the native coordinator to `observed` with its lease owner. `failRunSetup` failed the run, but nothing released that claim. `budgets.ts` reads an open coordinator as unfinished accounting, and Retry admission reads it as `controller_settling`. ## What Changed - Environment selection treats a native recovery binding as the task directory when it is the task's current binding and a bound isolated task directory. The directory strategy is kept; it never becomes a git worktree base. A recovery binding the issue no longer points to keeps its existing path. - New `settleFailedNativeRecoveryPreparation`. When a `bootstrap_incomplete` or `resume_dead_runner` restart claim fails its re-preparation before any provider dispatch, it settles the claimed coordinator as `terminal_failure` and wakes deferred work. It only changes a coordinator that is still `observed`, owned by that claim, without a result, for a failed run. Reattach claims (live runner) are not changed. - `doc/task-recovery.md` describes both rules. ## Verification - `environment-selection.test.ts`, RC-58 cases: a restarted fresh attempt bound to the task re-prepares as its task directory (`useIsolatedTaskDirectory` true). It failed before the fix and passes now. A binding the issue moved away from keeps the old behavior. - `native-runtime/native-recovery-preparation-failure.test.ts` (embedded Postgres): before settlement, Retry admission reports `controller_settling` (the dead Retry). After settlement the coordinator is `terminal_failure` with the run's error code, and admission no longer waits on the controller. Settlement is idempotent. Another owner, a reattach claim, a run that is not failed, and a coordinator finalizing elsewhere are left unchanged. - Local regression: `environment-selection`, `workspace-preparation`, `retry-workspace-continuity`, `native-recovery-preparation-failure`, `explicit-native-continuation` (588 tests passed). - Restart-chaos: local restart-chaos runs on `3a7abef4e5` passed, but neither one reproduced the exact failure (a restart during the Retry's bootstrap). The race depends on timing, so the unit tests above are the red/green proof for both defects. - `--replay w-20261107T011010-s7403/t02` (codex C1 K4 with the original fault sequence): PASS, R1-R6 and T. The replay's Retry did not cross a bootstrap restart. - Wave `w-20261107T064432-s7210` (codex+claude, C1+B1, K1/K4/K6, `--triggers retry`, concurrency 2): 8/8 PASS. It had 4 Retries, including C1 tasks with a fault fired right after the Retry click. No `git_worktree_base_not_git_checkout` and no dead Retry. The server log has no "incomplete native runner bootstrap" restart in this wave either. - `ce7e3f9b19` registers execution work before the settlement writes (Greptile finding). ## Risks - A native recovery now re-prepares a bound task directory through the task-directory path. That path still validates the binding, owner, status and directory identity before use. - Settling a failed re-preparation ends that run's recovery claim. It applies only when the previous runner is verified gone (`bootstrap_incomplete`, `resume_dead_runner`) and no provider dispatch started in this attempt. A Retry still passes its normal stop, ownership and admission checks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
af0fe8fe81 |
Keep personal MCP connections functional while idle tenants sleep (#19427)
## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). **What happened?** Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. **Expected behavior** Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. **Steps to reproduce** Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
909f4d1a33 |
fix(native): recover exact historical provider pack code (#19779)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain the provider pack that admitted their saved
state.
> - An image update can replace that pack's executable, proxy, or
contract.
> - Older sessions can predate the durable three-file launch artifact
receipt.
> - The current image's archive cannot satisfy a different pinned
manifest.
> - This pull request obtains the exact public code from its recorded
release image.
> - Full source, archive, and installed-file checks still control
provider admission.
## Linked Issues or Issue Description
Refs #19558, #19622, #19755.
A saved OpenCode session can fail with
`runner_remote_provider_archive_invalid` after an image update. Its
original provider state and checkpoint remain valid, but its pinned pack
differs from the controller's pack. The old manifest can name a release
that still has the exact code in GHCR. Matching the OpenCode version
does not prove matching proxy or contract bytes.
## What Changed
- Resolve a pinned historical pack from the recorded source-SHA image in
the public Paperclip GHCR repository. Verify the platform manifest, raw
config, source identity, dedicated code layer, selected pack manifest,
archive contents, paths, and hashes.
- Keep a private temporary cache of verified public code. Rebuild it on
an empty host. Bound each source to 4 GiB and three minutes, with two
active imports or cache verifications, 32 distinct queued sources, and
disk-headroom checks. Share warm-cache verification by manifest while
each caller keeps its own cancellation.
- Select this verified archive through the existing uploader. Preserve
installed-pack and preinstalled-pack checks. Refuse unavailable
historical transport before clearing the target. Keep the current-source
identity guard from #19755. Refuse an oversized layer argument before
any target effect.
- Attach the source wait to the actual per-turn startup. Stop, lease
loss, and startup completion cancel it. Keep existing Runner admission
and turn limits.
- Document the registry retention requirement and the distinction
between public code recovery and encrypted provider-state backup.
## Verification
- Genuine baseline RED: the real uploader rejects the selected old pack
before any remote effect. Treatment installs the exact old code through
that uploader.
- Current head: all 19 source/integrity/cache/deadline controls, all 45
archive controls, and 29 actual-caller and manifest/startup controls
pass (93 total). Four review regressions fail against the prior head and
pass with the fixes. The server and colocated test TypeScript graph
passes. Module-boundary and whitespace checks pass.
- Actual public GHCR cold reconstruction: 1,054,587,008 compressed code
bytes, 115.4 s, about 248 MiB maximum owner RSS. All 114 file layers and
the overlay passed source validation.
- Linux Node 24.22.2 fixture: the real historical pack passes the normal
uploader, full installed-file verifier, and pinned OpenCode version
check. It used one transfer, no application startup, no network, and no
provider calls. Revision12b0d2ba passed in 220.2 s with about 282 MiB
owner RSS. The earlier prototype passed in 108.5 s; both measurements
are retained. The three-minute source limit and the fixture's
four-minute limit were unchanged. The later review fixes add shared
cache verification, hash cancellation, and an early size guard. They do
not change extraction or the accepted Linux fixture input.
- Earlier fixture failures are retained: unsafe hardlink classification
was corrected with confined-link controls; a cached image lacked the
required zstd CLI; Docker Desktop bind storage caused GNU tar
directory-identity errors. The passing fixture uses the official Debian
zstd binary and an owned Linux tmpfs target. Production extraction code
is unchanged.
- Fresh Greptile review is 5/5 on
|
||
|
DottaandPaperclip
|
ae2d2956b3 |
Reject relative local Runner paths before native admission (#19776)
## Thinking Path > - Paperclip manages AI agents and their work. > - Heartbeat selects the runtime that will execute a task. > - A new local Native run uses the controller's Runner binary setting. > - A relative binary path is invalid. The resolver already rejects it, but that check occurs after Native admission. > - This pull request checks the same setting before it creates Native execution authority. > - Remote providers and admitted Native recovery keep their existing behavior. ## Linked Issues or Issue Description **What happened?** A relative `PAPERCLIP_RUNNER_BINARY` setting can reach local Native admission before the binary resolver rejects it. The run has transferred preparation ownership even though this setting cannot start the local Runner. **Expected behavior** Reject this invalid setting before creating a completion contract or Native coordinator. Keep the existing preparation owner and the run row unchanged. **Steps to reproduce** Set `PAPERCLIP_RUNNER_BINARY=relative-runner`. Select a new local `paperclip_runner` Native run. The selection must report the existing absolute-path error before Native admission. Refs: #19466 and #19340. Those merged changes repair Retry authority and saved-workspace recovery. This change only moves local path validation before new admission. ## What Changed - Add a pure absolute-path check beside the existing binary resolver. Both use the actual controller environment setting. - Run that check before a new local Runner admission. Skip remote execution, API-only providers, and an already admitted Native run. - Test the rejection, unchanged preparation ownership, controller setting selection, remote behavior, and immutable recovery. Keep the existing resolver checks. ## Verification - The full binary test file passed: 32 tests. - The full runtime-selection file passed against a dedicated PostgreSQL database: 62 tests. It exercises real admission transactions, contracts, coordinator rows, and instruction-copy controls. It replaces context materialization and remote workspace synchronization at their existing test boundaries. - After normal integration with current master, the normal Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. The original recovery-preparation case passed again on this integrated revision against a fresh dedicated database (1 passed; 61 filtered). The earlier full 32-test binary and 62-test selection results carry only where their source and executed boundaries are unchanged. - The first private selection wrapper failed during module resolution, before any case or database use. A TypeScript wrapper with the same body recovered that setup failure. This was not a product regression. - Reproduce with `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-codex-runner.test.ts --maxWorkers=1 --no-file-parallelism`. - Set `PAPERCLIP_RUNTIME_SELECTION_TEST_DATABASE_URL` to a dedicated disposable test database before running `pnpm --filter @paperclipai/server exec vitest run src/services/heartbeat/runtime-selection.test.ts --maxWorkers=1 --no-file-parallelism`. ## Risks - This check validates path syntax only. An absolute path can still fail the existing file-access or artifact checks later. - A wrong branch condition could reject remote or existing Native work. The tests cover these exclusions. - This does not prove that a real provider can start or that a deployed service is healthy. It changes no database schema or retry authority. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source inspection and code-execution tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
ee1e1a3994 |
fix(runner): adopt the turn a dying controller's turn.start already started (#19772)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote Runner hosts the provider (here Claude through ACPX) and
keeps working while the control plane restarts
> - A controller can die after the Runner accepts its `turn.start` but
before the controller checkpoints the accepted turn or emits
`turn.accepted`
> - The provider keeps running that turn. On reattach, recovery saw no
checkpointed active turn, and the Runner's replayed `turn/started`
failed the session with "Provider started an unexpected turn"
> - The run then failed with `native_event_replay_conflict`, replay was
blocked, and a finished issue moved from done to blocked
> - This pull request lets recovery adopt that turn when the Runner's
authenticated command result proves it is the controller's own turn
> - The benefit is that a controller crash loop no longer fails a run
that the provider is still doing correctly, and a foreign turn still
fails closed
## Linked Issues or Issue Description
**What happened?**
In a restart chaos run (Claude through ACPX on a remote Runner, crash
loop fault), a follow-up comment woke a new run. The controller sent
`turn.start` and the crash loop began about one second later. After the
restarts, the controller reattached to the same Runner and failed with
"Provider started an unexpected turn". The run failed with
`native_event_replay_conflict` and recovery required workspace repair.
**What did you expect?**
The new controller generation adopts the turn that the old generation
started, waits for its terminal, and finishes the run.
**Evidence**
The durable controller journal for the run shows command 4, `turn.start`
with `turnId: turn_lab_18ac…`, completed with `{ providerTurnId:
turn_lab_18ac…, status: accepted }`. The Runner emitted `turn.started`
for that turn 1.4 s after the command, after the old controller died.
The replacement's `session.snapshot` returned `activeProviderTurnId:
turn_lab_18ac…`. The controller's own checkpoint had no active turn and
the run had no `turn.accepted` event.
Related: #19711, #19691, #19718
## What Changed
- `codex-app-server-driver-impl.ts` `recoverSession`: when the
checkpoint has no active turn, no semantic result, and no autonomous
goal or disposition turn applies, adopt the single in-progress provider
turn if the transport proves that this run's latest completed
`turn.start` accepted that exact turn id. Any other active turn is not
adopted and still fails closed.
- `codex-session-state.ts`: a recovered session with an adopted turn
emits the missing `turn.accepted` before any of that turn's
notifications.
- `app-server-transport.ts`: new optional transport method
`controllerAcceptedTurnId()`.
- `runnerd-codex-transport.ts`: implement it from the durable command
journal (`controllerAcceptedTurnStartId`). The latest `turn.start` must
be completed and accepted, its result must name the requested turn id,
and no run attachment may come after it.
- `DURABLE_TRANSPORT.md`: document the adoption rule.
## Verification
- Red/green with the real `paperclip-runnerd` and the scripted ACPX
Claude sidecar: `npx vitest run
src/control-plane/acpx-controller-recovery.test.ts -t RC-59`. The old
controller sends `turn.start` and stops before the provider accepts it.
The provider accepts and starts the turn while no controller is
attached. A replacement controller adopts the live Runner through the
production transport and `CodexAppServerDriver.recoverSession`. Before
the fix: `activeTurnId` is null (the session then fails on
`turn/started`). After the fix: events are `turn.accepted`,
`turn.started`, `turn.completed` for the same turn, with one
`turn.start` to the provider.
- Driver tests: `npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts -t
"controller that died"`. Adoption passes. With no accepted result, or a
result for a different turn, `startTurn` still fails with "Provider
started an unexpected turn."
- Journal unit tests: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t
controllerAcceptedTurnStartId` (pending, failed, rejected, mismatched
turn, later attachment, later pending start all return null).
- Regression: `src/drivers/codex` (all files),
`src/live/runnerd-restart-continuation.test.ts`,
`src/live/runnerd-codex-transport-adoption.test.ts`,
`src/live/runnerd-codex-transport-release-resume.test.ts`,
`src/control-plane/acpx-controller-recovery.test.ts` pass. `pnpm run
typecheck:typescript` passes in `packages/paperclip-runner`.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.
## Risks
- Low to medium. The change only acts when the checkpoint has no active
turn and the Runner reports exactly one active turn. Adoption needs the
Runner's authenticated, run-scoped `turn.start` result for that exact
id. Without that proof, behavior does not change.
- If a `turn.start` result is still pending when the replacement reads
the thread, recovery does not adopt and fails closed as before.
- Not proven on Daytona. I did not run the paid replay.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandPaperclip
|
2d4f80504e |
fix(runner): retain exact ACPX checkpoints after rejected turn starts (#19762)
Preserve the exact idle ACPX conversation when a matching turn.start response rejects work with retryable SESSION_RESUME_REQUIRED. Require verified identity, no pending work, confirmed process cleanup and the provider lifetime fence before normal drain and suspension can publish a reusable checkpoint. Unknown acceptance and arbitrary closed states remain closed. The real Runner regression completes the next turn on attempts 2 and 3, requires successful terminal status for the exact submitted turn, and checks expectedIdentity on both resume requests. Controlled wire faults reject failed, cancelled and missing-identity recovery. Validation: 193 affected Rust tests on unchanged Rust source/fixtures; 134 recovery integration tests, 14 environment-gated skips; 21 Claude effort tests; all server typecheck layers. Exact-head CI: 59 successful checks and statuses, 6 intentional skips. Greptile Apex 5/5, no unresolved findings. Hosted deployment and task acceptance remain separate. Fixes: #19751 Refs: #19366 Refs: #19745 Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7475577cdb |
fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run executes in a Daytona sandbox and keeps working while the Paperclip server restarts > - The server reattaches that runner within a 7-minute reconnect grace > - Daytona stops an idle sandbox after its `autoStopInterval`, and it counts only SDK and lifecycle calls as activity > - With the server down, nothing touches the sandbox, so a 5-minute auto-stop kills the runner before the server returns > - This pull request makes the host tell the provider its reconnect grace, so Daytona never auto-stops a runner sandbox inside that grace > - The benefit is that a controller outage of up to 7 minutes no longer turns a healthy remote run into a blocked task ## Linked Issues or Issue Description Refs #19199 (the same floor for the warm runner idle window). Refs #19741 (follow-up delivery through restarts; no overlap). **What happened?** A restart-chaos wave on Daytona killed the server for 4m51s while a per-turn remote run was active. The environment used `autoStopInterval: 5`. The last sandbox activity was at 06:09:16. The server came back at 06:14:19 and then crashed again. At 06:15:21 recovery called the sandbox. The plugin found it not started and started it again. The runner identity marker was still on disk, but its process was gone: `native.remote_runner_recovery_failed {stage: process_verification, reason: process_not_running}`, then `runner_remote_process_identity_unavailable`. The task went from done to blocked. An earlier wave showed the same failure after a 4m45s outage. **Expected behavior** A sandbox that holds a live remote runner survives any controller outage up to the reconnect grace. The server reattaches the runner and the run continues. **Steps to reproduce** 1. Create a Daytona environment with `autoStopInterval: 5` and a `paperclip_runner` agent. 2. Start a remote run and kill the server for more than 5 minutes (less than 7). 3. Restart the server. Recovery reports `process_not_running` and the task becomes blocked. ## What Changed - `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns `REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox environment. - `server`: forward `runnerReconnectGraceMs` to the provider worker on lease acquisition and on the recovery reattach resume. It uses the same provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease identity, lease metadata, or saved environment config. - Daytona plugin: the effective auto-stop is at least the largest host floor (warm idle window or reconnect grace) plus 5 minutes. With the current grace this is 12 minutes. The plugin applies it at creation and raises an older sandbox's shorter interval on resume (bounded, best effort). - Daytona plugin: keep the forwarded grace out of the normalized config. Update the `autoStopInterval` manifest description. - `doc/execution-semantics.md`: document the reconnect-grace floor next to the warm floor. ## Verification - Red/green Daytona plugin tests: `cd packages/plugins/sandbox-providers/daytona && npx vitest run --config vitest.config.ts`. The new cases failed before the fix (for example `expected 120000 to be >= 720000`, and no `setAutostopInterval` call on resume). All 605 tests pass after the fix. - Server tests: `cd server && npx vitest run src/__tests__/environment-run-orchestrator.test.ts src/__tests__/environment-runtime.test.ts`. 247 pass. They prove acquisition forwards the grace only for `paperclip_runner` on a sandbox, that the reattach resume forwards it, and that it does not reach lease metadata or saved config. - `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are clean. - `pnpm verify` asks for broad CI coverage because a doc changed. Required CI supplies it. - Evidence from the kept wave database: lease metadata shows `autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode: per_turn`. The run had no events between 06:09:16 and 06:14:19. ## Risks - Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12 minutes instead of a shorter configured value. Cleanup still stops or destroys released sandboxes. Explicit `0` and longer values are unchanged. Auto-archive and auto-delete are unchanged. - I did not inspect the sandbox state in Daytona directly. Cleanup destroyed it before this investigation. The proof is the timing, the lease config, and the `process_not_running` result after a restart that kept the disk. - No live Daytona replay ran. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit, code execution) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
e2c7d81067 |
fix(server): keep reply recovery off web startup critical path (#19769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server restores retained native replies during startup. > - Fair reply recovery can admit more due work while an earlier group remains active. > - Waiting for that whole pass can hold web readiness on unrelated retained replies. > - This pull request tracks reply recovery with the existing background startup owner. > - The server can become ready while idle and shutdown still account for that work. ## Linked Issues or Issue Description Refs: #19726. **What happened?** The startup path awaited the native response worker's first pass. That pass can remain active while fair dispatch admits later replies. A controlled `startServer` regression stays in recovery until the held reply pass finishes. A deployed backlog supported investigating this boundary, but no captured runtime stack proves it caused the observed startup delay. **Expected behavior** Retained reply recovery must keep its authority checks, retry deadlines, and fair dispatch. Its completion must not be a condition for unrelated web readiness. Active recovery must still block idle and be joined at shutdown. **Steps to reproduce** Run the startup regression with a held native response worker promise. Before this change, `startServer` does not complete. After this change, startup reports ready, idle work remains active, and shutdown waits until the promise settles. ## What Changed - Track the initial native response pass with the existing heartbeat scheduler work owner. - Extend the actual startup wiring test to cover readiness, idle accounting, worker stop, and shutdown join. - Document the response recovery boundary for idle and readiness. ## Verification - Mandatory agent preflight passed on macOS arm64 with Node 26.4.0. - True RED: the new startup case failed because `startServer` had not completed while reply work was held. The same case passed after the change. - Final head `279e97587b` passed the full startup test file: 123/123 tests. - The unchanged real PostgreSQL later-page case passed. It holds 100 old publication rows, publishes the independent 101st reply once while the first pass remains pending, then verifies completion and no recurring idle SQL. - Final-head CI passed: 60 successful checks and 3 skipped checks, including `ci / verify`. Fresh Greptile is 5/5 with no open threads. - Server TypeScript check passed: `node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit`. - Initial test-launch and missing generated SDK setup failures were resolved before the regression ran. They are not RED or pass evidence. - No deployment or live recovery action was performed for this PR. Focused commands, from `server/`: ```sh node ../node_modules/vitest/vitest.mjs run src/__tests__/server-startup-feedback-export.test.ts --maxWorkers=1 --no-file-parallelism node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-response-publication-retry.test.ts -t 'publishes a later-page reply while older publication groups remain contended' --maxWorkers=1 --no-file-parallelism --hookTimeout 90000 ``` ## Risks Web readiness can complete before retained reply recovery finishes. The existing scheduler still owns errors, retries, scope checks, and delivery authority. The existing idle tracker and shutdown join retain in-flight work. Other startup recovery gates stay in place. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
beb3e55072 |
Continue a released Runner after confirmed graceful eviction (#19760)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core can release a completed Runner while its host retains the session. > - A host can later retire that Runner under its retention policy. > - Graceful retirement saves a suspended session before physical shutdown. > - A cached Core owner still expects the old physical task and rejects a new allocation. > - This pull request retires that dead local controller after verified stop. > - The benefit is ordinary session continuation for Codex and qualified Claude. ## Linked Issues or Issue Description Refs #19611, #19683, #19196, #19733 and #19157. After a host gracefully evicts a released task, the next run can receive a new allocation. A cached controller rejects it with `native_session_released_owner_scope_mismatch`. The detached-controller path can already recover the suspended session. ## What Changed - Require the exact source, old task and current account/workspace scope before releasing a cached controller. - Use the existing physical-stop receipt checks. Detach only the local controller, then use the same suspended-state admission as the detached path. - Preserve the current run's instruction reservation. Retain a failed detach for cleanup. - Add actual Core, PostgreSQL and native Runner tests for Codex and qualified Claude through ACPX. Route them once through the native CI lane. ## Verification - The original real-entry Codex case failed at the cached-task scope check after native retirement, confirmed process exit and a recorded stop receipt. - The current head's native CI lane passed all 16 eviction cases for Codex and qualified Claude. These check cached and detached owners, current instruction preparation, reused-allocation inspection, missing stop/state, foreign identity and non-suspended state. They verify the same provider session, two ordinary turns, one accepted source result and no replacement submit on refusal. - Local checks passed the composition matrix, 12 targeted executor controls, 43 CI lane-selection checks and server TypeScript. All eight refusal cases now assert the expected error and whether the checkpoint request or host read was reached. This addresses the first standard review's test finding. - Commands: `pnpm exec vitest run src/services/native-runtime/plugin-runner-eviction.integration.test.ts`, focused `native-session-executor.test.ts` controls, `node ../node_modules/typescript/bin/tsc --noEmit -p tsconfig.json`, and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42349579107) passed on head `9a25709ef43365b1f1849413b935785838eca19b`. [Standard review](https://github.com/paperclipai/paperclip/pull/19760#issuecomment-6703629255) is 5/5 on this head, with no unresolved findings. Whitespace and added-line secret/PII checks passed. ## Risks - Controlled provider processes and an authenticated local host fixture model the Launcher edge. This does not qualify live vendor transcript storage, mounts or a deployment. - This covers graceful eviction before allocation or before stopped-task inspection during `describe`. Loss after that inspection or an ambiguous claim remains held for reconciliation. No unavailable reply proves physical stop. - Missing or invalid suspended state still refuses continuation. The deterministic refusal rule from #19733 remains unchanged. Abrupt death without a suspended checkpoint stays with the separate #19157 work. - No new capability, mode, schema, dependency, deployment or activation is added. The existing draft #19157 and dependent branches remain unchanged. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
github-actions[bot]andlockfile-bot
|
e95dcadbd2 |
chore(lockfile): refresh pnpm lockfiles (#19773)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>canary/v2026.1107.0-canary.8 nightly/v2026.1107.0-nightly.0 |
||
|
Nicky Leach
|
8be6d70538 | fix(runner): select Claude effort again after a cold session resume (#19708) | ||
|
DottaandClaude Opus 5.5
|
3895195f4b |
fix(native): report a retained provider pack the controller does not ship as incompatible (#19755)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote OpenCode runs stage a controller-owned provider pack into the sandbox before the provider starts > - Since #19558, an ordinary OpenCode continuation retains the pack that an earlier release admitted for its saved provider session > - After a controller release changes the pack, the controller no longer ships those bytes, and the continuation must stay blocked > - The configured-pack upload path did not detect that case: it read the build-time archive receipt for the old digest and failed with `runner_remote_provider_archive_invalid`, a generic, retryable error that suggests corrupt image files > - This pull request compares the configured pack's identity with the selected identity before any upload and fails as `runner_remote_provider_artifact_incompatible` > - It also puts a fixed reason in each real archive integrity error > - The benefit is a correct, permanent classification that matches the bundled-image path, no wasted retries, and a diagnosable message for real archive corruption ## Linked Issues or Issue Description Refs #19558, #19743 **What happened?** On a hosted deployment, two remote OpenCode runs of a review agent failed before the provider process started. The error was `runner_remote_provider_archive_invalid` (classified as a generic `native_session_interrupted`). Both runs continued an issue whose provider session was admitted by an earlier release. In both runs, the run log shows `provider_pack.compatibility`, `provider_pack.verify`, and `provider_pack.verify_preinstalled` fail, then `provider_pack.archive.verify` fails after 4–6 ms. Fresh runs of the same agent on the same controller passed pack preparation with the current pack. The cause is not a cache race. The archives are created at image build time (`Dockerfile` `runner-provider-pack` stage) and the server never rewrites them. The retained authority selects the old pack digest. `readProviderPackLayers` then compares the receipt's `providerPackDigest` (current release) with that old digest and throws `runner_remote_provider_archive_invalid`. Directory staging (no archive) instead clears the staged remote pack, uploads the different pack, and fails only at the final verification. **Expected behavior** A continuation pinned to a pack that the controller does not ship fails as `runner_remote_provider_artifact_incompatible`, as the bundled-image path already does ("the image provider pack did not match the bundled release identity"). It does not clear or upload anything. Real archive corruption keeps `runner_remote_provider_archive_invalid` with a fixed reason that the run report shows. **Steps to reproduce** 1. Build a provider pack and its archives for digest A. 2. Call `uploadVerifiedProviderPack` with that pack and a selected (retained) digest B. 3. Before this change: the call fails with `runner_remote_provider_archive_invalid` (v1 and v2 archives), or resolves after it clears and re-uploads the remote pack (directory source). ## What Changed - `uploadVerifiedProviderPack` takes the configured pack's `sourceDigest`. If it is not the selected `expectedDigest`, the call fails with `runner_remote_provider_artifact_incompatible: selected provider pack <12 hex> is not the controller's pack <12 hex>; ...`. The check runs before the remote clear, the archive read, the directory hash and any upload. - The executor passes the configured pack's identity (`readRemoteProviderPackIdentity(root, false)`, the same reader that selects the current pack). - Archive integrity errors keep the `runner_remote_provider_archive_invalid` prefix and add a fixed reason: `receipt_unreadable`, `receipt_identity_mismatch`, `receipt_invalid`, `layer_invalid`, `layers_unreadable`, `archive_unreadable`, `archive_not_file`, `size_mismatch` or `sha256_mismatch`. The run report keeps only the outer message, so the reason must be in the message. Filesystem causes stay on `cause` and do not put private paths in the message. - The v1 reader now uses the shared `verifyArchive` helper. - `doc/execution-semantics.md` documents the rule. ## Verification - Red before the fix (production files unchanged, new tests only): `npx vitest run src/services/native-runtime/provider-pack-archive.test.ts` gave 5 failed / 38 passed. The v1 and v2 cases failed with exactly `runner_remote_provider_archive_invalid`, which is the hosted error. The directory case resolved after it cleared and uploaded the remote pack. The reason tests got a bare `runner_remote_provider_archive_invalid`. - Green after the fix: the same file gave 43 passed. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1245 passed. This includes the #19558 continuation cases. - `remote-provider-pack`, `remote-provider-pack-authority` and `remote-provider-pack-verification` tests: 69 passed. - `pnpm --filter @paperclipai/server typecheck`: all layers passed. - Review follow-up (`c8156841a9`): an unreadable numbered layer now reports `archive_unreadable`. The layered-corruption test deletes `0.tar.zst` and checks for that reason before any transfer. The archive file gave 43 passed. - Hosted evidence came from read-only run, event and issue-run reads only. No hosted retries, recovery or deploys. ## Risks - This change does not unblock an affected session. It makes the failure honest and permanent: `runner_remote_provider_artifact_incompatible` is already a permanent, pre-session failure code, so the run is not retried with the same result. To recover, the session needs its original artifacts (the captured provider-artifact path, which #19743 repairs for new admissions) or an authorized continuity break. - Low risk for other paths. Fresh launches select the configured pack's own identity, so the digests are equal. Restored provider artifacts and compatible installed packs return before this upload path. - Callers that match `runner_remote_provider_archive_invalid` with a substring still match. No code compares the exact message. - Not tested end to end against a hosted sandbox. The executor-level wiring of `sourceDigest` has typecheck coverage but no dedicated executor test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with shell, code editing, test execution and read-only authenticated browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
167e39c4ef | fix(runner): keep warm run.attach events on ordinary accounting (#19707) | ||
|
DottaandClaude Opus 5.5
|
b03ea621c6 |
fix(environments): retry a teardown interrupted by server death at once (#19749)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a remote run, Paperclip destroys its provider sandbox. The teardown first claims the lease with a 15-minute ownership fence, so no other cleanup runs at the same time > - #19696 made restart recovery end a fence when the claiming server process is proven to have exited. It only does that for claims that record the claimer > - The run-release teardown and the stop-only teardown did not record the claimer > - When the server died during a run-release teardown, the sandbox therefore stayed up until the 15-minute fence expired > - This pull request records the claimer on those claims, and stops a teardown attempt lost to a dead server from counting as a failed attempt > - The benefit is that a restart during sandbox teardown destroys the sandbox at the next startup sweep ## Linked Issues or Issue Description **What happened?** In a Daytona restart chaos wave, a run finished at 05:34:48. Its release claimed the lease and called the provider, and a SIGKILL stopped the server 2 seconds later. After restart the lease stayed `pending_cleanup` with an in-flight claim. The sandbox was destroyed at 05:49:49, when the 15-minute claim fence expired. There was no provider error. **Expected behavior** A teardown interrupted by the server's death is retried by the next startup sweep. It is not counted as a failed attempt. A real provider error keeps the normal backoff. **Steps to reproduce** 1. Release a run's sandbox lease with destroy, and hold the provider call. 2. Stop the server during the provider call. 3. Restart. On master, the sweep skips the lease until the fence expires 15 minutes after the claim. Refs #19696. ## What Changed - `environment-runtime.ts`: the run-release teardown claim (`destroyClaimedSandboxLease`) and the stop-only teardown claim save `pendingCleanupController`, bound to the claim's attempt id, the same way sweep claims do since #19696. - `heartbeat/recovery.ts`: sweep claims mark that they counted an attempt. When the sweep ends the fence of a lost claim, it gives that attempt back, so a teardown lost to a dead server does not move toward the cap or the long cooldown. Claims from other paths did not count an attempt, so nothing changes for them. - Workspace-repair and export-retry fences are unchanged. They protect workspace work, not teardown. ## Verification - Red, then green: - `environment-runtime.test.ts` → "retries a run-release teardown at the next sweep after the releasing server dies". The provider call is held during `releaseRunLeases(..., "destroy")`, the claim is marked as from an exited server, and a restarted sweep runs. On master the claim has no controller record and the sweep skips the lease. With the fix, the sweep destroys it at once and records attempt 1. - `heartbeat-pending-cleanup-sweep.test.ts` → "does not count a sweep attempt lost to a dead server as a failed attempt". On master the retry is recorded as attempt 4; with the fix it is attempt 3 again. - `environment-runtime`, `heartbeat-pending-cleanup-sweep`, `sandbox-creation-operations` and `environment-routes` pass (367 tests). `tsc --noEmit` for `server` passes. ## Risks - The proof that a claimer exited is the same-PID-space proof from #19696. A live, foreign or unidentified claimer keeps its fence. - Giving back a counted attempt applies only after that proof. A teardown that fails with a provider error still counts and backs off. - No schema change. The new fields live in existing JSON metadata. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
33ca62f776 |
Keep external-agent feature reads on their admission transaction (#19750)
## Thinking Path > - Paperclip manages work for AI agents. > - External native agents must pass current feature and authority checks. > - The broker checks feature settings inside its admission transaction. > - The old feature reader uses the root database pool. > - Two active transactions can leave no connection for that read. > - This change uses the admitted transaction for the same policy read. > - It keeps the existing company, principal and controller checks. ## Linked Issues or Issue Description Refs: #19340. That merged change fixes retained native Retry ownership. This change leaves that recovery contract intact. Related: #18744 and #17833 fix caller-owned transaction reads in other domains. Root searched live GitHub for open externalBotEnabled changes and found no duplicate. **What happened?** An external-agent admission can wait for another pool connection while it holds a row lock. A second transaction can occupy the remaining connection and wait for that lock. The feature read must use the transaction that already owns admission. **Expected behavior** Dot, Grok and Muse read current feature settings on the admitted transaction. Disabled providers remain disabled. Current authority checks remain in force. **Steps to reproduce** 1. Use a real PostgreSQL pool with two connections. 2. Lock the settings row in the admission transaction. 3. Occupy the second connection with a transaction that waits for that row. 4. Read the existing provider feature policy through the admitted owner. 5. Require the read to finish before the competing lock is acquired. **Paperclip version or commit** The fix was tested at `d8dece250b75cffee67d3b7f4f0e669826282164`, based on master `839d41aabbc6d042250548aedcaa5c50d369dca3`. It now includes master `131b566ca4fa866c4348739c75cfe48c805b3398` through normal merge `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a`. All seven fix files retain their tested blobs and modes. ## What Changed - Pass the admitted select owner through the broker identity contract. - Use that owner for Dot, Grok and Muse feature settings. - Add a read-only settings projection that reuses the existing managed and staging view. It does not create the settings row. - Add six real PostgreSQL controls for enabled and disabled providers. - Add three broker-path PostgreSQL regressions for exact transaction forwarding through Dot, Grok and Muse. - Keep all newer master broker changes and existing authority checks. ## Verification Local controlled regression passed at tested revision `d8dece250b75cffee67d3b7f4f0e669826282164`: - Baseline: restore only the five original master production leaves. Keep the same read-only settings projection and regression file. One enabled-Dot case failed with the intended `57014`; it still read the correct `true` policy. Five cases were unselected. - The trace held the admission lock at 2.441 ms. The second connection waited for that lock at 9.494 ms. The root-pool feature read finished at 2015.585 ms, after the competing statement timed out at 2014.982 ms. - Candidate: all six enabled/disabled Dot, Grok and Muse controls passed, with zero skips, in 14.528 s. Each read returned the configured policy before the competing lock was acquired. Every competing transaction committed without an error. - Fresh normal candidate production declarations passed in 17.080 s. The complete 615-root owning test program passed in 4.756 s. Own dependency setup and TypeScript export builds had passed before these controls. - Known worker groups and sockets closed. The single post-owner audit found no other sessions in the test database. No forensic database was changed. An earlier selector attempt skipped all six cases and opened no test database connection. It is not a regression pass or failure. Source inspection proved that Vitest 5 filters names with ` > ` separators, while its JSON reporter joins names with spaces. The selected control then used the unique title-only pattern; its separate baseline failure is preserved above. The later normal master merge preserves all seven tested files. The 46 incoming paths change Native/workspace recovery, types and presentation. They do not change the direct feature/settings read graph. The direct-policy evidence therefore carries from its tested revision. It was **not rerun at the merged revision**. Fresh merged-head checks at `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a` passed: the content-aware Shared/SDK producer (1.325 s), normal Runner core and surfaces TypeScript programs (0.936 s and 0.693 s), and the full normal App graph (28.933 s). The App check included all production layers and all four test programs: 616 normal test roots and 397/409/412 tree roots. Known build owners and sockets closed. Database/Eval and provider asset outputs remained unchanged. No install, Cargo, binary build, provider execution or PG test repeated. Fresh final-head remote CI and review remain **pending**. The test entry is `server/src/modules/external-agents/feature-reader-owner.postgres.test.ts`. It requires a dedicated external `PAPERCLIP_FEATURE_OWNER_TEST_DATABASE_URL` ending in `_test` and an owned `PAPERCLIP_FEATURE_OWNER_TRACE` path. Its normal migration hook, two-connection pool, case deadlines and assertions remain unchanged. No provider or Runner binary is needed for these controls. The review also identified a wiring coverage gap: the six contention controls call the readers directly. A new whole-three file now traverses the actual Dot/Grok/Muse broker factories, normal pending-pairing producers, dispatch and real execution-admission transaction. Pass-through observations compare the exact admission transaction with the owner supplied to the normal readonly settings projection. Each provider is disabled through normal settings and must return real 403, read actual false policy, leave the queued run without Native authority, and create no assignment. No grant, ready binding, policy result or successful authorization is mocked. With only the original five master production leaves restored, all three new cases failed at the exact missing-owner assertion (`readOwners` expected length 1, actual 0) after passing the real 403 and admission-scope checks. The full baseline had0 pass / 3 fail / 0 skip in 12.691 s. With the exact candidate leaves restored, the unchanged whole-three file passed 3/3 with zero skips in 10.101 s. Both variants used actual Node 24 / Vitest 5 forks and a real PG pool of two; all known owners/groups/sockets closed. The candidate's private temporary directory was removed; the failed baseline database/home remain retained. Root independently read all three raw baseline failures, rehashed source and outputs, verified known owner/group closure, then observed no remaining test sessions. The successful database was normally removed; the original failing database and home remain retained. The new test entry is `server/src/modules/external-agents/broker-feature-owner.postgres.test.ts`, using dedicated `PAPERCLIP_BROKER_FEATURE_OWNER_TEST_DATABASE_URL`. The complete normal 617-root owning test program passed in 4.883 s after a normal exported config refresh; prior production declarations and unrelated configs remained byte-identical. No compiler or dependency/Rust work repeated for the PG variants. These results were obtained with the seven production/source files at c203 plus the new test; they are not claimed as a new published-head remote CI result. The review repair is committed at `d043860eb93a26c4416f184697a212f027d538e2`; it adds only this 117-line test file to `c2031b0f`. Tracked production bytes are unchanged. Exact published head `d043860eb93a26c4416f184697a212f027d538e2` now passes 56 checks with four expected skips. Greptile reviewed this exact head at 5/5 and confirms that the broker-test gap is addressed. The complete review thread is resolved. Ponytail stopped without a verdict; one normal retry was requested and no active turn or new finding was observed. No earlier-head result is used as a new-head pass. The controls prove feature policy and transaction ownership. They do not prove deployed suspension, hosted Dot/Muse eligibility or full Native task execution. The separate private useful-case pass does not supply missing current-master or accounting evidence. ## Risks The broker identity contract changes for all three providers. A missed adapter could use the wrong owner. Current semantic, real-policy and exact broker-owner identity tests cover that error. The read-only projection reuses the normal settings view. It preserves managed and staging rules. An absent row is projected without a write; the existing initializing getter stays unchanged. No schema, public API, queue, timer, rate limit or provider deadline changes. No historical Retry or physical-stop proof is changed. ## Model Used OpenAI `gpt-6.1-sol` assisted with source changes and tool use. An OpenAI GPT-6 family coordinator reviewed the work; its exact runtime ID was not exposed. Context-window sizes were unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (tested revision and source-carry limits stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing command or contract change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6d112c74cf |
fix(native): do not charge a restart resume that never started the provider (#19747)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - #19720 stopped charging a crash restart when the previous restart resume made progress > - In a crash loop (several kills within seconds), the controller can die again before the resumed execution starts its runner > - The next claim then charged that resume, although it used no provider work > - This pull request does not charge a resume that never started; the next claim replaces it > - The benefit is that fast restart bursts do not exhaust a healthy run's budget, and the generation cap still bounds real loops ## Linked Issues or Issue Description Refs #19720, #19625 **What happened?** In a restart-chaos storm wave, 8 runs still reached `execution_recovery_budget_exhausted`, mostly in K6 crash loops (three kills in about 20 seconds). Example: a Codex run was claimed at generation 9, attempt 3. The controller was killed 6 seconds later, before the executor recorded `native.process_start_requested`. The next claim found no progress after generation 9's position, charged, and blocked the run at attempt 3. The recorded history of the blocked runs shows that almost every charge after the first restart was for a resume with no start evidence (no runner request and no runner event) before the next claim. Two runs were blocked at generation 12, the existing cap. **Expected behavior** A resume that never started the provider consumed nothing and must not be charged. A resume that started and then made no progress must still be charged. The generation-12 cap stays. **Steps to reproduce** Start a native run. Kill the server during the run. Let a restart resume do work. Then kill the server repeatedly within seconds, before each new controller starts the runner. After one or two such bursts the run is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `native-restart-recovery.ts`: `restartResumeKeepsAttempt()` replaces `previousResumeMadeProgress()`. It reads the previous claim's recorded `progressWatermarkSeq` and `progressAttempt` (`lastRecoveryClaimResume()`). - New `nativeExecutionStartedSince()`: after the watermark, it looks for `native.process_start_requested`, `native.process_identity_recorded`, `adapter.invoke`, `tool.execution.started`, or any event with a runner `source_event_id`. This is the same start evidence as `hasNativeExecutionStartEvidence()`, limited to events after the watermark. - When the previous resume never started, the claim keeps the attempt if the run is already at the attempt that resume was to run as. If the executor never applied that charge (killed before its execution lease), the claim charges once, so the run reaches that attempt and never goes above it. - When the previous resume started, the #19720 progress rule applies unchanged. - The generation-12 cap applies to both rules. - `doc/execution-semantics.md` states the rule. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`): - `does not charge for a resume killed before it started the provider (after the executor took its lease)`: one progress resume, then four crash claims with no start evidence. The attempt stays 2. A resume that starts and fails then charges to 3. Red on master (charged to 3, then blocked). - `... (before the executor took its lease)`: the same with the lease never taken. Green on master and with the fix, as a guard. - `charges the owed attempt once when an unstarted first resume is replaced`: the replacement claim charges once (attempt 2). - `bounds a crash loop that never starts the provider`: blocks after the generation cap. Red on master (blocked at the third claim). - The #19720 tests now record `native.process_start_requested` in their resumes, as a real resume does. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts`, `native-remote-retry-admission.test.ts`: 231 passed, 14 skipped. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A crash that happens before the runner start, every time, gets free resumes until generation 12. Then the budget applies. This is the same bound as the progress rule. - Start evidence is the executor's durable `native.process_start_requested` record, written before any backend may spawn a runner. If an older path launched a provider without it, that resume would be treated as unstarted until the generation cap. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f673bd3cb5 |
fix(native): verify legacy harness filesystem independence (#19748)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain provider checkpoints and recovery state across turns. > - Server replacement must preserve every remaining local data dependency. > - A legacy backup stamp can outlive a destroyed provider and an absent local backup. > - Surviving skill files can already have exact immutable package copies. > - Raw controller files also contain stream events that database checkpoints omit. > - This pull request records verified filesystem independence under the original-volume maintenance owner. > - Replacement startup checks durable bindings while existing resume failures and policies remain intact. ## Linked Issues or Issue Description Refs #19715, #19729, #19733. **What happened?** Startup treats every retained legacy backup stamp without a current backup row as a local filesystem dependency. This also blocks sessions whose provider was already destroyed and whose remaining local skill bytes have exact durable copies. **Expected behavior** An authorized maintenance owner can prove that an exact retained source has no uncovered local harness payload. Startup can use that proof after replacement. The proof must preserve failed recovery state and must not invent a provider backup. **Steps to reproduce** 1. Retain a terminal native session, its checkpoint, and its legacy backup stamp. 2. Record exact canonical destroyed-provider receipts. Keep only admitted immutable skill files on the original host. 3. Publish the owner-verified independence receipt. The old startup predicate still blocks; the updated predicate validates its durable bindings. ## What Changed - Add an internal original-volume capability. It requires a private nonce anchor, independently reviewed attachment evidence, current reserved runtime identity, and a witness that is at most 120 seconds old. - Verify retained and selected-source inputs, checkpoints, leases, destroyed-provider receipts, released ownership, and exact immutable skill references. - Inspect canonical persistence roots and compare surviving skill bytes and modes. Refuse any provider directory present outside the current persistence profile, including an old Codex home after a preconfigured-home switch. Unknown data, backups, history, owners, or source changes still block publication. - Preserve raw controller and supervisor bytes through the existing encrypted evidence archive owner. Match recovery records to the database checkpoint, authenticate the exact archive row, and compare the original bytes again before publication. The archive grants no executable authority. - Publish one audited receipt with source locks and a final comparison. Keep original files, stamps, warnings, statuses, and session pointers. A late capture retains a valid published winner and refuses an invalid receipt. - Validate published receipts from current database bindings on replacement hosts. Add no startup filesystem probe, object read, provider call, or recurring worker. ## Verification - Genuine baseline regression: restore the original startup predicate while keeping the actual receipt owner. The positive PostgreSQL/filesystem case fails with expected `false`, actual `true` after valid publication. Restore the changed predicate; the case passes. - Final rebased head `e78f4b4424`: 42 affected PostgreSQL/filesystem cases pass. This includes 32 owner guards, three paired continuation cases, six existing failed-source archive controls, and one existing lease-verified backup migration control. The other 113 tests in the selected existing files were explicitly unselected. - Genuine controller baseline: the unchanged prior owner publishes no raw archive and accepts missing checkpoint parts. Both new regressions fail. The repair passes encrypted byte-for-byte controller/supervisor restore, missing checkpoint, foreign source, active source, and changed-after-upload controls. - Genuine omitted-provider baseline: the unchanged owner accepts old Codex, OpenCode, and ACPX history after a preconfigured-home switch. All three refusal cases fail. The fix passes those cases and permits preconfigured execution with no omitted directory. - Genuine late-publication baseline: both concurrent winner tests fail before the repair. The repair keeps the existing valid receipt, refuses an invalid one, and keeps one audit event. A delayed raw upload also retains the first authenticated archive and its receipt. - Capability controls: 12 passed, including empty replacement refusal, stale or future evidence, changed deployment, missing volume, path aliases, changed anchor, and current remount identity. - The actual server-to-Runner boundary preserves explicit recovery policy on original and empty homes. The separate actual CREATED-sandbox preparation callback preserves checkpoint identity and policy without a backup. Provider I/O and final transport construction are controlled. - Warm sessions retain the same canonical refusal on both homes, make no backend/provider attempt, and retain the saved checkpoints. - Server TypeScript, module boundary checks, diff checks, and a private-data scan pass. The Runner TypeScript build was refreshed after rebase to include two new upstream exports. - Fresh Greptile review is 5/5 with no unresolved threads on `e78f4b44240b0fc33c2fd7cae2d5b4a2578e3153`; check `128330103712` completed successfully. Required CI remains pending. No live preservation or enrollment action ran. Focused command from `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts src/services/native-runtime/native-controller-migration.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts --testNamePattern 'owner-verified|legacy harness continuation|preserves a failed source online|migrates a lease-verified' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks - This proves that no uncovered harness payload remains local. It does not recover provider history or certify that no other copy exists. Existing governed-wait and warm-session failures remain. - Unknown providers, retained owners, extra payloads, missing references, or changed source bindings fail closed. Invalid receipts require investigation; this change does not repair or replace them automatically. - The maintenance capability requires independently reviewed Railway attachment evidence. The original anchor remains stable across normal releases; each new staging invocation needs current deployment and mount evidence. - Startup validation adds bounded per-scope database reads. It pages candidate scopes and caps each source history at 500 rows. Measure the affected cohort's actual startup latency on the qualified serving build before enrollment. - Raw archives use the existing backup ledger, encryption, retention, and restore machinery. They preserve evidence only. They do not repair missing provider history or grant a new resume policy. - Local tests control provider and transport boundaries. Live staging remains a separate step after review, merge, normal delivery, and a fresh original-volume capability. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and local code/test execution. The exact runtime variant and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
96bc021edb |
fix(server): certify settled legacy processes for future retries (#19377)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Bounded retries must preserve current authority, budgets and deadlines. > - The tenant can wait for supported future work after the source process and its cleanup stop. > - Retained process identity alone cannot prove that cleanup finished. > - This change records host-owned settlement and checks its exact identity before a retry can cover its wait. > - Uncertain process state, changed authority, live claims and cleanup debt keep the tenant awake. ## Linked Issues or Issue Description Refs #19001. Related dependencies: #19019 and #18988. Related #19340 covers explicit Native task Retry; this change covers future Legacy process retries. Current integrated source: `1645970a9322d1213ba25fc7b52366c285682977`, normally merged with actual master `131b566ca4fa866c4348739c75cfe48c805b3398`. The prerequisites above are merged. This PR targets `master`. **What happened?** A future issue, company or review retry can retain its failed source process identity. The deadline reader could not prove that host cleanup finished. Supported waits stayed blocked. Ordinary issue retries also needed an exact source-settlement check, and represented coalesced requests needed exact input coverage. **Expected behavior** A supported future retry covers only its existing wait, under current company and actor authority, after exact host settlement. Missing, malformed, stale, cancelled, Native or unfinished source state stays blocked. No retry gains new execution authority. **Steps to reproduce** Fail an ordinary process or fail environment preparation before adapter entry. Let normal cleanup schedule the existing retry. Inspect its settlement and deadline, stop and reconstruct the owner, and admit one useful successor at the saved deadline. Hold cleanup or change authority to verify that the reader refuses coverage. ## What Changed - Add nullable internal `legacyProcessSettlement` state through normally generated migration `0439_cultured_impossible_man`. No backfill or adapter result creates the record. - Write exact host settlement after stopped-process and cleanup checks. Compare the current run and reject live claims or cleanup debt. Optional persistence failure preserves normal cleanup and keeps sleep coverage blocked. - Add a separate preparation receipt for a claimed Legacy run that failed before any adapter or Native dispatch. Require the same boot, run times, preparing stage, released claim and completed cleanup. A null PID alone proves nothing. - Clear settlement on a new claim, process metadata or Native selection. Omit the internal record from public serialized data. - Check exact source lineage and current authority for ordinary issue, company and taskless retries. Cover completed coalesced requests only when the admitted retry represents their exact inputs. - Add the closed `current_review_retry` wait with current reviewer, stage and existing issue authority checks. Require a nonempty current stage ID. - Add actual producer, cold reconstruction, SQL negative, redaction and optional-persistence controls. - Repair CI fixture expectations for the additive column, startup mock, budget owner and popup return URL. Bind the controlled Native comment server explicitly to loopback. - Replace temporary Slack clock logging with an exact persisted-deadline assertion. Wait until that real deadline before the existing single default poll. Keep both forced polls, business assertions and timing limits. ## Verification - On final integrated source `1645970a`, normal Shared/SDK prerequisites, Runner TypeScript and the entire normal App layered typecheck pass. Tracked source and lockfile remain unchanged. - All 26 whole-file settlement, recovery-reader and bootstrap controls pass. Seven separate real PostgreSQL producer/negative cases pass at `cf16c6a`; each has one selected passing case and 18 filtered registrations. Filtered registrations are not acceptance. - Master then changed the preparation source-lock scope. Its three intersecting cases were rerun on final `1645970a`: an actual failed company process, an actual failed ordinary issue process, and held preparation cleanup. Each preserves its source identity and original deadline and admits one useful successor. Issue and preparation cases exercise normal automatic retry; the company case uses the normal explicit bounded-retry producer. - The unchanged 26 controls and four other PostgreSQL cases retain their `cf16c6a` proof after a scoped source comparison. Optional settlement/preparation persistence failures keep certification blocked while normal cleanup completes. An entered adapter without process identity stays uncertified. This is scoped carry, not a claim that every earlier case ran again on the final head. - All recorded test owners and controlled groups closed. Final read-only SQL audits returned no test sessions. One earlier postcase resource barrier observed a transient unattributed database backend; that database and original evidence remain preserved. Its later disappearance is recovery, not an explained cause. No assertion, deadline or test body was relaxed. - Migration0439 was generated normally after master history. All executable SQL and journal history are preserved; the five retained snapshots are0435–0439. Every Legacy-owned patch and unrelated actual-master blob/mode is preserved. - Published exact head `1645970a9322d1213ba25fc7b52366c285682977` passes all current required checks: 68 successful contexts and eight expected skips after exhaustive pagination and superseded-run accounting. The fresh complete Greptile review scores this exact head 5/5. All four inline review threads are resolved. The latest Ponytail turn stopped without a verdict; one normal retry was requested, with no active turn or new finding observed. No old-head score is used. The original old-base policy failure, its dependent aggregate failure, and the corrected description-gate failure remain recorded. No full local repository suite is claimed; remote typecheck, test and build gates pass. To repeat the real database coverage, set both `PAPERCLIP_RECOVERY_DEADLINES_TEST_DATABASE_URL` and `PAPERCLIP_RECOVERY_DEADLINES_TEST_ADMIN_DATABASE_URL` to the same fresh dedicated test database and run the selected `recovery-deadline-acceptance.test.ts` cases with one worker. An unset URL, skipped case or filtered registration is not acceptance. The fixtures use controlled local processes and real PostgreSQL. Reconstruction creates new service owners inside the same test process. They do not prove deployed Cloud alarms, physical host/database suspension, new-host recovery or Native capacity settlement. The original30-second retry remains below the Cloud270-second minimum wake lead. The merge projection also includes actual master `c3180398faca815fed7a62674bd4b0c4e88884ae`, whose common completion path now sends deferred/recovery notifications. Its unpublished normal merge tree is `16b028d57150f8caa576e913f38343a365b542dd`. Normal Shared/SDK, Runner TypeScript and the full App graph pass on that combined source. One unchanged ordinary-issue acceptance case was rerun there: an actual failed process retains the original automatic retry deadline, admits one useful successor, and passes the settlement, authority and no-duplicate assertions. It passed in 45.055 s; 18 filtered registrations are not acceptance. All known owners and groups closed, and the independent final database audit found zero remaining test sessions. The successful database was removed normally. Recorded worker socket peak is unavailable for this run; parent-process zero values are not used as worker measurements. All 29 Legacy-owned paths and every unrelated master blob and mode are preserved. Two outside-diff review observations concern existing waiting-chat compaction and cross-host Native result-lease recovery. Their files are byte-identical to current master. This focused Legacy change does not fix or qualify those separate paths. ## Risks Settlement proves exact local closure. It does not cover live provider work, Native ownership, cancellation, costs, claims, cleanup debt or changed authority. Independent checks remain. Ship normal migration0439 with the reader and writer. Preserve executable history and the newest five snapshots0435–0439. Generate the MCP child's next migration normally after this parent. The prerequisites are merged; the final remote checks must use this integrated exact head. This proof uses controlled local processes and external PostgreSQL. It does not establish deployed Cloud alarm acceptance, physical host or database suspension, cold host recovery, or R5 Native capacity settlement. The30-second retry is below the Cloud minimum wake lead. This focused repair follows the roadmap's bounded recovery and Cloud deployment work. It adds no new provider policy. ## Model Used OpenAI `gpt-6.1-sol` assisted with readers, fixtures, source review and verification through Codex. A coordinating GPT-6-family agent implemented and reviewed shared lifecycle wiring and integration. Tools included Git, native TypeScript checks and controlled tests. Exact served revisions and context limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and its available capability details - [x] I have checked ROADMAP.md and related work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public work and described the issue - [x] I have not included internal issue IDs or private links - [x] My branch name describes the change - [x] Required focused checks pass at their explicitly recorded source/carry scopes above - [x] I have added or updated meaningful tests - [x] I have recorded the internal contract and migration requirements - [x] I have documented risks and evidence limits - [x] All required published exact-head Paperclip CI gates are green - [x] Published exact-head Greptile is 5/5 with no open findings in this focused diff; separate unchanged master observations are identified above - [x] I will address reviewer findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
c3180398fa |
fix(native): resume a remote Runner after a settled suspension
fix(native): resume a remote Runner after a settled suspension A scheduled checkpoint retry can retain a remote reattach claim after closing its failed predecessor cleanly suspends and exits the Runner. Resume only that exact settled source after confirmed exit and a fresh identity/state check. Preserve pending work and uncertain-exit fences. Admit the last already-charged provider attempt and share one bounded recovery deadline with the final state read, retaining closed diagnostic labels on read failures. Cover fresh ACPX Runner relaunch on attempts two and three through actual processes and the durable close barrier. Six affected suites pass 1,579 tests, with 14 environment-gated skips; all server typecheck layers pass on Node 25.6.1. The complete provider turn continuation failure remains preserved as follow-up #19751. Refs: #19366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
bf45f66619 |
docs(runner): separate the legacy-adapter paragraph in the Runner README (#19753)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner package has a README that explains its contracts > - One section explains how a resumed run gets a fresh-conversation handoff > - The next text, about legacy adapters, has no blank line before it > - Markdown renders the two texts as one paragraph, so the legacy-adapter text looks like part of the native handoff text > - This pull request adds the missing blank line > - The benefit is a README that separates native and legacy behavior clearly ## Linked Issues or Issue Description **Issue type** Formatting error in package documentation. **Where is the issue?** `packages/paperclip-runner/README.md`, the paragraph that starts "Legacy adapters advertise `supportsToolRefreshOnResume`". **What's wrong?** The paragraph has no blank line before it. Markdown joins it to the previous paragraph about the native fresh-conversation handoff. **Suggested fix** Add a blank line before "Legacy adapters advertise". ## What Changed - Add one blank line in `packages/paperclip-runner/README.md` before the legacy-adapter paragraph. ## Verification - `git diff --stat`: one file, one line added. - I read the changed section. The legacy-adapter text now starts its own paragraph. - No code changed, so no tests are necessary. ## Risks - Low risk. This is a documentation-only change. - The Runner build-input digest includes every file in `packages/paperclip-runner`. This change therefore changes that digest, and the next release builds a new Runner image. The Runner behavior does not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
08b049f381 | Admit qualified Claude sources through the existing Core claim (#19196) canary/v2026.1107.0-canary.7 | ||
|
DottaandClaude Opus 5.5
|
e620ce3678 |
fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). **What happened?** Restart-chaos waves found these shapes: 1. **Comment wake cancelled.** A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. **Dead Retry.** A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. **Refused assignment.** A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. **Silent comment after recovery.** A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. **Comment delivered twice.** A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. **Expected behavior** An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
18b6b71d83 |
fix(native): refresh per-turn projections from the selected checkpoint (#19744)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Per-turn native sessions save controller checkpoints and suspended Runner backups under separate owners. > - A local controller projection can still name the first turn after the next turn has saved its checkpoint and backup. > - The old per-turn restore path selected that stale local reference instead of the newer canonical source. > - Admission then saw different turn identities and archived the valid backup with the old projection. > - This change uses the canonical selected checkpoint and the existing finished-owner proof before it replaces a stale projection. > - The benefit is that a third turn keeps the valid saved conversation and backup. ## Linked Issues or Issue Description **What happened?** After two successful per-turn remote runs, a third turn could fail before provider execution with unavailable provider resume state. The selected database checkpoint and suspended backup belonged to the second run. The local controller projection still belonged to the first run. **Expected behavior** The selected checkpoint must govern controller rehydration. Preserve the older projection and the selected run's backup. Do not overwrite current, newer, foreign, changed or unsettled ownership evidence. **Steps to reproduce** 1. Save two real controller checkpoints for one per-turn conversation and finish both owners. 2. Leave the first checkpoint's readable projection or external marker beside the second run's suspended backup. 3. Admit the third run through the real canonical selector and backend owner. 4. Before this change, admission archives the active backup with the stale projection and leaves no active controller state. Related: #18018 added per-turn controller restoration. #18718 refreshed warm projections. #18846 added finished-owner retirement. This change keeps the selected per-turn source from entering that retirement path because of an older projection. The open warm-only change #18436 concerns a different restart branch. ## What Changed - Resolve the canonical source before refreshing a prior per-turn controller projection. - Re-read it under the existing sorted coordinator locks. Replacing an older owner reuses finished-owner checks for both runs and verifies their full session scope. A same-source marker only rehydrates its exact selected identity before the normal retained-sandbox verification. - Recheck the exact initial local bytes after asynchronous reads and lock waits. Preserve the old control-plane directory in the existing archive, while keeping the newer backup and provider files active. - Restore a current-run external marker only from its own checkpoint before the existing bootstrap retry proof. A missing current checkpoint cannot select a previous turn. Retain readable current-run, legacy file-backed, backup integrity and provider resume checks. - Stage the existing OpenCode proxy bytes into the controlled fake-sandbox filesystem in the real ingress-retry fixture. Keep remote artifact path and digest checks unchanged. - Add real PostgreSQL/filesystem coverage for Codex and ACPX, readable projections and external markers, missing or corrupt parts, held owners, foreign scope, newer history and a current owner arriving during a real coordinator-lock wait. ## Verification - Baseline RED: the two new readable-projection cases failed against unchanged production code because active control-plane state was absent after admission. The real selector selected the second run. Setup failures were separate and are not counted as RED. - Exact final existing per-turn restore, active-owner refresh and retained-owner wait controls: 19/19 passed in 5.91s. - Exact final local source (`fd8d9d9021`, based on `131b566ca4`): 58/58 real PostgreSQL/filesystem and native Runner tests passed in 31.58s. This includes both existing ingress bootstrap retries through the real Runner and fake Codex/OpenCode providers, plus 56 focused owner cases. Cargo validated and rebuilt current source from a private APFS clone of an unqualified local cache; the original cache was untouched. A private local configuration resolves workspace DB, Shared and Runner sources without replacing behavior. The local CLI allowed a 90s setup hook; no default timeout changed. - Exact final full server and test-tree TypeScript check with Runner source paths: passed. The stock fresh-worktree check first failed on unbuilt Runner declaration entries; no product types were changed to hide that setup gap. - Exact final module-boundary and diff-whitespace checks: passed. - Native-server CI RED on the previous head: both real ingress retry cases failed because a current-run marker was returned without hydration. The aggregate gate failed from that one underlying lane. The correction preserves the own-checkpoint bootstrap path. The first local OpenCode attempt hit a controller-host fixture path guard before its intended failure; staging the same proxy bytes into the fake sandbox made both native cases pass. - Review regression RED: the real retained-sandbox fixture failed at `sandbox_resumed` on the first PR head. GREEN: the corrected owner hydrates the same-source marker and reaches the existing exact retained-lease/suspended-runner read. A mismatched item identity still refuses before any remote read. - Local environment: macOS arm64, Node 26.4.0, PostgreSQL 18.1. Provider execution is deliberately outside this admission fixture. The real three-turn workflow still needs verification on the normally deployed fixed source. ## Risks - Refresh requires settled, committed, released and stopped or fenced owners. Missing or changed evidence remains blocked. - The refresh holds existing coordinator locks while it validates bounded controller state. It performs no provider calls. - The old control-plane bytes remain in the archive. A restore write failure still fails admission; it does not reset provider state. - Final-head full CI is green, including typecheck, build, Rust, native-server, server and browser suites; Greptile is 5/5 with no unresolved threads. Hosted deployment and continuity verification remain separate acceptance steps. A successful replacement conversation is not proof of provider continuity; hosted continuity acceptance is still pending. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The host does not expose the exact backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
24cbea136f |
fix(native): export OpenCode launch artifacts through a confined workspace copy (#19743)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native recovery saves the qualified OpenCode launch files so a cold allocation can resume the same session (#19622) > - Core reads those files from the sandbox with the provider's bulk `syncOut` > - The launch root is under the sandbox account home, not under the workspace remote dir > - Daytona, Kubernetes, and CreateOS confine every export source to the workspace remote dir, so the export always fails > - Every remote OpenCode run on a durable-storage instance now fails 40-90 s after start with `native_session_interrupted` > - This pull request exports a confined scratch copy of the three launch files from inside the workspace runtime dir > - The benefit is that remote OpenCode runs work again, and the provider confinement guard stays unchanged ## Linked Issues or Issue Description Refs #19622 **What happened?** After #19622, every remote OpenCode run on Daytona fails at provider admission. The run error is `Daytona sync source path escapes the workspace remote dir: /home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/<digest>`. The error code is `native_session_interrupted`. The last successful run of the affected agent was immediately before #19622 reached the deployment. **Expected behavior** Core saves the qualified launch files and the run continues. The provider confinement guard still refuses any export source outside the workspace remote dir. **Steps to reproduce** 1. Configure an OpenCode agent on a Daytona environment with durable harness storage. 2. Start a run. 3. The run fails when Core captures the provider artifacts, before the first turn. ## What Changed - `captureRemoteNativeProviderArtifacts` now takes the workspace `remoteCwd`. - Before the export, a sandbox command copies the three launch files into a new scratch directory at `<remoteCwd>/.paperclip-runtime/paperclip-runner/provider-artifact-export/<uuid>/`. The command refuses a symlink or a non-regular file. It uses `cp -P`, so it never follows a link. - `syncOut` reads from that scratch copy. A `finally` step removes the scratch copy after success or failure. A failed removal is logged and does not replace the transfer result. - The launch root does not move. OpenCode still binds to the same absolute paths. Core still verifies each downloaded byte against the manifest SHA-256 values. - `.paperclip-runtime` is already excluded from workspace sync, so the scratch copy never enters the user workspace. - The Daytona `assertConfinedSandboxPath` guard is not changed. ## Verification - New regression `server/src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts`. It runs the real sandbox commands on a local directory tree that has a home-rooted launch root and a separate workspace. Its `syncOut` applies the same workspace confinement rule as the Daytona provider. - Before the fix: it fails with `Daytona sync source path escapes the workspace remote dir: .../home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/dfe9e4b1d660/bin/nodeCommand`. This is the staging error. - After the fix: it passes. The published bytes match, every export source is under the workspace runtime dir, and the scratch directory is empty after the capture. - Safety cases: a failed transfer removes the scratch copy and does not publish. A launch file replaced by a symlink is refused before any export, and nothing is published. - `native-session-executor.test.ts`: the durable OpenCode admission case now asserts that the capture receives `remoteCwd` and that every `syncOut` source is under `/workspace/`. - `pnpm vitest run` for the capture, confinement, remote OpenCode artifact, and session root entries tests: 14 passed. - `vitest run native-session-executor.test.ts -t "opencode|provider artifact|durable"`: 62 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Not run: a live Daytona run with this revision. The fix is not deployed to staging. ## Risks - Low risk. The capture now runs two short `sh` commands in the sandbox and holds a second copy of the launch files during the transfer. The files are bounded by the existing 512 MiB limit and are deleted afterwards. - The scratch copy uses `cp -P`, `mkdir`, and `rm -rf` from POSIX `sh`. The Daytona image already uses these tools for the workspace sync. - If the sandbox stops between the copy and the removal, a scratch directory can remain under `.paperclip-runtime`. Workspace sync excludes that directory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9f0ebf6f70 |
fix(accounting): close a run whose turn was lost on restore, and stop succeeded runs holding all work (#19724)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets hold new work while the cost of completed runs is unknown > - In a restart chaos wave, a native ACPX Claude run crossed a server kill, resumed, and succeeded. Its usage receipt never became complete. > - For the 10-minute closure bound, admission refused all work in the company. This included tasks that never crossed a fault. > - This pull request fixes the receipt for a turn that was lost on restore. It also stops a succeeded run from holding all new work. > - The benefit: one late usage report no longer freezes a company, and hard stops still apply. ## Linked Issues or Issue Description Refs #19700 (the accounting closure bound that this follows). **What happened?** A native ACPX Claude run crossed a server SIGKILL in the middle of a turn. The Runner restored the session, and the run succeeded 54 seconds after it started. All 17 usage receipts stayed incomplete. Accounting settled the run only at the 10-minute closure bound. During those 10 minutes, admission refused all company work with "Company cannot start work while completed runs await accounting". **Root cause** ACPX reports provisional usage (`runDeltaComplete: false`) until a turn ends. The kill lost the turn that was active, so that turn never got a complete report. The restored Runner failed that turn with `provider_turn_lost_on_restore`. Native turn accounting (`createNativeTurnAccounting`) required every terminal turn to have usage, so this turn kept every later receipt incomplete. This was true even after the continuation turn completed with full usage. The admission gate then counted the succeeded run as a pending terminal run for the whole company. **Expected behavior** The receipt closes when the run's turns end. A turn that was lost on restore makes the usage a lower bound. A succeeded run with only usage outstanding does not hold other work. ## What Changed - `native-usage-accounting.ts`: a turn that failed with `provider_turn_lost_on_restore` and has no complete usage no longer holds the receipt open. The snapshot marks `accountingUsageLost`. - `native-session-executor.ts` and `native-usage-delivery.ts`: such a receipt carries the provenance note "a provider turn was lost on restore and never reported its usage; observed usage is a lower bound". - `anthropic-pricing.ts` and `codex-pricing.ts`: the rate-card pricers keep a producer's provenance note. - `native-usage-delivery.ts` and `usage-receipts.ts`: remote producer delivery keeps a `lostOnRestore` marker on the receipt's producer event, so replay also excuses the lost turn. Receipts saved without the marker still replay. - `budget-exposure.ts` and `budgets.ts`: admission waits only for pending runs that ended without succeeding (`waitingRunCount`). A succeeded run whose provider reported observed usage is bounded like a running run: its observed cost counts toward spend and reservation capacity, and its settled charge can still reach the hard stop. - `budget-exposure.ts`: a closure that timed out because of a control-plane restart keeps the unpriced-usage exemptions, whatever its token count. Restart evidence is a replaced receipt source, a native recovery transition that is not an ordinary same-run retry, or a turn lost on restore. A second chaos run showed the need. It recovered and succeeded, but its final usage report was lost with the restart, so it settled as unpriced 0 cents. The blocking policy then opened a hard incident and paused the company. A timed-out closure with no restart evidence still counts. - `run-cost-accounting.ts`: a lost receipt that carries a positive provider-reported cost settles at that cost as an estimate marked as a lower bound, not as unpriced. - `paperclip-runner-tool-authority.ts` and `budgets.ts`: a running run's own tool mutations are evaluated as admitted work. Hard stops and pauses still refuse them. Accounting holds, recovering runs and budget inputs that await evaluation do not. In a third chaos run, a live run's write to its task document was refused during a temporary hold, so the run proposed a blocked result. - `doc/SPEC-implementation.md`: documents these rules. Policy choice: a succeeded run's provider work is over, so it has less unknown spend than a running run. Running runs never held admission. Failed, cancelled, timed-out and interrupted runs still hold new work, because their provider may not have stopped. A per-agent gate would still freeze unrelated work of the same agent. A per-reservation gate does not help when `reservationCents` is 0, which was the case in this wave. ## Verification - Live: a restart chaos wave ran master plus this PR (c2a48fb1e0) with no masking. It passed 6/6, and every task crossed a restart. There were 0 accounting holds, 0 refused agent writes, 0 company pauses, and every cost settled as estimated. - Red then green: `native-usage-accounting.test.ts`, "closes a run whose earlier turn was lost on restore, as a lower bound". It uses the event sequence of the chaos run. - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for a succeeded run with only usage outstanding". It also checks that the hard stop still applies. - I changed "neither pauses nor holds admission while a succeeded run's charge commits" to the new policy. A failed run in the same state still holds new work. - Red then green: `closed-native-receipt-accounting.test.ts`, "never hard-stops a blocking policy for a restart-derived timed-out receipt". It uses the zero-token row shape of the chaos run and a case with unpriced tokens. Without the fix, the company was paused. - Red then green: `runner-api.integration.test.ts`, "lets a running run write during a temporary accounting hold, but not after a hard stop". - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for the s7504 succeeded row shape". - Red then green: `closed-native-receipt-accounting.test.ts`, "settles a timed-out restart receipt at its provider-reported cost, without holding or pausing". It uses the OpenCode/OpenRouter row shape from a fourth chaos run. - `anthropic-pricing.test.ts`: the rate-card provenance keeps the lower-bound note. - Local: `closed-native-receipt-accounting`, the budget suites, `native-usage-accounting`, `native-usage-delivery.integration`, `native-runner-restart-recovery.integration` and `anthropic-pricing` all pass (187 tests). Server `tsc --noEmit` is clean. `pnpm verify` asks for CI coverage because the spec changed. ## Risks - A succeeded run whose final charge is higher than its observed cost can let one more task start before the charge commits. The hard stop then pauses the scope as it does for a running run that overruns. - A run that includes a lost turn settles with a lower-bound usage figure. The provenance says so, and a later receipt still reconciles through the late-receipt path. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4dc5281f51 |
fix(ui): offer the owner-stop confirmation for retained saved-work recovery (#19742)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run saves the agent's files after its turn (copyback). One controller owns copyback and records an owner receipt on the run > - #19721 made retained saved-work recovery stop spending attempts when a restart killed that owner. It records the receipt as `workspaceOwnerRecovery.state = "operator_required"`, and a waiting Retry reports `workspace_owner_stop_required` > - #19717 added one task control, **Confirm the previous server stopped**, for the direct finalization hold. That control did not cover the retained hold, so the task told the board to confirm but offered nothing to click > - This pull request shows the same control for the retained hold, with the recorded receipt token > - The benefit is that every copyback-owner hold has exactly one visible action, and the waiting Retry then continues by itself ## Linked Issues or Issue Description Refs #19721 (retained hold, server side), #19717 (the confirmation control). **What happened?** The retained recovery action named the stopped copyback owner, and the task's Retry said "Retry is waiting for a board member to confirm that the previous server stopped copying files". No control called the confirmation route for this state. **Steps to reproduce** 1. Retained saved-work recovery owns copyback for a finished run. 2. A restart kills the controller while it owns copyback, on another host or without automatic stop proof. 3. Open the task. **Expected behavior** The task shows one **Confirm the previous server stopped** action to an owner or administrator. It sends the recorded receipt token to `resume-workspace-finalization`. **Actual behavior** The task shows only the retained file-recovery controls. The token is in `evidence.workspaceOwnerRecovery.owner`, which the #19717 control did not read. ## What Changed - `WorkspaceExportRecovery`: when a retained export action has `workspaceOwnerRecovery.state = "operator_required"` and no recorded confirmation, render the existing confirmation control with that receipt's token, in place of the retained file-recovery controls. After the confirmation (state `resuming`), the ordinary retained controls return. - The confirmation control now takes the owner token as a prop, so the direct hold (`evidence.owner`) and the retained hold (`evidence.workspaceOwnerRecovery.owner`) share it. - Docs: one sentence in `doc/native-workspace-finalization-recovery.md`. - There are no server changes. The route already dispatches the retained path (#19721). ## Verification - Red then green: the new test "offers the same action for retained saved-work recovery held by a stopped copyback owner" fails against master's component and passes with this change. It checks that the request body carries the retained receipt token, that the host name is not shown, and that the retained controls return after `resuming`. - `cd ui && npx vitest run src/components/WorkspaceExportRecovery` (35 passed). - `pnpm check:token-gates` clean. UI typecheck clean. ## Risks - Low. The change is UI only. The server rechecks the receipt, the hold state and `runtime:confirm_owner_stop`. - While the hold is active, the "Recover saved files" control is hidden. It cannot release the receipt, so hiding it removes a futile action. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
131b566ca4 |
fix(workspaces): report safe Git restore diagnostics (#19736)
Return the restore stage and failure category with the existing HTTP409 rejection. Preserve the unsaved comment and editable draft. Keep nested saved-commit and clone-timeout diagnostics through preparation wrappers, and distinguish local Git file errors from authentication failures. Retain only safe process flags privately; no raw command cause is exposed.
All seven affected local suites pass (288 tests) and the layered server typecheck passes on supported Node25.6.1. Reviewed head
canary/v2026.1107.0-canary.6
|
||
|
DottaandClaude Opus 5.5
|
464e6e16e0 |
fix(native): resume copyback after a restart kills its same-host owner; one board action otherwise (#19717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save the agent's workspace files after the provider turn finishes (copyback) > - One controller owns copyback at a time. The run records that owner in `runnerProfileJson.nativeWorkspaceFinalizationOwner` (host, PID, token) > - A control-plane restart can kill the controller during copyback. The dead controller can never confirm its own stop > - Recovery then waited for a board confirmation. No surface offered that confirmation, so the run stayed `running` and the task stayed `in_progress` with no visible action > - This pull request lets the next controller on the same host prove the stop by itself. It also gives the board one clear action when that proof is not possible > - The benefit is that a restart during copyback resumes by itself, and a task never waits silently for a person ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave, a kill during workspace copyback left the run `running` and its task `in_progress`. Recovery reported `native_workspace_finalization_owner_unverified`. The run was still `running` when the wave timed out. **Steps to reproduce** 1. Start a native remote (sandbox) run. 2. Kill the control-plane process (SIGKILL) after the provider result is accepted, during copyback. 3. Restart the control plane on the same host. **Expected behavior** If the previous controller and its copyback children are proven stopped on this host, recovery resumes copyback without a new agent turn. Otherwise, the task shows one board action to confirm that the previous server stopped. **Actual behavior** The recorded owner was cleared only by `resumeNativeWorkspaceAfterOwnerStop`. Same-host recovery also required a completed-copyback barrier, which an interrupted copyback never has. The task presentation showed `working`, and the UI had no control for the confirmation. Refs #19578 (boot-ID fencing for unstarted runs; same proof style). ## What Changed - Root cause: `server/src/services/native-runtime/native-workspace-finalization-ownership.ts` sent a dead same-host owner to `requireStopVerification` when no `workspace_finalize` operation had succeeded. That case is exactly a restart during copyback. - New `native-workspace-owner-stop-proof.ts`. The owner receipt now records the controller's process group (`processGroupId`). Copyback children (tar, Git) are not detached, so they keep that group. A successor on the same host releases the receipt only when the receipt names another controller boot, the PID is dead or recycled (process start time), and the group is empty or has no member outside the successor's own process tree that started after the old controller. Age alone never proves a stop. - A still-live group member returns `local_stop_pending`. The finalization worker rechecks it on its existing ownership recheck interval. The board action stays visible meanwhile. - A successful local proof writes `issue.workspace_owner_stop_verified` with the proof and marks the recovery `resuming`, like the existing platform proof. - A receipt from another host, a receipt without a process group, or a host without `ps` keeps the fence. The task interruption is now `waiting_for_user` with `userAction: "confirm_owner_stop"` (new shared enum value), not `working`. - UI: the task chat shows one **Confirm the previous server stopped** action. It asks for stop evidence and calls the existing `resume-workspace-finalization` API. It is shown only to roles that hold `runtime:confirm_owner_stop` (company owner or administrator, local board, instance administrator). The inbox names the same action. - The copyback advisory lock, the coordinator finalization lease, and remote save fencing are unchanged. - Docs: `doc/native-workspace-finalization-recovery.md`. ## Verification - Red then green on embedded Postgres (`server/src/services/native-runtime/native-workspace-finalizer.test.ts`): - same-host controller killed with SIGKILL, its process group empty: before `owner_unverified`, after `acquired` with the proof logged and no agent wake - a non-detached orphan child of the dead controller still runs: `local_stop_pending`, no copyback; after the child exits, `acquired` - a receipt without a process group: still `owner_unverified` - a foreign-host owner: the task presentation was `working` (red), now `waiting_for_user` / `confirm_owner_stop`; after the board confirmation it is `working` again - The existing orphan test (a detached child) and all other finalizer tests pass: `npx vitest run src/services/native-runtime/native-workspace-finalizer.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/task-interruption-presentation.test.ts` (135 passed). - UI: `npx vitest run src/components/WorkspaceExportRecovery` (34 passed), including the action call body and the non-administrator case. - `pnpm check:token-gates` clean. Server and UI typecheck clean for the changed files. - Live Daytona restart-chaos wave: results are added in a comment below. ## Risks - A process-table check that reports a member wrongly as live only keeps the existing fence. The board action is still available. - A subreaper that adopts orphans does not change their process group, so the proof does not depend on the parent PID. - A child that a future change spawns with `detached: true` would leave the group. Copyback paths do not do that today. The existing detached-orphan test still fails closed, because a receipt without a process group is not released. - Old receipts have no process group. They keep the board action. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
7a72caeda1 |
perf(workspaces): batch online companion discovery (#19735)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration checks all local aliases before it captures a source. > - Each alias must retain its prepared runs and recovery holds in that check. > - The current reader scans the same company run table once for every alias. > - Shared Git sources repeat that scan even after the complete alias group is known. > - This pull request reads those narrow companion rows once per inspection. > - The existing owners still validate every source and recovery hold before migration can proceed. ## Linked Issues or Issue Description **What happened?** Online source qualification repeats the prepared and finalized run query for each workspace alias. A shared Git source with four aliases makes four copies of this query. **Expected behavior** One inspection should read the complete alias group once. It must return the same source and recovery proofs, and a later inspection must read fresh rows. **Steps to reproduce** 1. Create a shared Git source with four aliases and a valid settled prepared run. 2. Add a finalized companion and a recovery hold. Add a foreign-company run that names the same workspace ID. 3. Inspect the source and count companion discovery queries. Before this change, the regression sees four queries instead of one. Related public work: [source ownership guards in #19688](https://github.com/paperclipai/paperclip/pull/19688) and [bounded accounting closure in #19700](https://github.com/paperclipai/paperclip/pull/19700). ## What Changed - Read prepared and finalized companion rows in one company-scoped query after the complete alias group and foreign aliases pass validation. - Group the narrow rows by workspace ID. Keep run order and all existing source and recovery owner checks. - Add a PostgreSQL regression for proof equivalence, one scan, company isolation, and a new source obligation between inspections. - Align an existing Stop-accounting test with the durable closure deadline from #19700. Keep the incomplete debt, quiet database, cancellation, and single-charge checks. Product workers and timers are unchanged. ## Verification - Before the source change, the new PostgreSQL test passed its proof comparisons and failed because four companion scans ran instead of one. - The two workspace suites pass: 77 tests in `remote-workspace-online-source.test.ts` and `online-legacy-workspace-recovery.test.ts`. - CI exposed an old Stop-accounting assertion that expected no timer while an incomplete receipt had an intentional closure deadline. Both automatic CI attempts failed. The unchanged local variants passed, so that run did not reproduce the failure. Waiting for the real owner deadline then proved the stale assertion red. - After the test alignment, both Stop-accounting variants and all 77 workspace tests passed together: 79 tests across three files. The three unrelated Stop-recovery cases were not selected locally. Fresh CI covers the complete suites. - The affected suites cover retained recovery owners, foreign aliases, physical source locks, and changes before publication. - Tests use a disposable PostgreSQL 18.1 database. The local hook allowance is 90 seconds. Product timeout settings are unchanged. - Final head `0eed2e5062cbfb62df7f76168c27d2172fc2d7de`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42341640259) passed, with 56 successful checks and four configured skips. [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/19735#issuecomment-6702476739) is 5/5 with no open threads. This PR makes no claim about deployed latency. ## Risks - Each inspection now holds the narrow companion rows for the complete alias group in memory. The existing alias group bound still applies. - The query keeps the same company, workspace, and sync-state predicates. There is no runtime filter, row limit, schema change, or cache between inspections. - No public command or API changes. Public documentation does not need an update. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Used reasoning, code editing, shell tools, and test execution. The context window size is not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
DottaandClaude Opus 5.5
|
c97c0dd248 |
fix(workspaces): give run preparation its own source lock scope (#19734)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run prepares its task workspace under per-source advisory locks (`workspace-source-lock.ts`). The locks live in a transaction on a dedicated connection. An `AsyncLocalStorage` scope lets nested calls reuse that transaction. > - Timers and detached promises keep the async context that created them. A native recovery timer or dispatch can start a run from inside another operation's source scope. > - When that run prepares its workspace, it sees the other scope as "held". If that scope already committed, its dedicated connection is closed. The lock query then fails with `Failed query: select pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that scope is still open, the run's lock is tied to the other transaction and is released early. > - In a restart chaos wave, a reattached follow-up run failed setup this way 50 ms after "Recovering the existing native runner process". The setup-failure log did not show the driver cause. > - This pull request makes only a live scope count as held, and makes run workspace preparation take its locks in an independent scope. > - The benefit is that a recovered or dispatched run gets its own source lock, and a lost-connection failure is diagnosable from the log. ## Linked Issues or Issue Description **What happened?** After a control-plane restart, a native run was reattached (`reattach_remote_runner`). Its workspace preparation failed at once with `Failed query: select pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params: isolated_repository_source:<company>:<issue>`. The run ended `failed`/`setup_failed`. The stack showed the "already held scope" branch of `withWorkspaceSourceLocks`, not a new acquisition. The same failure occurred earlier in the same wave on a different issue without a restart. **Steps to reproduce** 1. Call `withIsolatedRepositorySourceLock` for issue A. In its action, start a detached promise that later calls `withIsolatedRepositorySourceLock` for issue B. 2. Let the outer action return so its transaction commits. 3. Let the detached call run. It fails with `Failed query ... pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`. **Expected behavior** Work that starts after a source scope ends, or independent work such as a run's preparation, acquires its own locks on its own connection. **Environment** Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1, comment trigger), embedded Postgres. Reproduced locally on embedded Postgres by the new tests. Related: #14910 (deferred comment wake reopens a closed issue). This PR does not change wake promotion. ## What Changed - `server/src/services/workspace-source-lock.ts`: a scope has an `active` flag. The flag is cleared when its transaction ends. `withWorkspaceSourceLocks`, `retainManagedProjectSource`, `retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld` use only a live scope. - `server/src/services/workspace-source-lock.ts`: new `withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope and ignores an inherited one. - `server/src/services/heartbeat/workspace-preparation.ts`: run workspace preparation uses `withIndependentWorkspaceSourceLocks`. - `server/src/services/heartbeat/run-completion.ts`: the "heartbeat execution setup failed" log records a bounded driver `causeCode` (for example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause text. - Tests: two embedded-Postgres tests in `isolated-remote-workspace-source-lock.test.ts`, one unit test in `run-completion.test.ts`. ## Verification - `cd server && npx vitest run src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed. - Red (old scope rule): "gives work detached from a finished scope its own source lock" failed with `Failed query: select pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the production error. - Red: "keeps an independent preparation's lock after the scope that started it commits" failed because the lock was released with the other transaction. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts` — 95 passed. - `cd server && npx vitest run src/services/remote-workspace-online-source.test.ts src/services/heartbeat/workspace-preparation.test.ts src/services/isolated-repository-rehome.test.ts src/services/isolated-task-directory-retention.test.ts src/services/isolated-task-directory.test.ts src/__tests__/execution-workspace-reopen.test.ts` — all passed. - `cd server && npx tsc --noEmit` — clean. - No live Daytona replay was run. ## Risks - Low. A nested call inside a live scope behaves as before. - A call that inherited an ended scope now opens its own dedicated connection and lock transaction. Before, it failed. - Run preparation never joins another operation's lock scope. No caller relies on that today. If one did, it would now take a second, independent lock and could see `workspace_source_busy` instead of silently sharing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
839d41aabb |
fix(server): retire owned plugin listeners before idle sleep (#19358)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Plugins receive durable agent lifecycle events through a registered host. > - The host uses a PostgreSQL LISTEN connection to wake delivery. > - An idle report could return `none` while that host-owned connection stayed open. > - This change gives that listener an exclusive owner and closes it during the current idle hold. > - Release or expiry restores listening before journal replay. > - Failed close and pending work continue to block sleep. ## Linked Issues or Issue Description Refs #18988, which is now merged. This focused ten-file change now targets master. Current integrated candidate: `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`, with actual master `644ee66f5d4ae8365fe8b9eca57df2be7405b5ab` preserved through normal merges. No history was rewritten. **What happened?** A normal local App with an installed lifecycle plugin could return an idle `none` report while its lifecycle LISTEN backend and socket stayed open. **Expected behavior** The host must close its own listener before it reports idle safety. It must retain other subscriptions, durable events, company scope, retry deadlines and useful reactivation. **Steps to reproduce** Start the normal server with a lifecycle plugin. Create an agent and wait for its durable acknowledgment. Stop the server, create another agent while it is stopped, and restart with the same files. Request an owned idle hold after ordinary replay. Check the exact listener backend and matched Node socket before shutdown. Let the valid hold expire naturally, then make an ordinary authorized agent update and check its durable plugin acknowledgment. Live executing or persistent preview services remain active. The separate hosted preview dependency [paperclip-cloud#478](https://github.com/paperclipai/paperclip-cloud/pull/478) retains its own App and operator prerequisites. This PR does not claim deployed preview or suspension acceptance. ## What Changed - Add a private, exclusive database listener owner that inherits the normal client policy. - Prepare plugin lifecycle retirement after native retirement under the same current idle hold. - Use one delivery-eligibility helper for dispatch and idle journal inspection. It preserves the same manifest, capability, company-disable query, and skip rules. - Close only the owned listener. Keep failed close ownership and inspection failures visible. - Resume listening on normal release or expiry, before journal replay. - Preserve disabled-feature retry history and the original deadline on re-enable. - Await lifecycle disposal during normal shutdown and consumer teardown. - Add owner regressions and document the lifecycle contract. ## Verification **Current integrated source:** `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`. Fresh normal Runner TypeScript passed in 0.785 seconds. The complete normal App layered semantic check passed in 28.284 seconds. All tracked source, the lockfile and the 18 separate private witness inputs stayed unchanged. Known compiler owners closed and the isolated temporary home was removed. **Current product qualification:** at measured integration `896cc9b62993e6ed10418d7a7417e1ddbf8dc89c`, all three whole files passed with no failures or skips: lifecycle delivery 40, runtime hooks 20, and runtime assignments 24. The actual physical LISTEN and natural-expiry controls passed. All owned clients and captured processes closed. Independent final session checks were zero before normal deletion of each successful test database. **Normal full-App physical retirement:** the original case passed in 47.574 seconds. It exercised real startup, readiness, SDK callbacks, durable ACK, foreign-company exclusion, physical stop, offline input and cold replay. The initial unknown hold and its ordinary release remain in the evidence. After the original 30.5-second cadence, a new valid hold returned quiescent. The exact current cold-App LISTEN backend and its matched Node socket closed before server stop. **Normal full-App natural expiry:** the separate original case passed in 77.391 seconds under its original 90-second body cap. The final valid 30-second hold expired naturally. A new listener backend and matched current App socket appeared before fresh input. An ordinary authorized agent-name PATCH produced journal event 20, the actual SDK callback and durable ACK. Foreign exclusion and readiness assertions passed. The final measured hold had no manual release, replacement hold or shortened TTL. The original preparatory unknown hold, ordinary release and 30.5-second cadence remain explicit. Both App cases used the normal current server helpers and actual SDK bundling/worker entry. A private compatibility change supplied the same owned HOME at the exact existing child-fork boundary. It did not change the product helpers, business assertions or timing limits. These separate acceptance fixtures remain outside the focused product diff. **Source carry to the current candidate:** all nine other owned files, listener mechanics, actual producer paths, SDK paths and measured hold behavior are unchanged. The sole owned intersection from newer master is an independent environment-delete cleanup callback in `app.ts`. The plugin witnesses do not call that route. Normal merges preserve every unrelated master blob and file mode. The fresh current semantic checks above qualify the integrated source. New master Native crash-recovery and saved-copyback Retry capabilities are not claimed by these plugin cases. **Actual master merge target:** newer master contains retained plugin Runner release/resume and other Native recovery fixes. The projected combined target passed fresh normal Runner TypeScript in 0.799 seconds and the full App semantic check in 29.715 seconds. All ten plugin-owned blobs and modes remain exact to the published head; every unrelated master change is preserved. The measured plugin cases create no Native provider turn or warm Native session. The unchanged native-first/plugin-second ordering still checks the native result before listener retirement. Later sparse-checkpoint and Native response-publication changes do not enter those paths. This is a scoped source carry, not new hosted Runner acceptance. **Failures retained:** an earlier reactivation fixture's real timer returned one millisecond before its original wall-clock expiry. Its private fixture now waits any positive remainder and retains the same expiry assertion and limits. The earlier partial-composition `unknown` result remains distinct from normal-startup proof. Current qualification also exposed missing normal workspace dependencies and a private guard that assumed loose Git refs. Normal owned dependency setup and an exact loose-or-packed reader fixed those setup failures. No business assertion, timeout, permission boundary or product sleep gate was relaxed. Repeat focused product coverage with a fresh external fixture database and the normal server Vitest command, one worker: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/plugin-lifecycle-delivery.test.ts --maxWorkers=1 --no-file-parallelism pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-hooks.test.ts --maxWorkers=1 --no-file-parallelism pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-assignments.test.ts --maxWorkers=1 --no-file-parallelism ``` Exact current head `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1` has 60 successful checks, three skips and one neutral result, with no pending or failing contexts. Greptile completed its full review at 5/5 with no actionable issue. All root and nested review pages were read; the earlier eligibility finding is resolved. Ponytail's latest turn stopped before a verdict. A normal retry was requested; no current Ponytail verdict or score is claimed. Previous `3ca86ec` CI and scores are historical. A full local repository typecheck/test/build or browser run is not claimed. Controlled local App/SDK/PostgreSQL proof does not establish live provider, deployed Railway/Neon suspension, signed Cloud alarms or global exactly-once delivery. ## Risks - A listener close failure or uncertain inspection must continue to block sleep. - Release, expiry, configuration changes, and journal arrival can race with retirement. Epoch and current-hold checks retain conservative results. - Delivery remains at least once across uncertain ACK boundaries. This change does not add global exactly-once delivery. - The database helper closes only its exclusive listener client. Other owners and subscriptions keep their resources. ## Model Used OpenAI `gpt-6.1-sol`, reasoning and code/tool use. The managing Codex agent implemented shared database/composition seams and reviewed the change; its exact model ID was not exposed. The runtime does not expose a verified context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>canary/v2026.1107.0-canary.5 |
||
|
DottaandPaperclip
|
1c91bacf97 |
fix(native): admit independent replies during contended publication (#19726)
## Thinking Path > - Paperclip manages work done by AI agents. > - A completed native run must deliver its saved final response. > - The response worker also recovers retained replies that can wait on publication locks. > - A pass waited for its old publication groups even when other check slots were free. > - This could delay an independent final reply beyond a full page of retained candidates. > - This PR enables the existing shared coordinator option that admits independent due work during an active pass. > - Existing authority checks, resource keys and the four-check limit still control publication. ## Linked Issues or Issue Description **What happened?** A native run completed after replacement recovery. Its final response and accounting were saved, but its final issue reply was missing. The response queue contained more than 100 retained chat replies and repeated endpoint-lock failures. The existing worker did not admit independent due entries until its current groups finished. **Expected behavior** A contended publication owner must not hold free worker slots. An independent accepted final reply must publish once, including when its candidate is on a later SQL page. **Steps to reproduce** Create 100 accepted, committed replies that share one publication owner. Create an independent reply after them in candidate order. Hold the old coordinator rows with a database transaction and start the real response worker. Before this change, the independent reply stays unpublished until the old group is released. Related: #18977 changes response selection in standing conversations. This PR changes only worker admission and preserves response selection. ## What Changed - Enable `admitDueWhileActive` in the native response worker's existing shared coordinator configuration. - Add a real PostgreSQL regression with the full 100-row SQL page, the default dispatch batch, real row-lock contention and a later-page independent reply. - Verify one comment per reply after settlement and no further SQL or scheduled deadline across 60 seconds of idle time. - Update one existing principal/rescope integration assertion to observe the reconciled invalidation revision. Keep its actual per-run replies, authorization, publication and idle checks. ## Verification - Regression on the unchanged worker: the independent reply remained unpublished while the first 100 replies were locked. The real PostgreSQL test failed as expected. - Final head `748fb9609104c016c33ff7caf282ba9e15e88495`: all six publication regressions and both principal/rescope controls passed (8/8, 47.96 seconds). Checks cover later-page admission, exact per-run comment counts, retry and rollback behavior, authorization, and 60 seconds with no idle SQL or scheduled deadline. - The tests used the canonical PostgreSQL wrapper and one owned disposable PostgreSQL 18.1 fixture. All clients and test databases were closed before fixture cleanup. - The current-head [full CI run](https://github.com/paperclipai/paperclip/actions/runs/42338953241) supplies typecheck, build and broad test coverage. All current-head checks passed; the merge gate is green. - Local full typecheck was blocked by unbuilt dependencies; it is not counted as a pass. The prior timer-state CI failure and the final successful rescope-generation regression remain recorded. - Greptile reviewed the final head at 5/5 with no unresolved threads. `git diff --check` passed. - Deployed acceptance remains separate. No serving instance was changed by the PR tests. ## Risks The worker can now use free slots during a slow pass. The shared coordinator still limits checks to four and preserves task, endpoint and principal exclusions. Authorization, result identity, retry limits, response selection and transaction ownership are unchanged. The regression checks idle quiet after all concrete obligations settle. ## Model Used OpenAI GPT-6.1 (`gpt-6.1-sol`), with reasoning, code execution and tool use. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c45c7cb854 |
fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run in a remote sandbox copies its workspace back to the host after the turn. A sparse checkpoint copies only changed entries, and the host validates each entry against the sandbox's own manifest > - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch file at the sandbox workspace root and removes it only in the same process > - A control-plane kill during any transfer leaves that scratch behind. The sparse checkpoint then recorded it as a workspace file, but providers never export it. Every later copyback and every saved-work recovery attempt failed with "Workspace checkpoint payload mismatch" > - The failed recovery then left an accepted Retry showing "Paperclip is finishing the saved result" with no run > - This pull request stops recording transfer scratch, makes the checkpoint export an exact copy, names the failing entry, and gives a Retry behind a spent recovery an honest reason and a working action > - The benefit is that a restart during a transfer no longer makes a task's saved work permanently unrecoverable ## Linked Issues or Issue Description Refs #19721 (RC-50: the killed copyback owner; this PR covers the separate RC-50b cause) **What happened?** Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a SIGKILL of the control plane, the native result was accepted, and then workspace copyback failed with "Workspace checkpoint payload mismatch". In s14 all 6 saved-work recovery attempts failed the same way. A Retry accepted during recovery kept `retryWait.reason = finalizing_saved_result` after the recovery budget was spent. No run was created. **Steps to reproduce** 1. Start a native Claude run in a Daytona sandbox. 2. SIGKILL the control plane while any provider transfer is in flight. 3. Let the run finish its turn and copy its workspace back. **Expected behavior** Copyback succeeds. Provider scratch is not workspace content. A Retry behind a spent recovery names the failure and can start a new recovery. **Actual behavior** The s14 sandbox (6f4a0b6a) still held `/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`. Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona provider stages transfer scratch at `remoteDir`, which is the lease `remoteCwd` (the workspace root). The runtime workspace wipe deliberately preserves `.paperclip-upload-*`. The checkpoint helper (`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar. The ordinary export excludes `.paperclip-upload*`, so the host received nothing for that entry. `validateCheckpointPayload` then threw "payload mismatch" on every attempt. The file-mode hypothesis was disproved: the sandbox umask is 022 and the scratch is mode 644. ## What Changed - `packages/adapter-utils/src/exclude-patterns.ts`: new `isSandboxTransferScratchPath`, which matches a root-level `.paperclip-upload-*` entry. This agrees with the wipe's preserve rule in `buildWorkspaceTarExtractCommand`. - `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox checkpoint helper skips that scratch. A mismatch error now names the entry and its recorded, baseline and received forms (kind, octal mode, short hash; never contents). - `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the checkpoint export requests `preserveSourceEntries: true`. The checkpoint output is an exact copy that the host validates entry by entry. An ordinary export applies the host umask and drops `._*` and `.paperclip-upload*` names at any depth. That gives the same permanent mismatch for a legitimate user file such as `._summary`, `notes/.paperclip-upload-plan.md`, or a `0775` script. - `server/src/services/explicit-native-continuation.ts` and `task-interruption-presentation.ts`: when the run's saved-work recovery is exhausted, a waiting Retry reports `saved_work_recovery_failed` with Retry available, instead of `finalizing_saved_result`. The existing task Retry path starts one new recovery generation. - `doc/execution-semantics.md`: a short additive note. ## Verification - `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses the real checkpoint helper and an export that behaves like Daytona's ordinary export: - Before the fix: `Workspace checkpoint payload mismatch at ".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …; baseline absent; received absent`. That is the wave error. - After the fix: the scratch is not recorded, and validation passes. - A second case shows that reserved-looking user names and `0775` modes need the exact export. - `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an end-to-end `prepareSandboxManagedRuntime` restore through a provider model. In that model, ordinary exports drop reserved names and apply the umask, and preserve exports copy exactly. Two red checks: - Without `preserveSourceEntries`, it fails with the mismatch on `._summary`. - Without the scratch skip, the stale tar leaks into the host workspace. - Green: the agent file, the `._summary` file and the `0775` mode arrive, and no scratch arrives. - `server/src/services/workspace-recovery-preparation-gates.test.ts`, "names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it" (s14 shape): - Red: `finalizing_saved_result` after 6 failed attempts. - Green: `saved_work_recovery_failed` with `canRetry: true`. The next Retry starts a new generation, recovery completes, and the Retry continues in exactly one new run. - Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime, seed-reuse and managed-seed (168 passed). Server preparation-gates (47), explicit-native-continuation and task-interruption-presentation (455), workspace-recovery admission and retry-copyback, and native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils and server. - Live evidence: I started the stopped s14 sandbox read-only, listed its workspace, and stopped it again. No Daytona replay was run. The runner content ID of this checkout matches no built image. ## Risks - A user file named `.paperclip-upload-*` directly at the workspace root is no longer captured by a sparse checkpoint. Providers already drop that name from every ordinary export, and the runtime already reserves it at the root. - The checkpoint export now uses the provider's exact-copy mode. A workspace file with setuid, setgid or sticky bits is refused by that mode, as it is for the existing exact-copy transfers. The ordinary export would have stripped those bits and failed validation anyway. - The stale scratch tar stays in the sandbox. It is ignored, not deleted, because a concurrent transfer can own a file with the same prefix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres, a read-only Daytona sandbox listing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
b21884bdcf | Release and resume retained plugin Runner sessions in Core (#19611) | ||
|
DottaandPaperclip
|
9efee7ee16 |
fix(runtime): save dynamic skill bundles before native admission (#19729)
## Thinking Path > - Paperclip manages agents, tasks, and their saved execution state. > - A replacement server must restore each admitted skill from shared storage. > - Installed company skills already have immutable versions. > - Connector skills add assigned resources and tool definitions for each run. > - Those dynamic bytes keep a null version ID and differ from the static skill. > - Native admission saved their input before it saved a durable package receipt. > - This change stages the exact packages first and publishes the input and receipt together. > - A new connector turn can then survive an empty server home. ## Linked Issues or Issue Description Refs #19675 for the existing runtime asset preservation owner. **What happened?** A new native run with an assigned Slack connector creates a concrete local skill bundle after the old runtime assets were migrated. Admission saves its execution input without a preservation receipt. The new run creates a local durability dependency again. **Expected behavior** With S3 configured, the exact dynamic bundle must be durable before the execution input becomes visible. Its null version ID and admitted bytes must remain unchanged. Local storage must keep its volume-backed behavior. **Steps to reproduce** 1. Configure S3 and a native agent with an assigned Slack resource. 2. Prepare a new native task through the normal runtime selection owner. 3. Inspect its runtime skill receipt or restore its skills with an empty server home. The previous owner saved no receipt. **Paperclip version or commit** Reproduced on master `d1ee25d0f7`. The fix is tested on `4a3396d9b9`. **Deployment mode** S3-configured server. Local tests use PostgreSQL 18.1 and a private disk transport with the S3 receipt contract. ## What Changed - Extend the existing runtime asset owner with package staging and transaction-bound receipt validation. - Stage exact unversioned skill packages before native admission. Keep storage I/O outside the run transaction. - Publish the frozen input and receipt together under the existing run, cancellation, and package collection locks. - Reject changed input bindings and retired package metadata. - Reuse valid same-run receipts without reading the previous host's bundle path or requiring a new object read. - Keep existing online preservation and local storage behavior. - Add a bounded preparation span and document its work. ## Verification - Genuine unchanged-production RED: the actual Slack overlay and real admission transaction save no preservation receipt. - Final rebased head: 68 tests pass in three suites in 25.36 seconds. These include all runtime-selection, runtime-preservation, and connector controls. - The new positive checks that no execution input or coordinator is visible during package readback. It restores exact skill and tool bytes, modes, digests, and null version IDs in a cold home. - Guards cover an uncertain storage failure, cancellation during upload, a changed binding, retired package metadata, and local mode. An existing receipt remains usable after removing the original bundle and during a storage outage. - The canonical PostgreSQL wrapper uses one owned disposable PostgreSQL 18.1 container. Each suite's database is cleaned up. Real package archives, streams, and readback run locally; no external provider or model is called. - Strict server TypeScript, module boundaries, changed-line private-data checks, and `git diff --check` pass. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/42337473062) passes all required build, typecheck, test, and Runner gates. Fresh Greptile is 5/5 with no open review threads. ## Risks New S3-backed admission now requires its dynamic packages to be saved before execution can proceed. A storage failure can therefore stop preparation. Existing resource cleanup keeps its authority. Failed or cancelled staging retains original files and the package owner's pending receipts. Same-run recovery retains its original capture provenance and cache behavior. The tests replace the external S3 transport; they do not qualify live S3 permissions or provider execution. No schema, API, workflow, company skill version, or provider protocol changes are included. ## Model Used OpenAI Codex (GPT-6 family; the exact runtime model ID and context capacity are not exposed in this session), with reasoning, repository editing, code execution, and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
efc0b359da |
fix(ci): rebuild and sign an unsigned reused Daytona e2e image (#19677)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid runner E2E campaigns run in `runner-full-stack-e2e.yml`.
Daytona cells need a signed, content-addressed runner image
> - The "Publish verified Daytona image" job reuses `e2e-content-<id>`
when the tag exists, and signs only after a fresh build
> - A run that stops between the push and `cosign sign` leaves an
unsigned tag. Every later run reuses it and fails `cosign verify` with
"no signatures found"
> - That failure skips the whole test matrix, including local cells. The
first Reliability TestOps launch (run 42298071794) ran 0/7 cells for
this reason
> - This pull request reuses a tag only when its signature verifies.
Otherwise it rebuilds, pushes and signs the tag
> - The benefit is that an interrupted run cannot block every later
Daytona campaign
## Linked Issues or Issue Description
Refs #19549
**What happened?**
Run https://github.com/paperclipai/paperclip/actions/runs/42298071794
(TestOps launch of the Reliability plan) failed in "Reuse or publish
immutable image" with `Error: no signatures found` (exit code 10). No
test cell ran. The report shows 7 missing attempts.
**Expected behavior**
A campaign with Daytona cells always produces a signed, verified image,
or it fails for a real build or signing error. A stale unsigned tag must
not block it.
**Steps to reproduce**
1. A run pushes
`ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-<id>` and
stops before `cosign sign` (for example, a runner shutdown).
2. A later campaign with the same content ID finds the tag and skips the
build.
3. `cosign verify` fails with "no signatures found", and the matrix is
skipped.
**Paperclip version or commit**
master
|
||
|
DottaandClaude Opus 5.5
|
644ee66f5d |
test(restart-chaos): R5 live tools and processes; fail a wave whose worktree changes mid-wave (#19697)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) grades every task on R1–R6; R5 catches silent stalls, including a run that stays `running` with no live evidence > - A B5 task runs `bench.sh` for about four minutes with no output, so a healthy Retry run was graded `R5:silent_dead_run` while its runner process was alive and its tool was still running > - Separately, a runner `dist` rebuilt in the harness worktree during a wave made later restarts run different code and produced a false signature (RC-32's investigation) > - This pull request makes R5 treat a run as live while a tool it started has not ended or while its recorded local runner process still runs, and fails a wave whose worktree changes mid-wave > - The benefit is fewer false R5 failures and no silently mixed-code waves ## Linked Issues or Issue Description Refs #19629 Refs #19641 **What is the problem?** R5 flags a quiet but live run (a long tool call, a live runner process) as `silent_dead_run`. A build, install or branch switch in the harness worktree during a wave goes unnoticed, though every restart re-runs the control plane from that worktree. **What is the expected behavior?** A run with a tool in flight, or with its recorded local runner process still alive (same PID and start time), is live for R5. A wave whose checkout or runner build output changes during the wave fails as a harness error. **Why does it matter?** False R5 failures send fixers after non-bugs, and mixed-code waves produce signatures that are not product behavior. ## What Changed - `observer.ts`: the run-event reader counts tool starts minus tool ends (PRP item ids are not unique per tool call), and remembers the run's latest local process identity until recovery reports it stopped. Samples carry `toolInFlight` and `processAlive`. `localProcessAlive` checks `ps` for the same PID with the same start time, so a recycled PID never counts. Daytona runs are not checked (their process is remote). - `grader.ts`: `deadRunningRun` skips a run with a tool in flight or a live process. - `types.ts`: `RunSample.toolInFlight` and `RunSample.processAlive`. - `wave-guard.ts` (new) and `main.ts`: fingerprint `HEAD`, the tracked diff contents (`git diff --binary HEAD`), untracked file contents, and size/mtime of `packages/paperclip-runner/dist` and the runnerd binary. The baseline is taken after the harness's own setup (preflight rebuilds provider entrypoints). Re-check after each fault and at the end; fail the wave (`worktree_changed_mid_wave`) if it changed. - `main.ts`, `scheduler.ts`: `faults.json` and a new `task-faults.json` (each task's launch time and crossing faults) are written after every fault. `--replay` falls back to that log when a task has no grade file (an interrupted or wedged wave), so a storm task replays the faults that crossed it at the same offsets. - `README.md`: both grading notes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 111 tests pass. New tests cover: - a B5 run quiet for 4 minutes passes with a tool in flight or a live process, and fails otherwise - tool and process liveness come from run events and clear on tool end and process stop - `localProcessAlive` accepts this process's real PID and start time and rejects a different start time - the guard catches rebuilt dist, a re-edited dirty file, an edited untracked file and a rebuilt runner binary, in a real temporary git repo - a storm task with no grade file replays the faults that crossed it at their offsets from its launch - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. ## Risks - Low risk, test harness only. - The R5 exemption narrows R5. A run that is truly stuck while its process lives (a hung provider) now passes R5 and falls to R1's time budget instead. - One `ps` call per running run per poll on local waves. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
db41bfcb38 |
fix(recovery): let an accepted Retry finish after a killed copyback owner (#19721)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run saves the agent's workspace files after the provider turn (copyback). One controller owns copyback at a time and records a durable owner receipt on the run (`nativeWorkspaceFinalizationOwner`) > - When copyback of an accepted result fails, retained saved-work recovery exports the files from the kept sandbox. A task Retry waits for that recovery before it continues > - A control-plane restart can kill the controller while it owns copyback. The dead controller can never confirm its own stop, and nothing in the retained recovery path could release its receipt > - The recovery worker then spent one attempt on each recheck of that receipt until it was exhausted, and the accepted Retry kept saying "Paperclip is finishing the saved result" with no run. The task waited forever > - This pull request stops spending attempts on that recheck, names the real wait, and lets the board release the exact receipt so the same recovery and the same Retry continue > - The benefit is that an accepted Retry never waits silently: it finishes, or it shows one honest action that makes it finish ## Linked Issues or Issue Description Refs #19717 (same-host automatic proof for a killed copyback owner; this PR covers the retained-recovery path and the case where no automatic proof is possible) Refs #19713, #19609, #19694 (related restart-recovery fixes; no overlap in files except this doc) **What happened?** Two Daytona restart-chaos waves. In each, a native run's copyback failed and the run failed with `native_workspace_sync_out_retry_exhausted`. The agent's result was accepted. Retained recovery started. A SIGKILL of the control plane landed while a recovery attempt owned copyback: - Wave s13, task t06 (Claude, crash loop): Retry was accepted at 02:30:35. A SIGKILL landed 2 s later. All later attempts failed with `copyback_stop_unconfirmed`. After 6 attempts the recovery was exhausted. The Retry showed `retryWait.reason = finalizing_saved_result` for more than 15 minutes. No run was created. - Wave s12, task t01 (Claude): attempt 1 failed with "Workspace checkpoint payload mismatch". The SIGKILL at 01:43:54 killed the owner of attempt 2. Attempts 3 to 5 failed with `copyback_stop_unconfirmed`. The Retry was accepted with HTTP 2xx and produced no run. In both kept databases the run carries an owner receipt for the killed process. **Steps to reproduce** 1. Accept a native result and make its workspace copyback fail, so that retained recovery owns the export. 2. Press Retry on the task. 3. Kill the control plane while a recovery attempt owns copyback, so that its owner receipt stays on the run. Use another host, or a receipt without automatic stop proof. 4. Let recovery run. **Expected behavior** Rechecking the receipt does not spend attempts. The task names the wait and does not offer a Retry that cannot help. A board member can confirm that the previous server stopped. The recovery then finishes, and the waiting Retry continues by itself. **Actual behavior** `withNativeWorkspaceFinalizationOwnership` returned `owner_unverified` for a retained repair without creating any board action (`native-workspace-finalization-ownership.ts`, `requireStopVerification`). `recoverRetainedWorkspace` reserved an attempt for each recheck and failed it as `copyback_stop_unconfirmed` (`workspace-recovery.ts`), until the budget was exhausted. The Retry gate in `explicit-native-continuation.ts` returned `finalizing_saved_result` for an accepted result with no verified repair, with no check for this hold. `resumeNativeWorkspaceAfterOwnerStop` accepted only an `owner_unverified` action on a `running` run, so no route could release the receipt of a finished run. The exhausted handoff told the board to press Retry again, which repeats the same failure. ## What Changed - `server/src/services/workspace-recovery.ts`: when retained recovery finds an unverifiable copyback owner, it records the recheck without reserving an attempt. The 30-minute recovery window still bounds it. The retained action records the exact receipt as `workspaceOwnerRecovery: { state: "operator_required", owner }`. A newer receipt replaces the record and clears an older confirmation. - `server/src/services/explicit-native-continuation.ts`: a Retry that waits for an accepted saved result reports `workspace_owner_stop_required` when the run's active retained action has that hold. It no longer reports `finalizing_saved_result` for this case. - `server/src/services/task-interruption-presentation.ts`: the new wait has a fixed message, and Retry is not offered for it, because another click cannot release the receipt. - `server/src/services/native-runtime/native-workspace-owner-recovery.ts`: the existing board route `POST /issues/:id/recovery-actions/resume-workspace-finalization` (permission `runtime:confirm_owner_stop`) also accepts a retained recovery action with that hold. The confirmation must name the receipt that is still on the run. A live owner on this host still rejects it. It clears only that receipt, records the stop proof and an activity entry, and returns the same recovery to the worker. A spent budget gets one new generation, as a deliberate recovery request does. It accepts no result and starts no agent turn. The owner check is shared with the existing path. - `server/src/services/system-recovery.ts`: an exhausted `copyback_stop_unconfirmed` handoff names the board confirmation instead of another Retry. - `ui/src/api/issues.ts`: the route receipt's `resultId` can be `null` (a retained recovery has no live result binding). - `doc/native-workspace-finalization-recovery.md`: an additive paragraph describes this path. ## Verification - Red then green on embedded Postgres: `server/src/services/workspace-recovery-preparation-gates.test.ts`, "lets an accepted Retry finish after a killed copyback owner instead of waiting forever" with one case per wave (s13, s12). Each case uses a real accepted remote export, a Retry through the HTTP route, a killed-owner receipt, and 8 recovery passes. - Before the fix (both cases): `attempts: 6, state: "exhausted"`. That is the wave state. - After the fix: no attempt is spent on the recheck. The hold is recorded. The Retry wait is `workspace_owner_stop_required` with `canRetry: false`. The board confirmation through the route returns 202 and clears the receipt. The same recovery completes. The same accepted Retry then finishes the saved result (s13: the task is `done`) or continues in exactly one new run (s12). - `npx vitest run src/services/workspace-recovery-preparation-gates.test.ts -t "killed copyback owner"`: 2 passed. - `pnpm verify` (affected selection): see the comment below. Server `tsc --noEmit` is clean. - No live Daytona replay. Both waves ran on old signed images. The local proof uses the same accepted-export fixture as the existing Retry tests, with the receipt state copied from the kept databases. ## Risks - A copyback owner on another host that is still alive is now rechecked within the recovery window without spending attempts. Before this change it spent them. The window still bounds the wait. - The board confirmation for a retained action is an operator statement, the same as the existing owner-stop path. It is limited to roles with `runtime:confirm_owner_stop`. It never clears a receipt other than the one it names, and a live same-host owner still rejects it. - #19717 changes the same ownership area and adds a UI action for the running-run case. That UI does not show a retained action yet. After both merge, a small follow-up can show its button for this hold. Until then the API route and the task's wait message cover it. - Not addressed here: the "Workspace checkpoint payload mismatch" that started wave s12. The checkpoint outbound transfer extracts without `-p` (`sandbox-managed-runtime.ts` checkpoint sync-out, `extractHostTarball` in the Daytona provider), so the host umask can change file modes that the remote manifest recorded. That is a separate signature. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5cc89b1e4d |
fix(native): do not charge a restart resume that made progress before the next crash (#19720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A control-plane crash (no graceful drain) makes the next server resume the run, and that resume charges one attempt > - #19625 stopped charging a resume after a graceful drain, but crash restarts still charge, even when the previous resume did real work > - On a deploy day with several restarts, a healthy run used up its budget and the task needed a manual Retry > - This pull request does not charge a crash resume when the previous restart resume made recorded progress > - The benefit is that restarts alone do not exhaust a run that keeps making progress, and genuine loops stay bounded ## Linked Issues or Issue Description Refs #19625 **What happened?** In restart-chaos testing (storm wave, OpenCode run), a run crossed four control-plane restarts: K1, K1, K2 (drain), K1. Each resume was clean: the runner restored the session, recorded `provider_turn_lost_on_restore`, and the #19699 continuation turn then ran tools for minutes. The two K1 resumes each charged one attempt (attempt 1 to 3). The drain resume did not charge (#19625). The fourth restart found attempt 3 and blocked the run with `native_restart_recovery_blocked` / `execution_recovery_budget_exhausted`. Seven runs in the same wave failed in the same way. In every case, each resume had accepted a new turn and completed tool work before the next restart. **Expected behavior** A run must not fail because it survived several controller restarts that each resumed cleanly and made progress. The budget must still stop a resume that fails each time, a crash loop that makes no progress, and a provider that fails repeatedly. **Steps to reproduce** Start a native local run. Kill the server (SIGKILL) while the run is active and let it resume. Let the resumed turn run tools. Repeat three times. The third crash resume is blocked with `execution_recovery_budget_exhausted`. **Paperclip version or commit** `master` at `8a34c1b62c`. ## What Changed - `claimNativeRestartRecoveries()` (`native-restart-recovery.ts`) records `progressWatermarkSeq` (the run's highest `heartbeat_run_events.seq`) in the history entry of each recovery claim. - New `previousResumeMadeProgress()`: the next claim reads the watermark of the most recent claim. It finds progress when, after that watermark, a `turn.accepted` event names a turn, and a later `tool.execution.completed` or `item.completed` event (not a `usage` item) belongs to the same turn. The evidence is committed run events, not timing. - When that progress exists, the claim is not charged (`chargesProviderAttempt: false`, and the `>= 3` exhausted gate is skipped). This uses the same path as the clean-drain rule from #19625. The history entry records `attemptKeptForProgress: true`. - Restore activity does not count: lineage items without a turn, `turn.failed`, work on the interrupted turn, `turn.submitted`, and an accepted turn with only usage or a failure. So a resume that dies or fails before it does work still charges. - The credit stops at controller generation 12 (`NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT`). A run that does some work and then crashes the controller every time still reaches the budget. - The first restart (which interrupts the original execution, not a resume) still charges. A history entry from an older release has no watermark, so it gets no credit. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`; between claims the test applies the executor's attempt charge and marks the controller dead): - `keeps recovering a run whose restart resumes each completed work before the next crash`: five crash resumes, each followed by restore noise and one completed tool in the continuation turn. Charges are `[true, false, false, false, false]` and the attempt stays 2. Red on master (attempt 2, 3, then blocked). - `still exhausts the budget when restart resumes complete no work between crashes`: resumes followed only by restore noise and a failed continuation turn. Attempt 2, attempt 3, then `execution_recovery_budget_exhausted`. Green on master and with the fix. - `bounds a crash loop that makes progress before every crash`: credit stops at the generation limit, the last attempt is spent, and the run blocks. Red on master. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts` and `native-remote-retry-admission.test.ts`: 218 passed, 14 skipped. - I ran the progress query against the live database of the failing wave. For all seven blocked runs, every resume had progress before the next restart, so this rule would have kept each of them recovering. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A run that makes progress and then makes the controller crash each time gets up to about ten extra resumes before the budget applies. The generation limit bounds this. - The rule depends on the PRP event shape (`payload.prpEvent.turnId` on `turn.accepted` and on completed work). Events without that shape get no credit, so the failure mode is the old behavior (charge), not an unbounded loop. - The query runs once per claim, on the `(run_id, seq)` unique index range after the watermark. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d1ee25d0f7 |
fix(runner): fail a pending Codex turn start at once with the runner's own terminal failure (#19722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane starts each Codex turn through runnerd and waits for the turn's start event (`#startTurnWork` in `runnerd-codex-transport.ts`) > - In restart storm 5, runnerd failed an accepted continuation turn before it started (`turn_binding_mismatch`, fixed at its source in #19718) > - The transport held that terminal for the pending start and waited the full 30 s bound > - It then reported "runnerd did not report the provider turn identity", which hid the real cause > - This pull request makes a pending start fail at once with the runner's own terminal failure > - The benefit is a faster failure that names its real cause, so recovery and operators act on the correct fact ## Linked Issues or Issue Description Refs #19718, #19699, #19608 **What happened?** Restart storm 5 (seed 7106), task t01 (Codex). Runnerd accepted the restart continuation turn. About 0.5 s later it committed a `harness.diagnostic`, a non-recoverable `turn.failed` (`turn_binding_mismatch`) and a failed `run.terminal`. The transport's event pump holds a turn terminal while a turn start is pending, so the start loop never saw it. The start waited 30 s, the #19608 adoption check found no active turn, and the run failed with "runnerd did not report the provider turn identity". **Expected behavior** When runnerd reports a final failure of the pending turn (a non-recoverable `turn.failed` or a failed `run.terminal`) before the turn starts, the start fails at once. The error names the runner's code and message. **Steps to reproduce** 1. Use fake Codex with `--foreign-usage-before-turn-started --turn-started-delay-ms 20000`. 2. Start a turn through the runnerd Codex transport. 3. Before this change, `turn/start` rejects after the full bound with "runnerd did not report the provider turn identity". ## What Changed - `runnerd-codex-transport.ts`: new `runnerdPendingTurnStartFailure()`. It finds a non-recoverable `turn.failed` or a failed `run.terminal` among the terminals held for the pending start and the committed events after the cursor. - `#startTurnWork` checks it while it waits for `turn/started`, before the adoption snapshot, and in the post-adoption wait. It throws `runnerd failed the provider turn before it started: <code>: <message>`. - The check runs only when the turn has not started. A recoverable terminal does not trigger it. The #19608 late-start adoption path is unchanged. - Fake Codex: new `--foreign-usage-before-turn-started` flag. It sends usage for a turn the runner does not own right after it accepts a turn. This makes runnerd fail the turn with `turn_binding_mismatch` before it starts. - New test in `runnerd-codex-transport-deadlines.test.ts` with real runnerd and fake Codex. ## Verification - Red before the fix: `npx vitest run src/live/runnerd-codex-transport-deadlines.test.ts` → the new test failed after 16.4 s with "runnerd did not report the provider turn identity". - Green after the fix: the start rejects in under 5 s with `turn_binding_mismatch`, and fake Codex saw exactly one `turn/start`. - The #19608 tests ("adopts a durably accepted turn whose start notification outlives the turn-start bound" and "does not adopt an accepted turn the runner no longer reports active") still pass. - `npx vitest run src/live/runnerd-codex-transport`: 9 files, 399 tests pass. - `pnpm run typecheck` in `packages/paperclip-runner` passes. `cargo test --bin fake-codex-app-server` passes. `cargo fmt` is clean. - `pnpm verify` exits 2 (broad input: the fake Codex binary). Required CI supplies the broad coverage. ## Risks - Low risk. The new failure fires only when runnerd itself already reported a final failure and no turn start was observed. In that state the start could not succeed, and before this change it failed after the bound anyway. - If a provider sends a non-recoverable terminal before its own turn start for a turn that later starts, the start now fails earlier. Codex sends `turn/started` before its terminal, and runnerd's terminal for a protocol failure is final, so I do not expect this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1107.0-canary.4 |
||
|
DottaandPaperclip
|
1a60d47804 |
Preserve settled native log files during online migration (#19719)
## Thinking Path > - Paperclip manages work for AI agents. > - The web server must retain run history across replacements. > - Older native runs can have local log files with no final size or digest. > - The normal importer correctly refuses to call those files complete. > - The existing raw archive mode requires the entire instance to stop work. > - This change gives settled sources a separate, guarded online archive path. > - Active work can continue while old evidence moves to object storage. ## Linked Issues or Issue Description Refs #18864, #19678, #19688. The offline historical log archive path cannot migrate a busy instance. This change preserves exact surviving bytes without claiming complete native-event coverage or changing the original run outcome. ## What Changed - Add an explicit online option for settled native log files. Keep the default importer and offline option unchanged. - Require a released, closed native finalizer and no pending attempt, allocation, physical owner, or task binding. A stale lease or terminal run status alone is insufficient. - Use the canonical physical finalization lock and row locks. Check the source generation before claims and publication. - Read only the canonical, private local file. Reject aliases, symlinks, hard links, changed files, and changed owners. - Verify remote bytes before publishing a read-only archive and activity receipt in one transaction. Keep original files, errors, run status, event rows, and completeness metadata. - Document the command and its authority limits. Add real PostgreSQL and loopback S3 transport tests. ## Verification - The original implementation failed the new archive regression. It returned blocked instead of migrating the settled source. - Final source integration suite: 15 tests pass on disposable PostgreSQL and the bounded loopback S3 fixture. Coverage includes active unrelated work, failed and successful sources, a 4 MiB segment boundary, owner changes, file changes, unresolved owners, task bindings, allocations, and filesystem aliases. - The qualified source TypeScript check and module boundary check pass. - All 73 tests in five affected existing log-store, object-journal, cancellation, and migration-worker suites pass. The first borrowed dependency setup loaded work contexts from another worktree and failed four tests. Explicit database subpath aliases resolved that setup fault; no worker code changed. - Earlier test fault injection used the wrong storage method, and the first segment test reused a closed fixture endpoint. Those fixture defects were corrected before the final 15-test pass. - Local tests use Node 26. CI supplies the standard full typecheck, build, and test gates. Production object-storage credentials, live Railway replacement, and provider continuation are separate deployment acceptance steps. - Final exact-head CI: 59 successful contexts and 3 skipped, with all typecheck, build, test, E2E, security, and review gates complete. [Workflow evidence](https://github.com/paperclipai/paperclip/actions/runs/42332349644). - The first CI attempt had 12 jobs interrupted by runner shutdown. Their logs end in cancellation, with no test assertion found. The failed/cancelled-only retry passed all 12 jobs and recomputed aggregates. The 38 original successful job records kept their original execution times and were not rerun. Original interruption logs remain retained. ## Risks - This path preserves raw history. It cannot recover missing source bytes or prove a complete native transcript. - Uploads can finish before a later source check fails. The run reference stays local in that case; a retry must still satisfy all authority checks. - The new option is explicit. It adds no startup worker, recurring poll, or background scan. - Source transactions hold scoped locks only for ownership checks and publication. Streaming and remote verification run outside those transactions. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use, and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
da850990f8 |
fix(restart-chaos): grade R4 on overlapping execution; wait out pending cleanup for R6 (#19710)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades every task on rules R1 to R6. R4 means "no duplicate execution" > - R4 counted every queued, running or scheduled-retry run in one poll as an active run, and failed the task when there were two > - A follow-up comment while a run is in progress normally queues a wake behind that run. The product cancels the wake when the running run finishes, so the wake never executes > - A Daytona wave failed an opencode task for exactly this case, so R4 reported a duplicate-execution failure that did not happen > - This pull request grades R4 on actual overlapping execution > - The benefit is that R4 failures again mean that two runs really executed together, or a comment was consumed twice ## Linked Issues or Issue Description **Bug description** `gradeR4` failed a task whenever one poll showed two runs in `ACTIVE_RUN_STATUSES` (queued, running, scheduled_retry). This includes a wake queued behind a running run that never starts. **Steps to reproduce** Daytona wave `w-20261107T013205-s11`, task t06 (opencode C1 K2): - Run b4b16a4d is running. - The C1 follow-up comment queues run c090799a. - c090799a is cancelled 1 s after b4b16a4d succeeds and never starts. - Before this change, R4 reported `concurrent_runs: c090799a(queued), b4b16a4d(running)`. **Expected behavior** R4 passes. It fails only when two runs execute at the same time, or when a side effect (a consumed comment) repeats. Refs #19544 ## What Changed - `tests/restart-chaos/grader.ts`: `gradeR4` fails when two runs are `running` in the same sample, or when the started→finished windows of two started runs overlap. A still-active run's window lasts until the last sample. Queued and scheduled-retry runs that never started do not count. The comment-consumed-twice check is unchanged. - `tests/restart-chaos/grader.test.ts`: a test with the t06 shape (a queued wake cancelled behind a running run) that passes; queued plus scheduled_retry without a start passes; overlapping windows fail even when no single poll saw both running. The existing two-running and comment tests are kept. - `tests/restart-chaos/README.md`: the R4 grading note. - R6 settle for Daytona (`tests/restart-chaos/daytona-sweep.ts`, `main.ts`). The product retries a `pending_cleanup` lease after `ENVIRONMENT_CLEANUP_RETRY_MS` (5 min), so a 2-min settle reported normal cleanup as a leak. R6 now waits `PENDING_CLEANUP_RETRY_MS + 90 s`, capped at 8 min; a test keeps the constant equal to the server's. An environment DELETE that returns 409 while a sandbox cleanup is pending is correct product behaviour. It is retried for the same window. If it is still blocked after that, it is reported as `environmentDeleteBlocked`, and it fails the wave and keeps the instance (whose database owns the cleanup work). It is not counted as a harness error. Tests cover the 409 retry and a real (500) delete failure. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 109/109 pass. `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - R6 evidence: in wave `w-20261107T015455-s12`, three leases were still `pending_cleanup` at the 2-min mark, and the environment DELETE returned 409 "Cannot delete this environment while a sandbox cleanup is pending". - The t06 sample sequence from `w-20261107T013205-s11` is the new test case. ## Risks - This change touches the test harness only. - R4 is now less strict about queued rows. If a queued duplicate starts executing, its start time makes its window overlap, so R4 still fails. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
aed053633e |
fix(environments): run pending sandbox cleanup when an environment is deleted (#19716)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Operators delete a sandbox environment, optionally with consent to destroy its sandboxes (`DELETE /api/environments/:id?destroyReusableSandboxLeases=true`) > - A sandbox whose teardown failed or was interrupted is kept as a `pending_cleanup` lease, and it blocks the delete > - The cleanup sweep retries such a lease only after a cooldown of up to 5 minutes > - A consented delete therefore returned 409 `pending_sandbox_cleanup` for minutes. A caller with a shorter deadline gave up, and the environment and its sandbox stayed behind > - This pull request makes the consented delete run one teardown attempt for the environment's pending cleanups now, then re-check > - The benefit is that a consented environment delete completes as soon as the provider teardown succeeds ## Linked Issues or Issue Description **What happened?** After a restart chaos wave, the harness deleted the wave's Daytona environment with `destroyReusableSandboxLeases=true`. The server returned 409 `pending_sandbox_cleanup` every 3 seconds for the whole 120-second client deadline. A run lease had entered `pending_cleanup` after its run failed, and the sweep's retry cooldown had not passed. The environment was not deleted. **Expected behavior** With explicit consent, the delete tears down the environment's own pending sandbox cleanups and completes when that teardown succeeds. A teardown that fails, or one already in flight elsewhere, still blocks the delete. **Steps to reproduce** 1. Leave a sandbox lease of an environment in `pending_cleanup` with a future retry time. 2. `DELETE /api/environments/:id?destroyReusableSandboxLeases=true`. 3. On master, the response is 409 `pending_sandbox_cleanup` until the cooldown ends. Refs #19696 (restart recovery for creation and teardown claims; this builds on it). ## What Changed - `heartbeat/recovery.ts`: `sweepPendingCleanupLeases` accepts `environmentRetry: { environmentId, actorId }`. Like the existing run-scoped explicit retry, it bypasses the cooldown for one attempt, never takes over an in-flight attempt, and logs `environment_lease.cleanup_retried` with reason `environment_deleted`. It touches only that environment's leases. - `routes/environments.ts`: when the delete has consent and the only blockers are pending cleanup and reusable leases, it runs that retry first, then re-reads the blast radius. The existing reusable-lease destroy and the final guard are unchanged. The new dependency is a narrow, optional `retryPendingEnvironmentCleanup` option; without it, behavior is unchanged. - `app.ts`: wires the option to the existing heartbeat service instance. ## Verification - Red, then green: - `environment-routes.test.ts`: "runs the environment's pending sandbox cleanup now and deletes with explicit consent" and "keeps rejecting when the consented pending cleanup attempt does not finish" fail before and pass after. "does not run pending sandbox cleanup without consent or past another blocker" passes in both. - `heartbeat-pending-cleanup-sweep.test.ts`: "runs one attempt for a deleted environment's cleanup during its cooldown, scoped to that environment" fails before and passes after. It checks that another environment's lease and an in-flight claim are untouched, and that the activity log is written. - `sandbox-creation-operations.test.ts`: "reclaims an issue-bound creation after its run failed and the issue finished on a later run" covers another chaos finding. A creation intent bound to an issue was interrupted by SIGKILL 0.5 s after it committed, its run ended `process_lost`, and a later run finished the issue. The test fails on the code before #19696 and passes on master, so #19696 already covers this case. The test is kept as a regression. - After rebasing on master, `environment-routes`, `heartbeat-pending-cleanup-sweep`, `agent-runner-qualification-routes` and `sandbox-creation-operations` pass (192 tests). `tsc --noEmit` reports no errors in the changed files. ## Risks - A consented delete now makes provider teardown calls for pending cleanups. Those sandboxes were already queued for destruction, and the operator consented to destroy the environment's sandboxes. - One explicit attempt counts toward the lease's attempt cap, the same as a user Retry. - The delete request now waits for those teardown attempts. Each attempt uses the existing plugin timeouts. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
51eb4ff705 |
fix(recovery): finalize an accepted native result after a hard kill (#19713)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run hands its accepted result to a controller lease, and the controller finalizes the task from that result > - A graceful shutdown releases that lease for the next server (#19591), but a hard kill (SIGKILL, OOM, host crash) cannot > - After a hard kill between result acceptance and finalization, the next server waited for the dead controller's 20-minute lease, so the run stayed `running` and the user's follow-up comment waited with it > - This pull request releases that lease at startup when the controller is proved dead by its boot ID and process identity, never by age > - The benefit is that the saved result finalizes within seconds after a restart, without another provider turn, and queued follow-up work continues ## Linked Issues or Issue Description Refs #19591 (the graceful-shutdown counterpart) and #19578 (boot-identity proof for legacy runs). **What happened?** A restart-chaos wave sent K1 (SIGKILL) 20 ms after a Claude run recorded `run.result.accepted` and `run.terminal`. The run then stayed `running` for more than 10 minutes, through four more restarts. The finalization row was `workspace_finalizing` with the stored result. Its lease belonged to the killed server: a different `controller_boot_id`, a `controller_pid` with no process, and `lease_expires_at` 20 minutes ahead. Startup recovery skips runs that already have a result. The result-commit worker waits until the lease expires. The task's follow-up comment could not run until then. **Steps to reproduce** 1. Let a native local run accept its result. 2. SIGKILL the server before finalization finishes. 3. Restart the server. The run stays `running` until the old lease expires. **Expected behavior** After a restart, a result lease held by a stopped controller of an earlier boot is treated as released at once. The stored result finalizes once, with no new provider turn, and the follow-up wake runs after it. ## What Changed - New `server/src/services/native-runtime/native-dead-controller-result-lease.ts`: `releaseDeadControllerResultLeases` finds `running` native runs with an accepted result and an unexpired lease. It requires the lease to belong to a different controller boot. It proves the controller stopped with `evaluateNativeControllerTakeover` (`requireStoppedController`): the process is dead, or its PID now names another process. It then expires that lease with a compare-and-set on the exact owner token, boot ID, and PID, and notifies the result-commit and finalization workers. - `server/src/services/heartbeat/recovery.ts`: `recoverNativeRunsAfterRestart` calls this first and records a `native.recovery.transition` event (`awaiting_result_finalization`) for each released run. - Only executor leases are released, because only the executor claim writes the controller identity together with the lease. Finalizer (`native-finalizer:`) and cleanup (`native-cleanup:`) claims keep an earlier executor's identity, so they are never released this way and expire on their own short leases. A finalization-resume token (`:resume:`) on a result row always belongs to the executor that adopted it, because a pending resume claim is taken only on a row without a result. - Also not released: a live owner, an owner without a recorded process, a lease of the current boot, a run this process is executing, and remote workspaces (`nativeWorkspaceSync` or `remoteWorkspace`), which keep their fenced lease. - The compare-and-set does not compare the stored expiry. Postgres stores microseconds that a JavaScript `Date` drops, and an exact-expiry comparison silently skipped the real lease in the live reproduction. - `doc/execution-semantics.md`: documents the rule. ## Verification - Red then green on embedded Postgres (`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`): - A hard kill with a dead controller's executor lease (first claim, restart claim, or resumed claim) now finalizes on the next startup: run `succeeded`, finalization `committed`, task `done`, one result, one run, no provider dispatch. These tests failed before the fix with `status: running`. - A lease expiry stored with microsecond precision is released. This test failed against the first version of the fix and reproduces the live miss. - A live controller of another boot keeps its lease, and so does a lease of the current boot. - A live claimant's finalizer or cleanup lease is kept when the recorded earlier executor is dead. - `npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 20 passed. - Related suites pass: `native-session-resumption`, `native-runner-restart-recovery.integration`, `native-run-projection-work`, `live-recovery-work`. One `live-recovery-work` legacy case timed out once under load and passed when run again by itself. - Live, local: `tests/restart-chaos/main.ts --workloads C1 --triggers result --faults K1 --rc06-masked`. The failing case above came from such a wave. I started its kept database and confirmed the stored lease state and the compare-and-set miss. Against that same database, the fixed release expires the stuck executor lease. Two later waves (12 tasks, `--triggers result`) passed every task. Run events were not kept for passing tasks, so I cannot confirm that a kill landed in the narrow pre-finalization window in those waves. - Live proof on head `8971874654`: local ACPX Claude, workload A1, seed 7411, `--triggers result` (a local variant that fires on A1 2.5–3.5 s after the run's result proposal), `--keep`. The run passed 4 of 4 tasks. In two tasks, K1 landed 1.5 s and 2.0 s after `run.result.accepted`. On the next startup, each run recorded "Server restart released the accepted native result of a stopped controller for finalization" (`controller_process_dead`). Each finished `succeeded` about 1 s later, with its task `done` and exactly one run per task (no extra provider turn). Before this fix, the same shape stayed `running` until the 10-minute budget ran out (seeds 7411, 7422, and 7423 from the RC-33b investigation in #19691). ## Risks - Startup could take a lease from a finalizer that is still running. Mitigations: the lease must belong to another boot, the controller process must be proved stopped on this host (the same proof the existing runner takeover uses), and the update matches the exact owner. Remote workspaces are excluded. Multi-host deployments keep the existing PID-liveness assumption that `claimNativeRestartRecoveries` already makes. - Low behavioral risk otherwise: finalization uses the existing reconciler path, which #19591 already exercises after a graceful drain. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
42f6b3a870 |
fix(runner): keep a restart-lost Codex turn's late usage from failing its continuation (#19718)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs continue a turn lost to a control-plane restart once, in the same conversation (#19699) > - In restart storm 5, a Codex run crossed one graceful restart. The continuation started, but the task stopped and needed a Retry > - The runner log shows the cause. The resumed Codex thread sent the usage of the lost turn after it accepted the continuation turn. The runner rejected that usage as `turn_binding_mismatch` and failed the run > - Recovery marks the lost turn as settled in the durable ledger. It does not tell the live Codex provider, so the provider treats the old turn's usage as a foreign event > - This pull request records the lost turn as settled in the live provider too > - The benefit is that a restart-lost Codex turn continues without a Retry ## Linked Issues or Issue Description Refs #19699, #19608, #19501, #15297 **What happened?** Restart storm 5 (seed 7106). Task t01 (Codex, B5) crossed one K2 SIGTERM restart. Recovery resumed the dead runner. Runnerd restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (recoverable). The control plane started the continuation (#19699). Runnerd accepted it as a new provider turn. About 0.5 s later, Codex sent `thread/tokenUsage/updated` for the lost turn id. The runner's durable state records these events: - `turn.accepted` for the new turn `01a1b3e2-…` - `harness.diagnostic` `turn_binding_mismatch`, method `thread/tokenUsage/updated`, `expectedTurnId` = new turn, `receivedTurnId` = lost turn `01a1b3e0-…` - `turn.failed` (`recoverable: false`) and `run.terminal` failed The control plane did not see `turn/started` and failed after 30 s with "runnerd did not report the provider turn identity". The task needed a manual Retry. **Expected behavior** Usage for the lost turn is history. The runner keeps it as a resume usage snapshot. The continuation runs, and the task continues without a Retry. **Steps to reproduce** 1. Start a Codex turn through the durable runner. 2. Kill the runner while the turn is active. Restore so that the provider reports no active turn (`provider_turn_lost_on_restore`). 3. Start the continuation turn. Let the provider send `thread/tokenUsage/updated` for the lost turn after it accepts the new turn. 4. Before this change, the runner fails the continuation with `turn_binding_mismatch`. ## What Changed - `provider_backend.rs`: when recovery finds that the previous active turn ended while the runner was offline, it now also restores that turn id into the live provider's settled-turn set. The durable ledger already settled it. The existing settled-turn path then turns its late usage into `paperclip/resumeUsageSnapshot` and does not bill its `last` usage to the run. - The code skips this step when the settled-turn epoch needs a rollover, the same as the durable ledger. - Fake Codex: new `--stale-usage-after-turn-start` flag. With durable turn ids, it sends usage for the previous turn right after it accepts the next turn, before `turn/started`. This matches the live Codex order. - New test `restart_continuation_ignores_late_usage_for_the_turn_lost_on_restore` in `tests/codex_provider.rs`. It uses the real durable Codex executor with fake Codex. ## Verification - Red before the fix: `cargo test -p paperclip-runner-core --test codex_provider restart_continuation_ignores_late_usage` failed with the same diagnostic as the live run (`turn_binding_mismatch`, method `thread/tokenUsage/updated`, `receivedTurnId: provider-turn-1`, `expectedTurnId: provider-turn-2`). - Green after the fix: the same test passes. The continuation emits `turn.started` for `provider-turn-2`, there is no `turn.failed` or `run.terminal`, and fake Codex saw exactly two `turn/start` calls (the original and one continuation). - `cargo test -p paperclip-runner-core`: all suites pass. `acpx_park_readiness::hosted_park_acpx_preserves_late_goals_and_rejects_foreign_or_busy_observations` failed once and passed on rerun. This change does not touch ACPX. - `cargo fmt --check` is clean. - Not run: the live restart-chaos replay of `w-20261107T023921-s7106/t01`. The local harness needs provider credentials that were not available to this session. ## Risks - Low risk. The change adds one already-ended turn id to the live provider's settled set. That set is only used to ignore stale events and to reject reuse of a settled id. So a provider that later reports the lost turn id as a new active turn is now rejected, as the durable ledger already does. - The change does not address a second, separate problem: after the runner failed the continuation, the control-plane transport waited the full 30 s turn-start bound and reported "runnerd did not report the provider turn identity" instead of the runner's `turn_binding_mismatch`. That message only applies when such a failure occurs. This PR removes this specific trigger. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
scotttong
|
f5fb8ada3b | Restore chat contacts, agent pins, and scrollable recent navigation (#19685) |