766 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude	08ff0828d6	fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 00:12:41 -06:00
DottaandClaude	16024ffa2a	fix(native): a same-boot stall never leaves a run waiting on itself (#20721) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - A second gap: after back-to-back stalls mid-tool, the executor returns and nothing in this process ever looks at the run again. No terminal state, no recovery and no Retry. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - It also makes heartbeat schedule that recovery as soon as such an executor exits. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode, T1, TM1). Three stalls of about 90 s came back to back, with less than 1 s between them. Each run was in a tool call. After the second thaw, `native.session.execute` failed for all three runs (lease lost, executor detached). After that, no run had any event: no terminal state, no `native.recovery.transition` and no Retry, for 20 minutes. The in-sandbox turns kept going. A single 90 s stall (wave s9340) stays green, because a later recovery signal happens to find the run. With stacked stalls, that signal can land while the executor is still tearing down, so it skips the run and sets no deadline. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat.ts`: when the executor ends with `NativeControllerDetachedForRestartError`, there was no shutdown or hot-restart handoff, and this process's own lease lapsed (`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery for the run when the executor exits. Recovery then reattaches the remote runner whose turn is still running, restarts an unstarted bootstrap, or fails the run with a working Retry. - `claimNativeRestartRecoveries`: a self-owned lease that is still live, with no driver here, now returns a `retryAt` at its expiry, so the wait is bounded. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Heartbeat test: an executor detaches after its own lease lapsed, with no restart. The same process must take the run back. Red before (timed out, the run stayed `running`), green after. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - Daytona wave `w-20261111T050504-s9331` from this branch (same seed and shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about 1 s between them). All 3 tasks reached `done` with no Retry. Run `632f0564` (claude) shows the exact s9331 sequence: `native.session.execute` failed after 216 s, then `reattach_remote_runner` 0.4 s later (before the third stall), then a second reattach after it, and the turn completed. Two tasks are graded FAIL only on the T1 crossing check: their own fault landed after their tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their status is `done`, with no Retry. - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 23:55:10 -06:00
DottaandClaude	200c6e40e2	fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run's first provider start fails before any session opens, Paperclip retries the bootstrap on the same run. Durable controller evidence proves that no provider session or provider event exists. > - A remote sandbox keeps the Runner state inside the workspace. A replacement sandbox gets that state back when the saved workspace is restored. > - When the provider refused `session.open` (for example an Anthropic 429), the Runner still suspended cleanly. Its state says `suspended`, but the provider state has no session identity. > - The same-run restart handled a Runner that never suspended. It did not handle a Runner that suspended before the provider opened a session. It failed with `runner_harness_state_mismatch: provider_identity_incomplete (provider_not_suspended)`, and the task showed "Recovery needed" without Retry. > - This pull request lets that proven restart set the unopened session aside and start the provider again. > - The benefit is that a rate-limited provider start continues by itself after the limit resets. ## Linked Issues or Issue Description Refs #20627 and #20683 (other restore paths). This PR changes only the same-run bootstrap restart. **What happened?** Two ACPX Claude tasks on remote Daytona sandboxes, on staging: 1. The first attempt sent `session.open`. ACPX rejected it with `classification=provider_rate_limit` (Anthropic 429). The Runner exited with `lifecycle=suspended`. 2. Paperclip recorded `recoveryMode: bootstrap_retry`, `providerSessionEstablished: false`, `providerEventsExist: false`, and a retry after the limit reset. 3. One hour later the retry ran as `bootstrap_incomplete` on the same run, in a new sandbox with the saved workspace restored. 4. Host preparation inspected the restored Runner state. The Runner was `suspended`, but the provider had no session identity. The inspection threw `provider_identity_incomplete (provider_not_suspended)` before the bootstrap restart branch ran. 5. The run failed. The task stopped at "Recovery needed" with only inspect actions. **Expected behavior** The restart sets the unopened session aside and opens a new provider session, as it already does for a Runner that never suspended. **Steps to reproduce** 1. Restart a remote run as `bootstrap_incomplete` with a proven bootstrap restart. 2. Give the sandbox a Runner state for this run with `lifecycle: suspended` and a provider state with no session identity. 3. Prepare the host. On `master` it throws `runner_harness_state_mismatch: provider_identity_incomplete`. The new `interrupted_bootstrap_unopened` cases of `bootstraps only an untouched provider session in a $outcome workspace lease` run these steps for a created and a resumed lease. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - During host preparation, a Runner state that is suspended with no provider session identity returns the new incomplete reason `provider_unopened`, but only when all of these hold: - the run is restarting as `bootstrap_incomplete`; - the state belongs to this same run; - the controller already recorded the bootstrap restart proof (its durable journal shows no provider turn, and the old Runner is stopped or absent). - the provider state is the one saved before any session opened (`remoteProviderStateNeverOpened`): `prepared`, no session or identity, no turn in progress, no unconfirmed exit. Only ACPX and Codex-family states qualify. - The created-lease and resumed-lease restart branches share one check (`restartableBootstrapState`) and accept `provider_unopened` exactly like `not_suspended`. They move the old session aside and claim a new, untouched session root. - Every other case still throws `provider_identity_incomplete`: another run's state, a restart without the proof, a normal resume, and close checkpointing. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`: 47 passed. - Red on `master`: the `created` and `resumed` `interrupted_bootstrap_unopened` cases fail with `runner_harness_state_mismatch: provider_identity_incomplete (identity_binding)`. Green with this change. - Fail-closed cases stay red-proof: `interrupted_bootstrap_unopened_unproven` (no restart proof) and `interrupted_bootstrap_unopened_other_run` (another run's state) still throw. - Full `native-session-executor.test.ts` and `pnpm --filter @paperclipai/server typecheck` pass. ## Risks - Low risk. The new branch needs the same proof that already authorizes a fresh bootstrap for a Runner that never suspended. The proof comes from the controller's durable journal, not from the sandbox. - The normal path is unchanged and has no added wait. - Not proven live: I did not confirm on staging that the controller recorded the restart proof for these runs. If it did not, the run still fails closed as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 21:35:38 -06:00
DottaandClaude	a46b9ac724	fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs (Paperclip Runner with an ACP harness, such as Claude) can stop because the provider account runs out of its usage allowance > - The ACP harness reports this as a "terminal limit failure". Paperclip blocked the task behind a board hold that called the session state ambiguous, and the only way forward was a manual Retry > - The provider also says when the limit resets, for example "resets Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only resets ("resets 2am"), so the agent-level wait fell back to one hour. Each later wake for the agent then hit the limit again and blocked another task > - This pull request reads calendar resets, says on the hold when the limit resets, and requests the task's Retry itself at that time when the reset is reported and bounded > - The benefit is that a usage limit no longer needs a person to click Retry, and the agent's other tasks wait instead of failing one by one ## Linked Issues or Issue Description **What happened?** A native ACP run failed with `native_provider_usage_limit` ("ACP agent reported a terminal limit failure. You've hit your weekly limit · resets Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold. The hold said "Inspect the original provider failure and explicitly resolve the ambiguous session state". Nothing resumed the task after the reset. The agent-level `provider_limit_wait` recorded "no reset time was reported, so Paperclip waits one hour", so every hour another wake dispatched, hit the limit, and blocked another task. In one 48-hour window on a staging instance, this blocked 7 tasks for one agent. **What did you expect?** When the provider reports a bounded reset, the task waits and continues after the reset, like the existing `provider_limit_wait` path. When the reset is unknown, the task fails with a clear message and a working Retry. **Steps to reproduce** 1. Run a native ACP agent until its provider reports a weekly or session limit with a reset time. 2. Look at the task: it is blocked with an "ambiguous session state" hold. 3. Wait past the reported reset: the task stays blocked until a person clicks Retry. Refs #20666 (other recovery work in the same staging census; not changed here) ## What Changed - `provider-quota-reset.ts`: parse calendar resets such as "resets Nov 12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and "Resets on August 2, 9am (UTC)". A past date rolls into next year. An impossible date (Nov 31) is refused. Clock-only parsing is unchanged. This also fixes the agent-level `provider_limit_wait` reset and the legacy `provider_quota` reset for these messages. - New `native-runtime/provider-limit-resume.ts`: - `readNativeUsageLimitReset` reads the reset only from the run's own durable events. The last turn event must be a provider-observed `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure, and the reset must be at most eight days out. - `requestProviderLimitResumeRetry` asks for the task's Retry (`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of the user who requested the run, or the run's accountable user for automatic wakes. It is idempotent per run and records its outcome on the hold. At most three automatic Retries follow one another without a successful run. - `registerProviderLimitResumeWork` is a scoped deadline owner (shared work scheduler, no polling). Each active usage-limit hold is due at its reset. Startup rebuilds it from the holds, and the executor notifies it in the hold's transaction. - `native-session-executor.ts`: for a terminal usage limit, the hold and run failure detail say when the limit resets and whether Paperclip retries then, instead of "ambiguous session state". The evidence records `providerLimitResetAt`. - `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its activity is recorded as system actor `provider-limit-reset` on behalf of the user, like `restart_recovery`. - `doc/execution-semantics.md`: document the wait and its limits. Previous-release story: holds saved by the previous release carry forward. At startup the owner reads each active `native_provider_usage_limit` hold. It rewrites the hold's message with the reset when the reset is known and still ahead, and it requests the Retry at the reset. Holds whose reset is unknown, more than a day in the past, or that have no accountable user keep their Retry and are recorded as not requested, so they are not read again. ## Verification - `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4 tests. Three fail on `origin/master` code (calendar resets return `null`), all pass with this change. - `npx vitest run server/src/services/native-runtime/native-provider-capacity.integration.test.ts` — 5 tests on embedded Postgres: - the existing usage-terminal test (both replay modes) still blocks with a Retry, and no automatic Retry is requested without a reported reset; - "waits for a reported usage-limit reset and then requests the task's Retry once": the hold says when the limit resets, an old-release hold is rewritten, nothing starts before the reset, one Retry receipt is created for the accountable user at the reset, and a second pass does nothing. Fails on `origin/master` executor code; - "starts the Retry from the hold's own deadline": a real delivery coordinator rebuilds the deadline from the hold and calls Retry dispatch once, after the reset; - "leaves a visible Retry when no user is accountable for the run". Fails on `origin/master` executor code. - Focused regression suites pass (642 tests): `task-retry`, `quarantined-task-retry`, `heartbeat-native-restart-suspension`, `native-startup-retry.integration`, `provider-failure-classification`, `heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`, `chat-run-publications`, `native-runner-restart-recovery.integration`, `native-remote-retry-admission`. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - `pnpm verify` selected a broad run (exit 2, because the docs file is a global CI input). CI supplies the broad coverage. ## Risks - The automatic Retry uses the accountable user's authority for automatic (system) wakes. It is the same Retry the hold already offered that user, and dispatch still checks Retry authority, stop proof, saved work and action reconciliation. It is never broader than a click. - A provider that keeps reporting resets that do not restore capacity gets at most three automatic Retries in a row; then the Retry stays visible. - The deadline owner reads all active usage-limit holds at startup (limit 500 rows). The number of such holds is small. - Not changed: the agent-level wait still holds new wakes for at most one day, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 20:57:03 -06:00
DottaandClaude	a1ec05a1a4	fix(native): read settled harness state again after a dropped sandbox connection (#20690) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one provider conversation across the per-turn runs of a task. After each turn, the controller reads the settled Runner and provider state out of the sandbox and saves it as the harness backup. > - A per-turn sandbox is released after the turn. The harness backup is then the only copy of the provider session. > - On staging, one dropped sandbox connection (`write ECONNRESET`) during that read lost the backup. The run itself succeeded, because its result was already accepted. > - The next run of the task could not resume the conversation: `PRP provider resume state is unavailable`. When the task was waiting for a reply, the run stopped at "Recovery needed" with no Retry. A person had to send a message to start a new conversation. > - This pull request reads the settled state again when the sandbox connection drops during that read. > - The benefit is that one network reset no longer loses a task's conversation. ## Linked Issues or Issue Description Refs #20627 (warm backup restore when the source sandbox is idle) and #20683 (relaunch of a settled runner). This PR changes a different step: the backup capture at the end of a turn. **What happened?** A per-turn OpenCode task ran on a remote Daytona sandbox with reuse disabled. 1. Run A accepted its result. The Runner suspended. 2. The controller read the settled state out of the sandbox. The read failed with `remote checkpoint failed: write ECONNRESET`. `harness_state.backup.persist` and `session.checkpoint.persist` failed. 3. Run A still finished as `succeeded`, because the result was durable. The sandbox was released. No harness backup of turn A existed. 4. Run B started in a new sandbox. It failed in 0.2 s with `native_session_recovery_failed: Error: PRP provider resume state is unavailable`. When the prior turn yielded for a reply, the checkpoint policy does not allow a replacement session, so the task showed "Recovery needed" with only inspect actions. On staging this happened on 4 tasks in 48 hours. In all 3 predecessor runs that I could inspect, the backup capture failed. 2 of them show the `ECONNRESET` in the run log. No deploy happened between run A and run B in one case, so a deploy is not the cause. **Expected behavior** A transient connection drop while reading settled state does not lose the provider conversation. The read is repeated, and the backup is saved. **Steps to reproduce** 1. Run the first durable turn of a remote OpenCode session. 2. Make the sandbox `syncOut` of the backup directories fail once with `ECONNRESET`. 3. Call the settled checkpoint. On `master` it throws `write ECONNRESET`, and no backup is saved. The new test `uses the bundled Linux image authority for remote opencode-new-durable-reset without controller overrides` runs these steps. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - The settled-state capture (inspect plus `syncOut` of each persistence directory into a pending directory) is now one step that can repeat. The step writes nothing durable, so it is safe to repeat. - A dropped, reset or refused sandbox connection repeats the step, at most 3 attempts, with 1 s and 2 s delays. Each repeat clears the pending directory and inspects the live state again, so the copy always matches the inspected identity. - Any other failure fails at once, as before: a state mismatch, a missing directory, an unsafe archive, or a TLS failure. - The durable backup upload and the local publish run only after a complete capture. They did not change. - `isTransientRemoteCheckpointReadFailure` classifies the error with the existing `classifyRemoteConnectionError`, plus a message match for SDK errors that lose their `code`. - The first completeness inspection and its visibility-window re-reads use the same bounded retry (`retryTransientRemoteCheckpointRead`). A dropped provider-state read keeps its `provider_state_unreadable` refusal for every other caller, but the checkpoint retry looks through it to the dropped read. Undici `fetch failed` and `no response returned` also count as drops. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts`: 1343 passed. - Red on `master`: the new `opencode-new-durable-reset` case fails with `Error: write ECONNRESET`. Green with this change. - The `isTransientRemoteCheckpointReadFailure` test checks that state mismatches, unsafe archives and TLS failures do not repeat. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. The normal path has no added wait. A repeat adds at most 3 s, and only after a connection failure. - A repeated read inspects the live state again. If the state changed between attempts, the existing identity checks still refuse it. - A connection that stays down for all 3 attempts fails the same way as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 20:46:29 -06:00
DottaandClaude	06af074452	fix(native): deliver the retry after a failed native turn, once per chain (#20681) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime decides a task's next status after each run, through the status arbiter and the status-decision committer > - When a native turn fails without a classified provider error, the arbiter keeps the task's status and asks for one retry (`schedule_retry`) > - The native-status wake dispatcher refused that retry wake, because its provenance check accepted only `enqueue_continuation` effects > - So the task stayed `in_progress` with no live run, no queued wake and no visible cause, until a person noticed it > - This pull request lets the dispatcher accept the committed retry effect, keeps the retry on its chat route, and limits the automatic continuation to one per chain > - The benefit is that a failed turn always ends with either queued work or a blocked task with a working Retry ## Linked Issues or Issue Description **What happened?** A native run that ends with a failed turn and no classified provider error leaves its task stranded. Examples: an OpenCode server that the sandbox killed with SIGKILL during a turn (`opencode_server_exited`), or an OpenCode turn that ended without an answer (`opencode_empty_response`). The run's accepted result is finalized as `run_failed_partial_evidence_preserved`. The committer queues a `schedule_retry` wake. The dispatcher then skips the wake with `chat_control_recovery_proof_unresolved`. The task stays `in_progress` with no run, no wake and no blocker. **Steps to reproduce** 1. Run a native task. Make the provider turn fail with `opencode_server_exited` or `opencode_empty_response` and an accepted `needs_review` result. 2. Finalize the run. 3. Run the native-status wake dispatcher. 4. See that the retry wake and its dispatch receipt are `skipped`, and that no run starts. **Expected behavior** The task gets its automatic continuation. If that continuation fails the same way, the task is blocked with a clear board blocker and a Retry that starts a new run. **Paperclip version or commit** `origin/master` at c444da7d05. **Additional context** The task stays `in_progress` for hours with no owner. On a staging deployment, two tasks stayed in this state for more than 7 and 11 hours. ## What Changed - `server/src/services/chat-control-recovery-stop.ts`: the native-parent proof accepts the `schedule_retry` effect, as well as `enqueue_continuation`, when the effect targets the wake intent. Other effect kinds (for example `notify_owner`) are not changed. The other checks (applied decision, delivered effect, same company, task and agent) do not change. - `server/src/services/native-runtime/status-arbiter.ts`: new input `failedTurnContinuationUsed`. When it is true, a failed run is decided as `native_run_failed_continuation_exhausted`: the task is blocked with a board blocker and no wake. New constants `NATIVE_RUN_FAILED_RETRY_CAUSE` and `NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION`. - `server/src/services/native-runtime/native-failed-turn-continuation.ts` (new): `isFailedTurnContinuationRun` reads the run's dispatcher-set `nativeStatusWakeIntentId` and proves from the intent row (company, agent, task, committer actor, `retryCause`) that the run is the automatic continuation of a failed turn. A copied or unknown id does not count. - `server/src/services/native-runtime/native-run-finalizer.ts`: passes `failedTurnContinuationUsed` to the arbiter for failed runs that the run still owns; the committer rechecks that ownership under the issue lock (`requireProviderFailureOwner`). - `server/src/services/native-runtime/native-continuation-delivery.ts`: the failed-turn retry keeps the failed run as its chat conversation source, so its answer reaches the original chat. - `doc/execution-semantics.md`: documents the retry wake, its provenance and the one-continuation bound. - New test `native-failed-turn-continuation.integration.test.ts`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-failed-turn-continuation.integration.test.ts` - RED on `origin/master` (c444da7d05): the dispatcher reports `dispatched: 0` and the wake is skipped with `chat_control_recovery_proof_unresolved`. - GREEN with this change: for both `opencode_server_exited` and `opencode_empty_response`, the dispatcher starts one continuation run. When that continuation fails the same way, the task is blocked with the board blocker, no new wake is queued, and the task Retry is accepted and starts a new run. A copied wake id does not exhaust the continuation. - Related suites pass: `status-arbiter.test.ts`, `native-provider-failure.integration.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `chat-control-admission-retry.test.ts`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` selects broad verification (exit 2) because `doc/execution-semantics.md` changed. Required CI supplies that coverage. ## Risks - Behavior change: a failed native turn now gets the automatic continuation that the arbiter already requested. That continuation is a new run of the same task, with the normal admission gates. It does not replay the failed turn. - The continuation is bounded to one per chain, so a deterministic failure cannot make a wake loop. - Tasks that are already stranded by the old behavior keep their skipped wake. Their task Retry still works. - Upgrade safety: an old pending `schedule_retry` intent is accepted by the new dispatcher. A run that started before this change has no proven intent, so it gets one continuation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, read-only API access). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 20:10:48 -06:00
DottaandClaude	7435abcc63	fix(native): wait out a run-event append before saving provider-pack admission (#20682) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run in a remote sandbox records a provider-pack admission receipt on its run row after the provider session opens. Artifact retention then requires that receipt. > - `recordNativeProviderPackAdmission` locked the run row with `FOR UPDATE NOWAIT`. > - A run-event append locks the same run row for a few milliseconds (`FOR NO KEY UPDATE` since #20670, `FOR UPDATE` before). Both conflict with `FOR UPDATE`. Startup appends many events at the same moment. > - When the two met, the admission write failed with `55P03`. The admission state stayed `pending`. Artifact retention then failed the run with `native_provider_artifacts_admission_unverified`. The task stopped as "Recovery needed" with no Retry. > - This pull request makes the admission write wait for the run-row lock within the existing 1000 ms `lock_timeout`. The coordinator claim keeps its `NOWAIT` fence. > - The benefit is that an ordinary concurrent event append no longer fails a healthy run before its first turn. ## Linked Issues or Issue Description Refs #20670 (changed the append lock mode; the conflict with `FOR UPDATE NOWAIT` remains). **What happened?** On a hosted staging instance, three OpenCode runs on remote sandboxes failed in the first minute with `native_provider_artifacts_admission_unverified`. Each run log shows `native provider-pack admission receipt could not be saved` about 0.2–0.7 s after the runner authenticated. The run profile kept `nativeProviderPackAdmissionState.state = "pending"`. The tasks stayed blocked. **Steps to reproduce** 1. Start a native run and hold its `heartbeat_runs` row lock in another transaction, as a run-event append does. 2. Call `recordNativeProviderPackAdmission` for that run. 3. Observe `could not obtain lock on row in relation "heartbeat_runs"` (`55P03`). **Expected behavior** The admission waits a short, bounded time for the append to commit, then saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` sets `lock_timeout = 1000ms` for every call, not only `waitForClaim` calls. - The run-row lock is now `FOR UPDATE` without `NOWAIT`, so it waits for a concurrent append within that bound. - The coordinator (`native_run_finalizations`) lock keeps `NOWAIT` when the caller does not wait for the claim. The existing fence tests still pass. - New PostgreSQL test: `waits for a concurrent run-event append's run-row lock instead of losing the admission` (both `waitForClaim` modes). ## Verification - RED on `origin/master` (`c444da7d05`): the new test fails with `55P03` in both modes. - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts` (37 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Low risk. The normal path has no contention and does not wait. Under contention the admission waits at most 1000 ms, before any provider turn. - A lock wait between the coordinator row and the run row can now meet a deadlock. PostgreSQL detects it and raises an error. That error takes the same caught path as the earlier `55P03`, so the result is no worse than before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 19:39:42 -06:00
DottaandClaude	6e524d8a69	fix(workspaces): stop retrying a copy-back that hit a full sandbox disk (#20678) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a native run, Paperclip copies the agent's sandbox workspace back to durable storage. This is finalization. > - The copy-back builds an archive inside the sandbox. When the sandbox disk is full, the copy-back fails with `remote_workspace_sandbox_disk_full`. > - Finalization treated that failure as transient. It retried three times. Each attempt took about 15 minutes for a large workspace, and nothing freed space between attempts. > - Saved-work recovery already reclaims the retained sandbox's inactive provider cache when a save hits a full disk. It runs only after finalization ends. > - This pull request ends finalization on the first full-disk copy-back failure, with the accepted result kept, so saved-work recovery starts at once. > - The benefit is that the task gets its files saved sooner, and the user sees what happened instead of "failed after 3 attempts". ## Linked Issues or Issue Description Refs #19619 (related: records why the read-only capture fallback failed; this PR does not change that path) **What happened?** On staging, a native run finished its turn and its result was accepted. The workspace copy-back then failed three times with `remote_workspace_sandbox_disk_full` (about 7.5 GB of workspace, about 6 GB free). The three attempts spanned about one hour across deploys. Finalization then ended as `native_workspace_sync_out_retry_exhausted`. Saved-work recovery reclaimed the inactive provider cache (about 3 GB freed) and saved the files. **Expected behavior** A full sandbox disk is recognised on the first failure. Paperclip does not spend the finalization budget on attempts that cannot succeed. The agent's result is kept. The user sees that the disk is full and that Paperclip saves the files again without another provider turn. **Steps to reproduce** 1. Run a native task on a remote sandbox whose workspace is larger than the free sandbox space. 2. Let the provider turn finish. 3. Observe three copy-back attempts that each fail with `remote_workspace_sandbox_disk_full`. ## What Changed - `recordRetryableFailure` treats a workspace failure whose cause starts with `remote_workspace_sandbox_disk_full:` as an exhausted budget on the first attempt. The code stays `native_workspace_sync_out_retry_exhausted`, so the existing export hold, saved-work recovery, and export Retry apply unchanged. - The run error for that case says that the sandbox disk is full and that Paperclip saves the files again without repeating the agent's work. It no longer says "after 3 attempts". - Other copy-back failures keep their bounded retry. - `captureRemoteWorkspaceArchive` rethrows the read-only fallback's own failure when that fallback only lost its connection (`transport_closed`, `provider_unavailable`, `provider_rate_limited`, `sandbox_state_transition`). That failure is not disk-full, so finalization keeps its bounded retry for it. Other fallback failures keep the disk-full code. - `remoteWorkspaceFailureCause` records the server-built read-only capture failure, so a failed save names where the fallback stopped. - The transport exports `REMOTE_WORKSPACE_SANDBOX_DISK_FULL_CODE`. The finalizer and tests use it. - `doc/execution-semantics.md` describes the new rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-run-finalizer-telemetry.test.ts src/services/remote-workspace-transport.test.ts` (195 pass). - New test "hands a full sandbox disk to saved-work recovery on the first copy-back failure" fails on `origin/master` (the failure stays `retryable_failure`) and passes with this change. It completes an accepted result first and checks that the result stays linked and that no agent wakeup is created. - New transport test "keeps a full disk only when the read-only fallback did not just lose its connection": the two dropped-connection cases fail on `origin/master` and pass with this change. - New test "keeps other copy-back failures on their bounded retry" passes before and after. - `cd server && npx tsc --noEmit` passes. `pnpm verify` selected 2504 test files because the finalizer is widely imported; its first selected batch passed, and required CI supplies the full run. ## Risks - Low risk. The terminal state is the same state that three failed attempts produce today. Only the timing changes. - A sandbox whose disk frees space by itself between attempts now waits for saved-work recovery instead of the next finalizer attempt. Saved-work recovery repeats the save with its own bounded attempts, so this does not lose work. - Upgrade safety: a run saved by an older release in `retryable_failure` with this cause ends on its next full-disk failure. No migration is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. Extended reasoning was not configured. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 19:31:33 -06:00
DottaandClaude	2eede8b318	fix(native): give a relaunched settled runner its admitted provider pack (#20683) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a server restart or a same-run retry, a remote native run reattaches to its runner in the sandbox (`reattach_remote_runner`). > - When the controller saved the runner artifact, the executor prepares that run as an adoption. It skips provider-pack selection, because a live adopted runner keeps its launched provider. > - The reattach probe can instead find the runner settled and exited (`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the transport relaunches the runner from its durable state. That relaunch must start the provider again, so it needs the provider-pack launch inputs. > - The executor did not give them. OpenCode failed with "OpenCode executable is missing from the provider pack". ACPX failed with "ACPX launch profile omitted its provider-pack authority or verified artifact digests". Both ended the run. > - This pull request selects the saved (admitted) provider-pack authority on that settled-exit relaunch and verifies it as an exact installed identity, as an ordinary reattach does. > - The benefit is that a restart or a same-run retry can resume an exited runner with the same provider pack, instead of failing. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, two OpenCode runs and one ACPX (Claude) run were interrupted by a deploy (SIGTERM) or ended a turn without a result. The server scheduled a same-run retry that reattached to the existing remote runner. The runner had exited cleanly (`runner_remote_process_exited lifecycle=suspended`). The retry then failed within three seconds with `runner_remote_provider_artifact_incompatible`. **Steps to reproduce** 1. Prepare a remote OpenCode or ACPX run with a persisted runner artifact (`controllerPersistence.runnerArtifact`). 2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`, where the runner process is not running and its state is `suspended` and settled. 3. Observe that the transport options have no `adoptExistingRunner`, no `providerPackAuthorityDigest` and no `opencodeCommand` / `acpxSidecarSha256`. **Expected behavior** The relaunch gets the admitted provider pack's launch inputs, or it fails with a clear pack-identity error. ## What Changed - `createRunnerdBackend` (native session executor): when a persisted-runner reattachment finds no live runner to adopt, select the saved provider-pack authority (`reattach: true`), so `expectedProviderPackManifest` is the admitted pack. - `prepareRemoteRunnerAssets` no longer returns early for that relaunch. It verifies the installed pack as an exact identity and sets the active pack root, the same path as a reattach without a persisted artifact. - A live adopted runner is unchanged: it still skips pack selection. - The test `uses the bundled Linux image authority for remote opencode|claude-x64` now also covers the settled-exit relaunch for OpenCode and ACPX. ## Verification - RED on `origin/master` (`c444da7d05`): the extended test fails for `opencode` and `claude-x64` (the relaunch options have no provider-pack inputs). - GREEN with this change: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "uses the bundled Linux image authority"` (14 passed). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify`: see the result below. ## Risks - Upgrade safety: the relaunch reads the saved authority, so a newer controller does not select a newer pack for an older session. It requires an exact installed identity. If the sandbox no longer has that pack, the relaunch fails with a pack-identity error, and the existing Retry path applies, as before. - No change to a live adoption, to fresh runs or to the normal path before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 19:14:43 -06:00
DottaandClaude	e33ede9a8a	fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 19:05:15 -06:00
DottaandClaude	cec903c6f0	fix(native): retire a finished remote owner whose gone Runner left a delivery unacknowledged (#20677) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native follow-up run on a remote sandbox first proves that the previous per-turn owner finished. Then it retires that owner's controller journal and starts a new Runner. > - The proof requires a settled journal. A command that the Runner did not acknowledge holds the owner. > - On staging, a per-turn owner committed its result and its Daytona sandbox was destroyed after the turn (`paperclip_runner_destroy_after_turn`, cleanup `success`). Its journal still held a `run.attach` or a `semantic_tool.result` that the Runner never acknowledged. > - The next run failed with `prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(run.attach.pending)` (or `semantic_tool.result.pending`). A Retry or a new message gets the same journal, so it fails the same way. The task stays blocked. > - Retirement already excuses the Runner's own lifecycle commands when the prior sandbox lease ended on another sandbox. This pull request applies the same rule to deliveries that only that gone Runner could receive. > - The benefit is that these follow-ups start by themselves. Commands that carry user intent still hold the owner. ## Linked Issues or Issue Description **What happened?** A census of staging failures (48 h, Nov 9–10) found 6 failed runs with `prior_owner_terminal_state_indeterminate` and `retirement_refused=journal_commands_unsettled(run.attach.pending | semantic_tool.result.pending)`. The last one was at 18:56 UTC on Nov 10. In two examples that I checked: - The previous run succeeded. Its result was committed and its coordinator lease was released. - Its lease policy was `ephemeral`. The lease is `expired`, with `failureReason=paperclip_runner_destroy_after_turn` and `cleanupStatus=success`. - The follow-up got a new sandbox (a different provider lease) and refused in about 11 s. It started no Runner. The task became `blocked`, and the next action was "Send a message to continue". **Expected behavior** When the record proves that the previous Runner is gone, an undelivered attachment, tool answer or read cannot run. The follow-up retires the finished owner and starts a new Runner. Retirement replays no command. **Steps to reproduce** 1. Finish a per-turn remote native run whose controller journal keeps a `run.attach` or `semantic_tool.result` command in `pending`. 2. Let the sandbox lease of that run end (`expired` or `released`) on its own sandbox. 3. Start a follow-up run that gets a new sandbox. 4. Observe `retirement_refused=journal_commands_unsettled(run.attach.pending)`. **Paperclip version or commit** Staging builds of Nov 10. The refusal still reproduces on `master` at `48c6e996da` (the new tests fail there). ## What Changed - `native-session-executor.ts` (`readSettledRunnerdProjection`): when the prior run's sandbox leases ended on another sandbox (`remoteRunnerGone`, the existing proof), unfinished `run.attach`, `semantic_tool.result`, `session.snapshot` and `session.goal.get` commands no longer hold the retirement. These join the lifecycle commands that the rule already excused. - In the same case, a journaled `semantic_tool.result` answers its `semantic_tool.input`. The controller journals the answer after the tool ran. Only its delivery was outstanding. - No other proof changes. The prior run must still be terminal, with a committed result, a released coordinator lease, no recovery state, a stopped or fenced controller, a delivered `run.terminal`, and no live work after it. Quarantine copies must pass the same checks. - `doc/execution-semantics.md`: describe the rule. ## Verification - `server/src/services/native-runtime/native-recovered-owner-followup.test.ts` (real Postgres): - New: "retires a finished remote owner with an unacknowledged pending-attach / pending-tool-result when its sandbox lease ended on another sandbox". It fails on `master` with the exact staging errors `journal_commands_unsettled(run.attach.pending)` and `journal_commands_unsettled(semantic_tool.result.pending)`. It passes with this change, and the log shows the retirement. - Held cases stay held: a lease that is still retained, a lease on this run's sandbox, no lease record, and a tool input with no journaled answer (`journal_tool_input_unanswered`). - New: "keeps a stopped local owner held by an unacknowledged run.attach". A local Runner is not gone by record. - Whole file: 95 passed. - Neighbouring suites pass: `explicit-native-continuation.test.ts` and `native-controller-recovery.integration.test.ts` (460 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2: a global CI input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Live proof: not done. The redeploy chaos scenario (K15) is not on `master` yet. Tests only. ## Risks - The rule applies only when every sandbox lease of the prior run is `released` or `expired`, has a release time, and names a different provider lease than this run's. A reusable sandbox that this run resumed keeps the existing hold. - A command that carries user intent (`turn.start`, `turn.steer`, `request.resolve`, `interaction.receipt`, `external_provider.operation`, `session.goal.set`/`clear`, `session.close`, `run.cancel` and others) still holds the owner. - Retirement archives the old journal under `continuity-breaks` and replays nothing, as before. - No persisted shape changes. Old saved journals are read by the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 18:49:26 -06:00
DottaandClaude Opus 5.5	e9db5e0d03	perf(github): read the inherited review waits of an issue list in one pass (#20673) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can review a GitHub pull request. After the agent publishes the review, the task can wait in review for the next GitHub reply. A later run of the same task can inherit this wait. > - Issue lists and inbox attention show these waits. For each waiting task, the server reads the inherited GitHub review wait: the source comment, the latest review, and the review publication. > - The server did these reads one time for each waiting task. A list request with N waiting tasks sent N copies of the prepared review statement and N copies of the publication read. > - On a hosted database, the prepared review statement ran about 270,000 times in about 2.5 hours (up to 87 calls per second). The run identity read before it ran the same number of times. The list read that calls it ran only about 1,400 times, so each list read started about 190 review reads on average. No other caller reads the waits of many runs in a loop. > - This pull request reads the reviews and the publications of all waits of one list request with one statement each. Each wait gets the same result as before. > - The benefit is about 200 times fewer review statements and fewer pool connections for each issue list. ## Linked Issues or Issue Description Refs #20400 (shares review-wait reads between the callers of one request). Refs #20551 (authorizes the waits of one issue page with one read; this change does the same for the inherited GitHub review reads of that page). **What happened** `listExternalReviewWaits` called `readInheritedGitHubReviewWait` one time for each candidate wait, inside `Promise.all`. Each call sent a run identity read, a source-message read, the prepared `inherited_github_review_wait` statement, and a publication read. A list with N waits sent each statement N times. **Expected behavior** One list request sends one review statement and one publication read for each company. The result of each wait does not change. **Steps to reproduce** 1. Put many tasks of one company in review with an inherited GitHub review wait. 2. Open the issue list or the inbox. 3. Count the statements of the database. The review statement runs one time for each waiting task. ## What Changed - `server/src/services/chat-channels/github/review-wait.ts`: - Split `readInheritedGitHubReviewWait` into shared steps: the source comment read, the review checks, the publication key, and the published result. The single-run read uses the same prepared statement and the same checks as before. - Add `readInheritedGitHubReviewWaits(db, runs)`. It reads the source comment of each run with the same lineage check as before. Then, for each company, one `DISTINCT ON` statement reads the latest review for each (source comment, task, agent) key, with the same joins, filters and order as the prepared statement. One read finds the processed publications. Each run gets the same result as a separate call: `null`, `revoked`, or `published`. - `server/src/services/native-runtime/external-review-wait.ts`: `listExternalReviewWaits` uses the batch read. The rest of the list and its authorization do not change. - `server/src/__tests__/github-review-wait.test.ts`: a new test reads 18 runs in one batch and compares each result with a separate single-run read. The runs include each revocation case (connection disabled or inactive, review state, review agent, repository, repository id, thread, publication, identity, company), a newer unreviewed head, no identity, no task, an endpoint of another agent, a run in another company and a duplicate run. The test also makes sure that the batch sends one review statement and one publication read for each company. The existing revocation tests also check the batch read. - `server/src/__tests__/helpers/github-review-wait.ts`: the seed helper can add a wait to an existing company and agent. ## Verification - `pnpm vitest run server/src/__tests__/github-review-wait.test.ts server/src/__tests__/github-review-check-state.test.ts server/src/__tests__/issue-list-review-wait-authorization.integration.test.ts server/src/__tests__/blocked-inbox-review-wait-scope.test.ts` - Focused local run of every test that imports the changed modules or the seed helper (7 files, 335 tests passed), including `external-chat-wait.integration.test.ts`, `external-review-wait.test.ts` and `legacy-execution-recovery.test.ts`. The change to the shared seed helper makes `pnpm verify` select the full server suite, so CI supplies that coverage. - Local measurement on real PostgreSQL (embedded PostgreSQL, `ANALYZE` after seeding). One company and agent, each wait has 5 reviews and 10 outbound messages. "Before" is the previous list code: one `readInheritedGitHubReviewWait` call for each run in `Promise.all`. "After" is `readInheritedGitHubReviewWaits`. The runs alternate order in each pair. Both return equal results in every sample. | Waits | Version | Statements | Review statements | Median ms | Min–max ms | Samples | | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 800 | 200 | 106.8 | 98.0–117.5 | 10 | | 200 | after | 402 | 1 | 44.3 | 41.6–50.9 | 10 | | 20 | before | 80 | 20 | 9.8 | 8.6–14.7 | 12 | | 20 | after | 42 | 1 | 6.3 | 5.3–7.0 | 12 | - `EXPLAIN (ANALYZE, BUFFERS)` of the batch review statement for 200 waits: index scans on each join, 6.7 ms execution. The single statement: 0.05 ms execution and about 2.7 ms planning for each call. - The run identity read and the source-message read still run one time for each wait. They are cheap on the hosted database (about 0 ms each), and they keep the lineage check for each run. ## Risks - Low risk. The batch statement uses the same joins, filters and order as the prepared statement, and `DISTINCT ON` keeps the first row for each key. The array filters can also match a mix of two keys. The code uses only rows of exact requested keys. - The single-run read and its prepared statement do not change. The finalizer, the status committer, and the attachment reuse paths still use it. - Errors do not change: a 403 from the lineage check gives `revoked`, and other errors are thrown. - No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 18:35:39 -06:00
DottaandClaude	c14f2fe45d	fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20645 lets a local same-run scheduled resume send the one same-conversation continuation for a turn lost on restore. It applies only when durable records prove three facts: the previous runner stopped, the attempt was admitted within the budget, and it is the same checkpointed conversation. > - It read the third proof from the coordinator's saved `failure_detail` while the phase was `retryable_failure`. > - In production, the resume reconciler (`claimNativeSessionResumptions`) claims the run first. It moves the coordinator to `observed` and clears `failure_detail`. So the executor never saw that evidence, and #20645 refused every real scheduled resume. > - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the exact path (`runner.drain` timeout, scheduled resume, `provider_turn_lost_on_restore`), sent no continuation, and needed Retry. > - This pull request makes the reconciler record that evidence durably when it claims the run, and makes the executor read it from there. > - The benefit is that #20645 works on the real path. The proof rule is unchanged. ## Linked Issues or Issue Description Refs #20645 **What happened?** Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus #20645), task codex B5 TM2: - `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`; - `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`, `recoverable`, `providerSessionEstablished` and `checkpointExists` all set; - `native.local_process_stopped`; - attempt 2 started; - `turn.failed` with `provider_turn_lost_on_restore` for the same turn ID that the checkpoint was running. No continuation was sent. The run failed with `native_provider_terminal_failed` and was preserved with `uncertain_provider_action`. **Expected behavior** The proven scheduled resume sends the single continuation. **Steps to reproduce** The new integration test below. The live path is timing-dependent (the L7 command in #20645). ## What Changed - `native-finalization-reconciler.ts`: when it claims a same-run resume, the reconciler appends one `same_run_resume` entry to `recovery_history`. The append is bounded like other history writes, and it runs in the same update that clears the failure. The entry copies `recoveryMode`, `recoverable`, `providerSessionEstablished`, `checkpointExists`, `successorRunId` and `replacementDenied` from the saved failure, with the controller boot and provider attempt. - `scheduled-resume-continuation.ts`: the evidence for the third proof now comes from either of two durable records: - the saved failure, when the executor claims a `retryable_failure` coordinator directly (unchanged); or - a reconciler claim: an `observed` coordinator whose last history entry is a `same_run_resume` entry naming the executor's own controller boot. The two other proofs are unchanged: the stop receipt, and an admitted attempt from 1 to 3. - `native-session-executor.ts`: passes the coordinator's last history entry and the controller boot to the decision. - `doc/execution-semantics.md`: describes the recorded evidence. Readers of `recovery_history` that look at the last entry do so only for restart claims, Stop recovery and shutdown suspension. Restart recovery's previous-claim lookup skips non-claim dispositions, so a `same_run_resume` entry does not change those paths. **Still refused** (tested): - every #20645 case; - the last history entry is not the resume claim, or there is no history; - the resume claim was recorded by another boot; - the claimed evidence is a bootstrap retry, has no checkpoint, or records a replacement; - a reconciler-claimed resume without a stop receipt. ## Verification - `native-scheduled-resume-continuation.integration.test.ts` now also runs through the real reconciler claim (`claimNativeSessionResumptions`) before `executePaperclipNativeSession`. On master the reconciler-claimed allowed case fails (`resumeInterruptedTurn` is false). With this change it is true, and the claim admits attempt 2. A reconciler-claimed resume without a stop receipt still passes false. - `scheduled-resume-continuation.test.ts`: decision cases for the reconciler-claim entry. - `native-session-resumption.test.ts` passes. 54 tests pass across the three files. - `cd server && npx tsc --noEmit` passes. - Live: not yet. The failing wave above is the reproduction. A rerun is timing-dependent. ## Risks - One more `recovery_history` entry per scheduled resume, inside the existing 20-entry bound. - The proof rule itself is unchanged. This only reads the same evidence from the record where it now survives. - No schema change. A run claimed by an earlier build has no `same_run_resume` entry, so it keeps the existing failure: refused, with Retry offered as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 18:21:25 -06:00
DottaandClaude Opus 5.5	48c6e996da	fix(native): continue a task whose quarantined completed turn's holds settle (#20662) ... <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a remote run starts, workspace preparation attaches the verified saved work of earlier runs of the same task. > - Preparation failed with `remote_workspace_recovery_receipt_invalid` when any earlier run had a recovery receipt that it could not verify. > - A refusal cannot change the saved state, so every Retry failed in the same way, and the task stayed blocked. Recoveries must fall back automatically. > - This pull request skips an unverifiable source. It attaches nothing unverified and starts from the workspace's newest verified save. > - The benefit is that Retry and continuations start again, while strict receipt checks still decide what is attached, and no saved work is changed. ## Linked Issues or Issue Description Refs #20631 **What happened?** On staging, after the deploy that included #20631, a user pressed Retry on a task whose previous run failed with `native_controller_recovery_missing_evidence:runner_not_settled`. The new run failed in setup in under a minute with `setup_failed` / `remote_workspace_recovery_receipt_invalid` (`executionRecovery: bootstrap`, `providerWorkStarted: false`). The task stayed blocked. Another Retry would fail the same way. #20631 accepted one shape of this refusal (a source that saved only the snapshot its own restore loaded). This task had a different shape. **Expected behavior** A Retry starts. A source whose receipt cannot be verified is not attached, and its saved state stays intact for workspace recovery. **Steps to reproduce** 1. Let a remote native run fail before provider work and leave a recovery receipt or restore hold that does not verify (for example a receipt on an unsaved reference, a receipt from another epoch, or a saved reference without a receipt). 2. Press Retry on the task. 3. See the new run fail in setup with `remote_workspace_recovery_receipt_invalid`. **Paperclip version or commit** Staging deploy that includes `e3bc4ca798`. The gap remains on `master` at `a3b34a9d88`. ## What Changed - `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an unverifiable source. Each of the five refusal shapes now skips the source and logs `remote_workspace_recovery_receipt_invalid` with a reason: `receipt_without_reference`, `reference_unreadable`, `receipt_on_unsaved_reference`, `receipt_unverified`, and `receipt_snapshot_missing` (was `remote_workspace_recovery_snapshot_missing`). - `retainRemoteRecoveryAttachments` no longer throws for a retained recovered copy that it cannot verify. It does not publish that copy again. The files stay in the workspace. - The writer check (`remote_workspace_recovery_writer_changed`), the workspace access check, and snapshot read and digest failures still refuse. - `doc/execution-semantics.md` describes the fallback rule. ## Verification - Red on unfixed code: `remote-workspace-recovery-receipt-fallback.test.ts` (moved out of `workspace-recovery-preparation.test.ts` to keep that file under 1,000 lines) "starts a Retry from the newest verified save when a predecessor has %s" (6 shapes: `receipt_without_reference`, `unreadable_reference_with_hold`, `receipt_on_prepared_reference`, `superseded_snapshot_with_hold`, `receipt_from_other_epoch`, `receipt_for_other_run`). All 6 failed with `remote_workspace_recovery_receipt_invalid`. The source is a native run that failed with `runner_not_settled`, and the writer is a Retry (`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry query. - Green: the same 6 tests pass. They also assert that nothing is attached, and that the source run row, its workspace operations, and its snapshot row are unchanged. - Updated safety tests: the canonical-attachment fault matrix now expects no attachment for receipt faults, and still expects a refusal for a missing or corrupt archive and for a wrong company or task. The retained-history test expects an unverifiable retained copy to be left out of the published list, while verified copies stay published. - `npx vitest run src/services/remote-workspace-recovery-receipt-fallback.test.ts src/services/workspace-recovery-preparation.test.ts`: 91 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. - Not verified live on Daytona. I could not read the staging rows for this task: the browser had no free tab. The tests cover every refusal shape of this function, so the exact staging shape is one of them. ## Risks - Previous-release state: the rule reads the same rows as before (run reference, result hold, workspace operations, recovery actions, snapshots). Rows that earlier releases saved carry forward. No migration. A blocked task recovers on its next Retry or wake. - A Retry can now start without files from an unverifiable source. Those files are not lost: the source's snapshot, restore hold, receipts and recovery action stay unchanged, and workspace recovery still owns them. The skip is logged with a named reason. - A retained copy that fails verification is dropped from the published list. Its directory stays in the workspace and is no longer retired automatically. - A missing or corrupt snapshot archive still refuses. A permanently lost archive therefore still blocks attach. This PR does not change that. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 18:06:41 -06:00
DottaandClaude	9296c74e8d	fix(native): let a restart claim take back its lapsed lease at budget admission (#20650) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart, restart recovery claims an interrupted native run (`resume_dead_runner`) and grants a 20-minute lease. Budget admission then re-checks that the claim still owns the run before the resumed run may start provider work. > - A host sleep, or the restart-chaos fault TM2, can make that stored lease look expired while its owner is alive. #20625 lets the owner take back such a lease when the row still shows exactly its own claim. That rule did not cover budget admission. > - So a claim whose lease lapsed before dispatch was refused with `Native recovery no longer owns provider dispatch`. The run failed with `adapter_failed`, recovery held it with `native_continuation_requires_reconciliation`, and the user had to click Retry. > - This pull request applies the #20625 rule in budget admission: the claim takes its lease back through the existing re-acquire condition. > - The benefit is that a run resumed after a restart survives a time jump before dispatch with no Retry. Every fence and every budget check stays the same. ## Linked Issues or Issue Description Refs #20625, #20474 **What happened?** Restart-chaos wave L7 (seed 9107, local) on a branch with #20625, task codex B5 TM2. A TM3 restart claimed the run (`resume_dead_runner`, generation 2). TM2 then aged every database time by one hour before the resume reached dispatch. Budget admission (`lockRunBudgetRecoveryOwnerInTransaction`) required a live lease and threw `Native recovery no longer owns provider dispatch`. The run failed with `adapter_failed`, recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`, and the task needed Retry. **Expected behavior** The claim's own lapsed lease is taken back, and the resume is admitted when the budget allows it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` (timing-dependent). ## What Changed - `native-execution-lease.ts`: two new helpers. Both use the #20625 own-claim rule. Neither takes a lock out of order. - `readOwnNativeRecoveryClaimLease` (read-only) returns `live`, `lapsed_own_claim` or `lost`. It compares the lease against the database clock. A lapsed lease counts as `lapsed_own_claim` only when all of these hold: - the same lease owner; - this process's controller boot, pid and process start; - a non-null `recovery_state`; - `nativeExecutionLeaseHeldCondition({ remote: false, controllerGeneration })`, which applies the #20625 own-claim rule and every #20474 untouched check; - no non-local environment lease on the run. - `extendOwnNativeRecoveryClaimInTransaction` extends such a lease by the local execution TTL, with one conditional UPDATE on the same predicate. - `budget-reservations.ts`: - `lockRunBudgetRecoveryOwnerInTransaction` still refuses another owner, and refuses a lapsed lease unless it is the run's own untouched claim. It no longer locks the run row, so the documented coordinator → issue → run order holds. - The lease is extended only at the end of an admitted reservation (`reserveRunBudget`) or an admitted dispatch recheck (`recheckRunBudgetReservationInTransaction`). At those points the caller already holds the run row lock. A refused admission, for example one over budget, leaves the lease lapsed. - `heartbeat-native-restart-suspension.test.ts`: the test "still settles a recovery whose claim merely expired" encoded "expired means not the owner". #20474 and #20625 changed that rule for an untouched own claim. The test now has two cases: - an expired claim that another actor recorded on (`awaiting_evidence`) still settles with no provider session; - an untouched expired claim is dispatched. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new user of the rule. ## Verification - New `server/src/__tests__/native-recovery-dispatch-lease-reacquire.test.ts`: - `reserveRunBudget` takes back the claim's own lapsed lease and admits the resume. This fails before the change. - An over-budget resume is still refused, and its lease stays lapsed. - Fenced variants each throw `Native recovery no longer owns provider dispatch` and leave the row unchanged: a graceful-shutdown suspension, an appended failure settlement, `awaiting_evidence`, `blocked`, a claim from an earlier generation, another boot, a remote sandbox lease, and another lease owner. - At the dispatch gate (`recheckRunBudgetReservationInTransaction`), an admitted own claim gets its lease extended, and a fenced claim is refused with the row unchanged. - Related suites pass (308 tests): `native-execution-lease-reacquire.test.ts`, `closed-native-receipt-accounting.test.ts`, `heartbeat-native-predispatch-retry.test.ts`, `run-dispatch/adapters/postgres.test.ts`, `heartbeat-native-restart-suspension.test.ts` and `heartbeat-restart-claim-lapse.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: one L7 wave (seed 9107) ran on the first commit of this branch (`w-20261110T212307-s9107`). It logged 0 `Native recovery no longer owns provider dispatch` refusals, against 1 in the previous wave. Codex B5 TM2 got past dispatch, but it still needed Retry for another reason: after a `runner.drain` timeout, the scheduled same-run resume did not send the continuation of #20645. That case is tracked separately. The other five tasks passed. ## Risks - This widens when a restart claim can take back its own lapsed lease, and only at budget admission. It is local only. It needs the same owner, boot and generation, and the row must show exactly that claim. A claim that any other actor touched still fails as before. - Budget hard-stop behaviour is unchanged. The lease is extended only after budget admission passes. - No schema change and no migration. Rows saved by earlier releases carry the same history shape. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 17:43:46 -06:00
DottaandPaperclip	de7ceac0de	fix: preserve compatible historical Codex WEB evidence (#20665) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native session migration preserves local evidence in encrypted archives. > - Each WEB receipt binds the exact provider directory and exclusion policy. > - The Codex credential refresh change added two excluded files and changed that policy hash. > - A new reader then rejected old receipts even when their facts, files, skills, and authority still matched. > - This pull request accepts the exact prior Codex policy while checking archive paths against the current credential policy. > - Operators can keep valid saved evidence across the upgrade without copying it again. ## Linked Issues or Issue Description Refs: #19814, #20646. **What happened?** A reader upgrade rejected saved Codex WEB evidence because the profile hash changed when `auth-refresh.json` and `auth-refresh.json.tmp` became excluded credentials. The other receipt bindings still matched. This blocked the native harness migration gate. **Expected behavior** The exact known prior profile remains readable when the current archive path checks accept every saved entry. An archive that includes either newly excluded credential file must still fail. Unknown profiles and changed authority must fail. **Steps to reproduce** 1. Preserve WEB evidence with the frozen prior Codex profile and no credential refresh files. 2. Read its unchanged receipt with the current profile. 3. Observe that the old reader returns false solely because the profile hash changed. **Paperclip version or commit** Reproduced against `a3b34a9d88`; final branch is based on `e9343d8501`. ## What Changed - Accept the exact prior Codex profile hash only when the complete current profile has the reviewed successor hash and the provider is Codex. - Keep the current entry and excluded-path checks. Keep facts, controller, archive, skill, and ownership checks. - Add a frozen prior policy fixture with its literal hash. Test old receipt reads, both credential files, unknown policies, changed authority, and current writes. ## Verification - Real PostgreSQL baseline: the frozen old-policy receipt test failed because the reader returned false. The unchanged fixture passes with this fix. - All 18 selected PostgreSQL cases pass on final head after the master rebase: six new cases and twelve existing WEB archive and authority guards (14.93 seconds). - Full layered server typecheck passes on final head after the rebase. The module boundary check passes. - A single read-only native-only candidate check on the deployed dependency graph validated all 54 retained scopes. The 29 prior failures passed, and the other 25 stayed valid. The check made no storage calls or data writes. It did not replace final deployed enrollment acceptance. - Final head `14112d0e3abb6e318a86fe9799d95bddd2dbf76b` passes all [required CI checks](https://github.com/paperclipai/paperclip/actions/runs/42919867108): 55 successful checks, four expected skips, and one neutral review. The legacy security status also passes. Exact-head Greptile review is 5/5 with no open review threads. ## Risks The compatibility branch accepts only the two reviewed policy hashes. A future policy change can require another explicit compatibility decision. The reader still rejects saved credential refresh files. It does not rewrite receipts or change execution authority. No stored schema or operator command changes. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window capacity are not exposed in this session. Used reasoning, source editing, shell execution, and tool calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 17:22:08 -06:00
DottaandClaude	30403781b9	fix(native): keep crediting proven unstarted restarts past the generation limit (#20647) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run survives a control-plane restart through restart recovery, which resumes the interrupted provider turn > - Restart recovery charges a provider attempt only when the provider did work that could be lost, and it stops after three charged attempts > - From controller generation 12 on, every restart charged an attempt before that work rule was checked > - In a busy restart wave, one long task sees 12 to 15 restarts, most caused by other tasks. A restart that hit before the provider accepted a turn was charged, the budget ran out, and a person had to click Retry > - This pull request keeps the work rule past generation 12 for restarts that a durable record proves did not start the provider, and bounds a crash loop with a count of consecutive restarts without provider progress > - The benefit is that a long task survives many unrelated restarts without a manual Retry, while a run that never makes progress still stops ## Linked Issues or Issue Description Refs #20625 (same restart recovery path). No public issue exists for this case. **What happened?** `restartResumeKeepsAttempt` returned "charge" for every restart once the run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller generations. It did this before it read the work-based outcome. In a local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1) the run reached generation 13. The last restart interrupted a resume whose runner started but whose provider had not accepted a turn. That restart was charged at the last attempt, so the claim failed with `execution_recovery_budget_exhausted` and `native_restart_recovery_blocked`. The task needed a manual Retry. **Expected behavior** A restart that a durable record proves did not start provider work does not spend an attempt, also past generation 12. A crash loop that never makes provider progress still stops automatically. When it stops, the task shows a Retry that works. **Steps to reproduce** Run a native task past controller generation 12. Make the last restart hit after `native.process_start_requested` but before `turn.accepted`, with the provider attempt at 3. The new integration test `does not charge a restart past the generation limit whose resume never started the provider` reproduces the failure before the fix. **Paperclip version or commit** `master` at ed5e7493bb. ## What Changed - `restartResumeKeepsAttempt` now returns `{ keepsAttempt, noProgressRestarts }`. A new helper judges the interrupted attempt as `provider_not_started`, `provider_progressed` or `charged`, with the same proofs as before. - `noProgressRestarts` is 0 when the interrupted attempt is proven to have made provider progress. Otherwise it is the previous claim's count plus one. A clean-drain claim carries the previous count. - Before generation 12 the decision does not change. - From generation 12 on, a restart keeps the attempt only when the interrupted attempt is proven `provider_not_started` and the count is below the new `NATIVE_NO_PROGRESS_RESTART_LIMIT` (12). Provider work (progress, or an accepted turn without progress) and unprovable cases still charge. - Each restart claim writes `noProgressRestarts` into its `recovery_history` entry. No schema change: `recovery_history` is JSONB. - Claims saved by earlier releases have no count. They are read as zero. - Integration tests: the three generation-cap remote tests now check the new bound. New tests cover the unstarted restart past the limit, the consecutive-limit stop with a working Retry, the progress reset, an accepted turn without progress, an unrecorded previous claim, and an old-format claim. - The durable-ownership plan (section F) records the rule. **Why 12.** It equals the existing generation limit (`PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT`). The count includes restarts before generation 12. Thus a run that never makes provider progress is bounded exactly as before: the existing test `bounds a crash loop that never starts the provider` passes unchanged. Only proven progress resets the count. In the failing wave, the task had made progress shortly before its last restart, so its count was 1. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - Red before the fix: with only the new constant added to the old code, the 9 new and changed tests fail. With the fix, they pass. The existing crash-loop, progress and unstarted-resume tests pass without change. - The consecutive-limit test checks that the stopped run is `terminal_failure` / `native_restart_recovery_blocked` with no lease owner and nothing scheduled, that `taskInterruptionPresentation` shows `failed` with `canRetry: true`, and that `requestTaskRetry` accepts the Retry. - `cd server && npx tsc --noEmit -p .` - Full local run of the three suites above: 322 passed, 14 skipped. - Live: local restart chaos wave L6 (seed 9106, concurrency 3, `--env local --adapters codex,claude --workloads A1,B5,C1 --faults RR1,RR2,RR3 --tasks 6`) at 7f7803ab0c: 6 of 6 tasks passed, and no task needed Retry. Task t03 (Claude B5 RR1) reached controller generation 13 again. The claim at generation 13 resumed the run (`resume_dead_runner`), and the server log has no `execution_recovery_budget_exhausted`. The same task on master was blocked at that claim and needed Retry. ## Risks - A run that makes provider progress between restarts past generation 12 still charges each of those restarts. This is the same as before, by design. - An upgrade or a mid-run deploy reads old claims as count zero. A run that was already in a no-progress loop can get up to 12 more proven-unstarted restarts before the bound applies. It cannot be charged where the old rule did not charge. - A rollback ignores the new field and restores the old rule. - Past generation 12 the claim now reads the event log, as it already does before generation 12. This is not on the turn-start path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 15:27:45 -06:00
DottaandClaude	805531cc71	fix(native): never leave a cleanup-gated task without Retry or an owner (#20639) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs close their provider session at the end of each attempt. When close cannot prove that the runner stopped, the runner keeps a cleanup gate (`native_session_cleanup_quarantined`) in server memory > - #20572 hid Retry for every gated run and said "Paperclip finishes the cleanup automatically". Only a remote environment release settles the gate. A local gate that needs an operator never settles, and a server restart loses it > - A restart-chaos wave on master found two tasks that rested blocked with no Retry, no scheduled retry and no recovery owner. A person could not continue them > - This pull request makes the run's own recovery retire the gate when the host proves the old processes stopped. It also offers Retry whenever nothing settles the gate, and it makes that Retry work > - The benefit is that a cleanup-gated task either continues by itself or shows a Retry that starts. It never rests without both ## Linked Issues or Issue Description Refs #20572, #20578 **What happened?** Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2: the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05 completed their provider turn. The runner checkpoint owner lapsed while the server was paused (`native_runner_checkpoint_owner_lost`). The session close then failed with `native_session_close_unrecoverable: runner did not durably suspend before checkpoint`. The provider had drained and no tool call was pending. The runner kept an operator-required cleanup gate. Thirty seconds later the run's own same-run recovery verified that the runner process and group had stopped (`native.local_process_stopped`). The gate still refused the recovery, and the run ended `failed` with `native_session_cleanup_quarantined`. The task showed `canRetry: false` and a cleanup wait until the deadline. Nothing owned that wait. **Expected behavior** When the old runner and provider processes are proven stopped, the run's own recovery continues. When nothing settles the gate, the task offers a Retry that starts. **Steps to reproduce** 1. Run restart-chaos L7: `--env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`. 2. Observe a claude TM2 task rest `blocked` with `canRetry=false`, `retryWait=true` and no scheduled retry. **Paperclip version or commit** master at ed3ffc3bc7. ## What Changed - `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It returns proof only for a local run with a pid-bound `native.local_process_stopped` record (no later launch) and with every recorded provider process gone. An ACPX session open names no provider PID; its sidecar counts only when it shares the runner's process group, which the stop record proves empty. - `native-session-executor.ts`: add `retireStoppedOwnSessionCleanupGate`. With that proof, it retires the run's own operator-required gate through the existing `completeTerminatedLocalNativeSessionCleanup` boundary and records `native.session_cleanup_gate_retired`. - `heartbeat.ts`: the native recovery preamble calls it right after it records the local process stop, before admission. The recovery decision itself does not change. - `explicit-native-continuation.ts`: Retry or a new message accepts the run's own stop proof when the retained-state inventory proof does not apply. It still starts a fresh session, and it never resumes the old turn. - `task-interruption-presentation.ts`: hide Retry for a gated run only while a live owner settles the gate: an unreleased remote environment, or a scheduled close in this process. Otherwise offer Retry. The same rule applies to a Retry held on `cleanup_quarantined`. - `doc/execution-semantics.md`: describe the new rule. ## Verification - Red before the fix, green after (5 tests failed on master sources): - `server/src/services/task-interruption-presentation.test.ts`: "offers Retry when nothing settles the gate by itself", "holds a Retry on a cleanup gate only while an owner settles it", and an invariant test: a failed presentation without Retry must name a wait, and a cleanup wait needs a settling owner. - `server/src/__tests__/task-interruption-query.test.ts` (real Postgres): an unreleased remote lease hides Retry; the released lease offers it. - `server/src/services/quarantined-task-retry.test.ts`: "starts a task Retry for a run's own quarantine proven stopped by its own process record" (stopped, and provider alive stays held). - New: "retires a run's own failed-close gate once its runner and providers are proven stopped" for codex PIDs, an ACPX sidecar in the runner's group, a sidecar with its own group, a live provider, and no stop record. - `npx vitest run` of `quarantined-task-retry`, `task-interruption-presentation`, `task-interruption-query` and `explicit-native-continuation`: 575 passed. - `npx tsc --noEmit -p server` passes. - Live: LIVE_PROOF_PLACEHOLDER ## Risks - The gate retirement removes process ownership only. It runs only for a local run whose exact runner and group and every recorded provider process are proven gone. A provider without a recorded process identity keeps the gate. - Retry on a gated run still runs every existing stop check. If the stop cannot be proven, the Retry waits and the task keeps a Retry, instead of a wait that nobody settles. - No persisted shape changes. Runs saved by earlier releases use the same fields. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 15:20:42 -06:00
DottaandClaude	37c50f70c2	fix(native): continue a turn lost on restore in a proven same-run resume (#20645) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run keeps its provider conversation when its local runner dies. When the restored runner finds that the active turn was lost, it reports `provider_turn_lost_on_restore` and does not invent a result. > - After a control-plane restart, the admitted restart attempt may send one continuation in that same conversation (#15297). The agent then reconciles unfinished work. The task is not resent. > - A runner can also be lost without a server restart, for example when `runner.drain` times out. The run then resumes itself after the retry delay from its exact checkpoint. That path never allowed the continuation, because the executor passed `resumeInterruptedTurn` only for a `resume_dead_runner` restart claim. > - So the lost turn failed with `native_provider_terminal_failed`, automatic replacement was refused (`uncertain_provider_action`), and the user had to click Retry. > - This pull request allows the same single continuation for that resume, but only when durable records prove the same three facts that a restart claim proves. > - The benefit is that a runner lost mid-turn continues in the same conversation with no Retry. No case that is refused today becomes allowed without that proof. ## Linked Issues or Issue Description Refs #15297, #20625 **What happened?** Restart-chaos wave L7 (seed 9107, local), task codex B5 TM2, run on master after #20625. A TM3 restart resumed the run (attempt 1). During the TM2 freeze the TS runtime lost its runner: `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`. The executor saved `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`. After the delay, recovery verified the runner process stopped (`native.local_process_stopped`). The same run then resumed as attempt 2 (`native.process_start_requested`, `providerAttempt: 2`). The restored runner reported `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). No continuation was sent. The run failed with `native_provider_terminal_failed`, recovery stopped with `uncertain_provider_action` / `preserve_without_replay_v1`, and the task needed Retry. **Expected behavior** The run sends the one same-conversation continuation, exactly as it does after a restart claim, and finishes without Retry. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` (timing-dependent: the drain timeout must land during an active turn). ## What Changed - New `server/src/services/native-runtime/scheduled-resume-continuation.ts`. It decides whether a same-run scheduled resume may continue a lost turn. Each of the three proofs comes from a durable record: 1. **The previous runner is stopped.** The latest process record of the run in `heartbeat_run_events` (`native.process_start_requested`, `native.process_identity_recorded`, `native.local_process_stopped`) must be `native.local_process_stopped`. Its `processPid` must equal its `processGroupId`, and neither the pid nor the group may be alive now. This is the existing `readNativeLocalProcessStop` check, read in the claim transaction before this attempt records its own `native.process_start_requested`. 2. **The attempt was admitted under the bounded budget.** The executor's claim moves `native_run_finalizations` from phase `retryable_failure` to `observed`. It charges a provider attempt, and the claim refuses any attempt above 3 (`native_session_retry_exhausted`). The decision also requires `1 <= attempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`. 3. **It is the same provider conversation.** The saved `failure_detail` must show `recoveryMode: "exact_checkpoint_resume"`, `recoverable: true`, `providerSessionEstablished: true` and `checkpointExists: true`, with no `successorRunId` or `replacementDenied`. At run time the existing runtime guards still apply: the backend must recover the checkpointed session (`recovered`, not a continuity-break replacement), and the lost turn must equal the checkpoint's `activeTurnId` and the snapshot's interrupted turn. The provider kinds stay codex, acpx and opencode, with no compaction, goal control, held result or pending runtime request. - `native-session-executor.ts`: computes that decision in the coordinator claim. It passes `resumeInterruptedTurn` when either the restart claim is `resume_dead_runner` (unchanged) or the decision allows it. - `doc/execution-semantics.md`: describes the rule and the refused cases. **Cases that stay refused** (each has a test in `scheduled-resume-continuation.test.ts`): - a restart-recovery claim (it keeps its own existing rule); - a remote execution target; - a coordinator that is not `retryable_failure`, including a fresh run; - `recoveryMode` `bootstrap_retry` or `ambiguous_state`, or no saved failure detail; - a failure that is not recoverable; - no established provider session; - no checkpoint; - a recorded replacement (`successorRunId` or `replacementDenied`); - an attempt outside 1..3; - no stop receipt, a stop receipt superseded by a newer launch, a stopped process that is still alive, or a run that never had a process. The runtime-side refusals are unchanged and already covered by `native-session-runtime.test.ts`: real provider errors, historical error-null failures, and no restart permission. ## Verification - `cd server && npx vitest run src/services/native-runtime/scheduled-resume-continuation.test.ts`: 22 tests, covering the decision table and the stop receipt in embedded Postgres. - `cd server && npx tsc --noEmit` passes. - Live: not proven. An L7 wave on this branch (seed 9107) did not reproduce the drain-timeout path, so this pull request has unit and database tests only. In that wave the same cell (codex B5 TM2) needed Retry for a different reason: TM2 aged its restart-recovery claim lease before dispatch, and budget admission refused with `Native recovery no longer owns provider dispatch`. This pull request does not change that path. Four other tasks passed. Claude C1 TM2 rested blocked without a Retry button, which is a separate cleanup-quarantine issue. ## Risks - This allows one more path to send the same-conversation continuation. It is limited to local runs. It needs the same proofs as a restart claim, and it uses the attempt the resume already charged. It does not change the generation cap or `restartResumeKeepsAttempt`. - A rolling deploy reads `failure_detail` and stop events written by earlier releases. Those fields and events already have this shape, so an older saved failure is judged by the same rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 15:16:51 -06:00
DottaandPaperclip	2150820bd8	perf(native): parallelize durability eligibility checks (#20646) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server must prove that retained run state is durable before volume-free enrollment. > - The proof reads selected native sessions and validates their existing recovery receipts. > - Serial database checks made a valid inventory take 59.665 seconds. > - The signed preparation request has a 55-second limit. > - This pull request overlaps independent reads within fixed limits. > - The benefit is a faster eligibility check with the same preservation rules. ## Linked Issues or Issue Description Refs #20599 and #20601. A search found no duplicate of this remaining serial-read bottleneck. **What happened?** An actual full startup eligibility check returned eligible with no blockers in 59.665 seconds. Controller checks took 44.134 seconds and harness checks took 8.592 seconds. Signed preparation timed out at 55.002 seconds. **Expected behavior** A durable instance can complete signed preparation within its existing deadline. **Steps to reproduce** 1. Retain many native task sessions with database-backed controller checkpoints and valid filesystem-independence receipts. 2. Run the full startup eligibility check. 3. Observe serial checkpoint and receipt reads consume the signed request budget. ## What Changed - Check the existing terminal-controller metadata and checkpoint-existence predicate within the existing eight-reader source batch. - Validate harness independence receipts with at most four concurrent readers per existing page. - Stop admitting more harness reads after an invalid result or error. Join all started reads before returning or throwing. - Keep large controller, full-checkpoint and legacy archive reads serial. - Add real PostgreSQL regressions for overlap, concurrency limits and failure settlement. ## Verification - Genuine controller regression RED: all 16 checkpoint reads ran, but peak overlap was one. GREEN: 10 focused PostgreSQL controls passed, including bounded overlap, joined failures, active and warm handoffs, and failed-source preservation. The other 45 cases were filtered, not claimed as passes. - Genuine harness regression RED: overlap was one instead of four. GREEN: all three new PostgreSQL controls passed. Six existing authority, replacement and refusal controls also passed. - Final head `c059e031`: the same 10 controller and 9 harness PostgreSQL controls passed after rebasing onto current master and refreshing Runner artifacts. `node node_modules/typescript/bin/tsc --noEmit -p server/tsconfig.json` also passed. The first `pnpm exec` attempt could not start the compiler because package-manager bootstrap fetches failed; it is not counted as a typecheck pass. - `git diff --check` passed. - Actual private-process acceptance with deployed `8e4c5e81` dependencies and only the two changed owners: the complete check passed in 37.596 seconds with no blockers. Nine runs were running before and after; 1,750 sessions were selected. This is 17.4 seconds inside the unchanged request limit. The earlier complete baseline was 59.665 seconds on 1,745 selected sessions, so these are observed timings on a changing dataset, not a controlled exact speedup. No serving process was patched. - The first private candidate check stopped on a native-controller blocker in 18.216 seconds. A targeted follow-up found no blocker without any preservation or business write. Its cause was not identified. It is retained as a nonpass and is not a speedup result. - Current-head Greptile review is 5/5 with no actionable findings. Required CI is running: https://github.com/paperclipai/paperclip/actions/runs/42898013184. ## Risks - The scans now use more simultaneous database reads. Limits remain eight for controller source readers and four for harness proofs. - Invalid or failed proofs still refuse eligibility. All started readers settle before failure leaves the scan. - Previous-release state carries forward unchanged. This changes no stored shape, checkpoint identity, lease rule, schema, deadline or provider protocol. - Large legacy payload reads remain serial, so their existing memory bound is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. This is a measured performance defect in the existing enrollment path. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution and parallel agents. The runtime does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 14:57:11 -06:00
DottaandClaude Opus 5.5	4398a723c0	fix(native): settle tool callbacks and holds that a server restart orphaned (#20628) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs send provider tool calls to the server as durable semantic inputs. The server runs the tool and saves the result in the controller journal > - Close must prove that every admitted tool call has a saved result. Otherwise it reports `native_session_settlement_incomplete` and the runner holds a cleanup gate (`native_session_cleanup_quarantined`) > - A server restart in the middle of a tool call leaves its input without a result. The new controller asks the tool authority for a receipt. Tools without a receipt (assigned app tools, `connections_search`) return nothing, so the input stays pending forever > - Runs that already failed this way kept no-replay holds for a turn that had completed. The task stayed blocked with no Retry and no continuation > - Restarts are normal. Work that a restart interrupts must settle by itself, not wait for a person > - This pull request settles such a callback as interrupted, with an unknown outcome, when its owner is provably gone, and settles the holds that earlier releases left once their controller is gone. It never runs the tool or the turn again > - The benefit is that close drains after a restart, and tasks that earlier releases blocked this way continue on their own ## Linked Issues or Issue Description Refs #20572 (that PR fixed the presentation of this quarantine. Its Risks section names this runner drain rule as a separate follow-up.) Refs #20641 (it settles a resolved no-replay hold when a sibling action saved the run's files. This PR covers a different case: the hold's own recovery saved the files, and its run's turn completed. Both use the same recovery receipt, `completedWorkspaceRecoveryCondition`.) **What happened?** A remote native run started an assigned app tool call. The server restarted 3 seconds later. The new server adopted the runner, and the provider turn completed about 9 minutes later. The tool call never got a saved result. Close reported `native_session_settlement_incomplete` with `runnerSuspended: true`, `providerDrained: false` and one pending callback. The session quarantined, and the same-run recovery was refused at admission. A second run showed the same shape with a `connections_search` callback after a later restart. The first run then had two resolved-but-blocked holds with `preserve_without_replay_v1` and an unknown action outcome: the cleanup-quarantine hold, and a continuation hold whose workspace recovery saved the files. The task view said "Saved work recovered. Send a message to continue." with no Retry and no continuation. The task waits on a child task and stays blocked. **Expected behavior** After a restart, a tool call without a saved result settles automatically. A durable receipt is replayed. Without a receipt, the agent is told that the outcome is unknown. Close proves the drain, and no cleanup gate is left. Holds that earlier releases left for a completed turn settle without a person, and the task continues when its own wait ends. **Steps to reproduce** 1. Start a native run and call an assigned app tool. 2. Restart the server while the tool call runs. 3. Let the adopted turn complete, then close the session. Close reports `native_session_settlement_incomplete` and quarantines. The task stays blocked by the no-replay holds. **Paperclip version or commit** master at fd6e71a7ff. ## What Changed - Root cause: `DurablePrpControlPlane.#recoverPendingSemanticInputsOnce` (`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`) keeps an input pending when `recoverSemanticToolInput` returns null. `PaperclipRunnerToolAuthority.recover` returns null for every tool without a durable receipt. Nothing else could settle the input, so `semanticToolResultsSettled()` stayed false and `#closeOnce` in `runnerd-codex-transport.ts` threw `NativeSessionCloseUnrecoverableError`. - The control plane now passes `admission` to `recoverSemanticToolInput`: `this_controller` when this instance dispatched the callback live, `earlier_controller` when it restored the input from an earlier controller's journal. The field is not written to the result command or the journal. A live dispatch leaves this record when its result is journaled. - New `packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts`: - `dispatchToolCallback` records the callbacks that this process runs, and the ones it saw fail. - `recoverToolCallback` replays a durable receipt first. Without one, it settles an `earlier_controller` callback that this process does not run as `tool_call_interrupted_by_restart` (error result, `outcome: "unknown"`, `retryable: false`). A `semantic_tool_outcome_unknown` rejection from the authority settles the same way, only under the same owner-gone proof. - `native-session-executor.ts`: `SessionAuthorityEpoch.execute` runs through `dispatchToolCallback`. `SessionAuthorityEpoch.recover` runs through `recoverToolCallback`. The three recovery call sites pass `admission`. - New ownership kind `native_completed_turn_cleanup_hold` (`server/src/services/native-runtime/completed-turn-cleanup-hold-settlement.ts`, registered in `recovery/ownership-registry.ts`). It lists a native run that failed with `native_session_cleanup_quarantined` after its saved checkpoint recorded a completed turn, and that has a system-recorded no-replay hold. After `proveOwnerGone` proves the closing controller gone (its in-process cleanup gate went with it), it settles each such hold as `restored`, removes its replay block, and logs `issue.recovery_action_resolved`. A hold that still protects files not verified as saved stays for workspace recovery. Operator decisions are never reopened. It starts no turn. `heartbeat.ts` registers the ordinary resume path (`resumeRemoteStopComments`) for saved user input. - Docs: `doc/execution-semantics.md` and `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Red before the fix, green after: - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts` "host-tool callback orphaned by a server restart". It uses the real control plane and an authenticated runner. A callback starts 3 seconds before the restart, the old controller retires with no result, the new controller adopts the runner, the turn completes, and then recovery runs. `paperclip_call_assigned_tool` and `connections_search` settle as `tool_call_interrupted_by_restart`, the runner accepts the late result, `semanticToolResultsSettled()` becomes true (the condition close requires), and the tool is not dispatched again. Before the fix, 4 of 5 cases failed: recovery received no admission and the callback stayed pending. - `server/src/__tests__/completed-turn-cleanup-hold-settle.test.ts` uses the staging shape of both holds (real Postgres, rows as earlier releases saved them). Before the fix, the ownership reconciler did not list them (4 of 4 failed), and the execution blocker was the files hold with `canRetry: false` and `canContinue: false`. After the fix, both holds settle as `restored`, the execution blocker is null, no run is created, the task keeps `blocked`, and the next pass lists nothing. - Safety cases: - A callback that this process still runs stays pending. After it finishes, the next recovery pass settles it. - A callback that this controller dispatched and saw rejected stays pending, as before. - A durable receipt is replayed and not replaced. - Holds stay while the closing controller is alive (its gate may still exist). - A hold whose files are not saved stays; the run's other hold settles. - Holds of a run whose turn did not complete are never listed. - Focused suites pass: `durable-prp-control-plane.test.ts` and `runnerd-codex-transport.test.ts` (506 tests); runner adoption, release-resume, semantic readiness, ACPX reconciliation and external driver tests; `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` (1496 tests); `ownership-reconciliation-report.test.ts` and `interrupted-preparation-hold-settle.test.ts`. - `tsc --noEmit -p packages/paperclip-runner/tsconfig.test.json` passes. `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify` returns exit 2 (a global doc input changed), so required CI supplies the broad coverage. ## Risks - Previous-release story: no persisted shape changes. `admission` is computed at recovery and never saved. A journal that an earlier release saved with a pending callback restores as `earlier_controller`, so that session settles at its next adoption. Holds that earlier releases saved settle on the reconciler's first pass after deploy. No migration. - No double execution: recovery never calls the tool again. It replays a durable receipt or reports an unknown outcome. The agent is told to check the current state before it repeats the call. Hold settlement starts no turn and replays nothing. - The owner-gone proof for a callback: the new controller holds the run's checkpoint fence, and the earlier controller retired its journal writer. If the earlier process is still alive during an overlapping deploy, it can still finish the tool, but it can never record a result in this session. The unknown outcome is accurate in that case. - Hold settlement requires `proveOwnerGone` on the closing controller's stamp, because the cleanup gate lives in that process. On a server that has not restarted since the quarantine, the holds stay. - The in-process failure memory is bounded to 4,096 entries. Eviction only lets a much older failed callback settle as interrupted later. It never replays it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 14:39:45 -06:00
DottaandClaude Opus 5.5	e3bc4ca798	fix(native): settle warm follow-ups that a restart left behind (#20631) ... <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native warm runs on a remote sandbox continue the saved session of the previous run. Admission first proves what happened to the previous Runner. > - A deploy restarted the staging server while warm Runners were idle. Three follow-up runs then failed before provider work, each with a different refusal, and their tasks stayed blocked. > - The refusals asked for proof that a restart had made impossible. A restart is a product failure, so recovery must settle it automatically and keep the strict checks for real uncertainty. > - This pull request accepts the proof that the restart leaves behind, in three places: the warm owner on a replaced sandbox, the suspended Runner with an undelivered suspend receipt, and the restore hold of a run that saved nothing new. > - The benefit is that these follow-up runs start, and state saved by older releases carries forward without a migration. ## Linked Issues or Issue Description Refs #20567, #20346 **What happened?** On staging, a deploy replaced the webserver process at about 18:04 UTC on 2026-11-10 (lease controller boot `348ec0bb` on one deployment, then boot `3e84c195` on a new deployment that started at 18:04:16). Three runs of one Codex agent on Daytona failed between 18:08 and 18:15: 1. A warm follow-up got a newly created sandbox. The previous warm Runner stayed `ready` in its own retained sandbox. Live adoption denied it with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` (`remote-warm-session-recovery.ts:127`), because only a resumed sandbox may be adopted. The per-turn retirement path refused warm owners (`lifecycle_not_per_turn`). 2. A warm follow-up resumed the same sandbox and directory. The previous Runner had been suspended by the shutdown, but the controller stopped before it acknowledged the suspend result. The Runner kept its terminal fence (`pendingTerminalDelivery` for `runner.suspend`). The read refused it with `native_controller_recovery_missing_evidence:runner_not_settled` (`native-session-executor.ts:7032`). 3. A continuation into a fresh workspace failed with `remote_workspace_recovery_receipt_invalid` (`remote-workspace-recovery-attachments.ts:60`). Its predecessor had restored the committed save, failed before provider work, and kept a restore hold with no recovery receipt. Its saved reference named the snapshot it restored. **Expected behavior** After a restart, the next run continues. It retires a finished owner that nobody can adopt, reads a finished suspension, and skips a source that left no unsaved files. Real uncertainty still refuses with a named reason. **Steps to reproduce** 1. Run a warm native Codex turn on a reusable Daytona sandbox and let it commit. 2. Restart the server (deploy) while the Runner is idle. 3. Start the next turn. Get a new sandbox (case 1), or resume the same sandbox after the shutdown suspended the Runner (case 2). 4. For case 3, continue a task whose previous run restored the last save and failed before provider work, into a fresh workspace. **Paperclip version or commit** Staging before `2e3cbe57` (the runs predate that deploy). The gaps remain on `master` at `9b21dcbd`. ## What Changed - `inspectRemoteWarmSessionRecovery` returns `null` when this run's sandbox is `created` or `replacement`. Such a sandbox holds no Runner to adopt. The caller then uses ordinary admission. - `inspectFinishedPerTurnRunnerdOwner` also retires a **warm** owner on a remote sandbox runner when this run's sandbox is `created` or `replacement`. It uses all the existing proofs: terminal run, committed result, released coordinator lease, stopped or fenced controller, settled journal, and quarantine copies. Local warm Runners and resumed sandboxes keep the live path. - `restoreRemoteSuspendedRunnerCheckpoint` accepts a suspended Runner whose only open fence is its own undelivered `runner.suspend` receipt. The fence must name `runner.suspend` with lifecycle `suspended`, it must be the last controller command, and the restored journal must not name that command differently. A provider cleanup, unsent events, a turn's terminal fence or a warm transition still refuse with `runner_not_settled`. A later authorized Runner process reconciles the receipt, as the Runner already does. - `attachPreservedRemoteWorkspaceSnapshots` skips a source with a receipt-less restore hold when its saved reference names the exact committed snapshot that its own succeeded `remote_workspace_restore` loaded at the same epoch. Any other saved snapshot still needs the canonical receipt. - `doc/execution-semantics.md` describes the three rules. ## Verification - Red (unfixed code), each in the exact staging shape: - `native-recovered-owner-followup.test.ts` "retires a finished warm owner left behind by a restart when this run's sandbox is created/replacement" failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. - `native-controller-recovery.integration.test.ts` "defers a remote warm continuation…" failed: live adoption rejected a created/replacement sandbox with the same error. - `native-controller-recovery.integration.test.ts` "reads a resumed sandbox's suspended Runner with an undelivered suspend receipt" failed with `native_controller_recovery_missing_evidence:runner_not_settled`. It reads through a sandbox runner that refuses paths outside the workspace base. - `workspace-recovery-preparation.test.ts` "decides a receipt-less restore hold of a source that restored_unchanged" failed with `remote_workspace_recovery_receipt_invalid`. - Green: the same tests pass. Focused files pass: `native-recovered-owner-followup.test.ts` and `native-controller-recovery.integration.test.ts` (104 tests), `workspace-recovery-preparation.test.ts` (85 tests). - Safety cases (strict checks stay strict): - A warm owner on a resumed sandbox is not retired. - A warm owner with an unsettled journal (pending command, unanswered tool input) stays held and names `retirement_refused=journal_…`. - A warm owner whose finalization lease is still held stays held. - A turn terminal fence, a provider cleanup, unsent events, a fence behind a later command, or a journal that names the command differently still fail with `runner_not_settled` and cache nothing. - A restore-hold source that saved another snapshot, whose restore did not succeed, or whose restore was at another epoch still fails with `remote_workspace_recovery_receipt_invalid`. - Neighbouring suites pass: `native-session-executor.test.ts`, `plugin-runner-release-resume.integration.test.ts`, `explicit-native-continuation.test.ts` and `native-harness-filesystem-independence-continuation.integration.test.ts` (1786 tests). The last one asserted the old warm refusal on a created sandbox. It now shows that the warm follow-up resumes the same provider session for the new run. - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` returns exit 2, because a global doc input changed (`doc/execution-semantics.md`). Required CI supplies the broad coverage. ## Risks - Previous-release state: all three rules read state that earlier releases already save (lease outcome, coordinator rows, Runner state in the sandbox, journal commands, run results and workspace operations). Rows saved by older releases carry forward. No migration. - Case 1 starts a new Runner for a warm session whose old Runner may still exist, idle, in its retained sandbox. Its controller is gone and its result is committed, so it can accept no new commands. The provider session follows the existing resume-or-replace policy. - Case 2 trusts a suspend fence only in the exact shape that the Runner writes before it sends the result. Other fences still refuse. - After case 1, a warm follow-up resumes from its own warm checkpoint. That checkpoint carries no replacement policy, so a failed provider resume is still reported as `native_session_recovery_failed` and is not replaced. This is the existing warm behavior. - Case 3 skips a source only when it provably saved nothing beyond a committed snapshot. That snapshot stays in its own workspace. - The three tasks that already failed stay blocked until their next wake or a user message. This PR does not add an automatic continuation for a completed no-replay hold. A follow-up can add that. - `native-runner-restart-recovery.integration.test.ts` (runs only with a runner container) no longer probes a replacement sandbox, because that probe now retires the owner by design. - Not verified live on Daytona. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only browser access to staging). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 14:38:30 -06:00
DottaandClaude	24661683db	fix(recovery): bootstrap a remote follow-up that a restart cut before its Runner launched (#20642) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart, restart recovery classifies each claimed native run: reattach a live Runner, resume from a checkpoint, or bootstrap again when durable records prove no provider work > - A follow-up run on a continued session gets a rebound admission checkpoint. It names the new run, but it keeps the previous turn's provider session ID > - Recovery read that copied provider session as this run's provider work. A follow-up killed after `native.process_start_requested` but before its remote Runner launched was classified `reattach_remote_runner` > - The reattach then had no Runner to find, failed with `native_continuation_requires_reconciliation`, and the task needed Retry > - This pull request stops the copied session from counting as this run's work, and keeps the existing durable proofs as the authority > - The benefit is that these restarts bootstrap again by themselves and do not repeat provider work ## Linked Issues or Issue Description Refs #20561, #20555 **What happened?** Two Daytona restart-chaos tasks needed a Retry click: - Wave `w-20261110T173243-s9202` t02 (claude C1, K2): the follow-up run was killed at `stage.asset.home`, after `native.process_start_requested`. Recovery chose `reattach_remote_runner`. Verification failed with `native.remote_runner_recovery_failed {stage: runner_state, reason: state_read_failed}` (`runner_remote_state_unavailable: exit=1`): `runner-state.json` did not exist because the Runner never launched. The storage fault window (ST7) had ended 30 s before, so it was not the cause. - Wave `w-20261110T174930-s9205` t04 (codex C1, RR1): the scheduled retry of a `process_lost` run was killed at `session.checkpoint.restore`, after `native.process_start_requested`. Recovery chose `reattach_remote_runner`. Verification failed with `native_remote_recovery_lease_mismatch: stage=runner_verification field=identity_run_id`: the local Runner identity belonged to the previous run, because this run's Runner never started. **Expected behavior** A run whose durable records prove that no Runner received a provider turn bootstraps again without a person. **Steps to reproduce** 1. Start a remote follow-up run on a continued session (for example, a comment after a completed turn). 2. Kill the server after `native.process_start_requested` and before the remote Runner launches. 3. Observe `reattach_remote_runner`, a failed verification and a Retry banner. **Paperclip version or commit** master close to `6d6b2d0e` (waves); still reproduced by the new tests on master `90dd864bff`. ## What Changed - `native-restart-recovery.ts`, no controller checkpoint for the run (`remote_runner_never_received_commands`): a follow-up with `sessionIdBefore` qualifies when it carries only its continued-session checkpoint, and a predecessor's settled Runner identity (`carriesInheritedRunnerIdentity`) no longer disqualifies it. A recorded process identity, a Runner-sent event, a provider event or a pending request still keeps the run held. - `native-restart-recovery.ts`, controller checkpoint exists: the existing Postgres proof (`nativeRunnerCheckpointRemoteBootstrapProof`) now also runs when the run carries only a rebound checkpoint whose provider session a settled earlier run of the same session saved (`carriesSettledPredecessorSession`). The copied provider session then does not count as this run's provider evidence. The controller checkpoint stays the authority: any turn command or turn event keeps the run held. - `doc/execution-semantics.md`: describe the rule. ## Verification - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (real Postgres): - "restarts a follow-up whose controller never saved a checkpoint (s9205 t04)" and the variant "with its predecessor's remote runner identity": red on master (`reattach_remote_runner`), green now. - "restarts a follow-up whose saved controller checkpoint proves no provider turn (s9202 t02)": red on master, green now. - Held cases stay held: own recorded Runner identity, a carried active turn, a provider session that no settled earlier run saved, a `turn.start` command, a projected `turn.started` event. - Whole file: 171 passed. - `npx tsc --noEmit -p server` passes. - The executor's existing `bootstrap_incomplete` path handles both shapes: a scoped root that belongs to the previous run goes through the existing prior-owner verification, and a root of this run restored from the controller checkpoint goes through `runnerdStateProvesIncompleteBootstrap` / `runnerdStateProvesNoRemoteProviderTurn`. ## Risks - A run with any recorded Runner process, Runner-sent event, provider event, turn command or pending request keeps today's reattach or hold. - A bootstrap claim charges one provider attempt, as today. The bounded limit (3) still applies. - No persisted shape changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 14:37:56 -06:00
DottaandClaude	ed5e7493bb	fix(native): take back a lapsed lease held under the run's own restart claim (#20625) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an execution lease on its `native_run_finalizations` row. The owner renews it every 5 minutes. After a host sleep (or the restart-chaos time fault TM2), the stored lease looks lapsed although the owner is alive. > - #20474 lets the same owner take back a lapsed lease when no other actor recorded anything on the row. One of the checks is "`recovery_state` is null". > - Restart recovery writes `recovery_state` (`resuming_session`, `awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update that grants the lease. The executor keeps that value until a result is accepted. > - So a run that a restart resumed can never take back its own lapsed lease. Renewal interrupts the turn, and failure settlement is refused with `native_session_lease_lost`. The run fails with `adapter_failed`, recovery holds it with `preserve_without_replay_v1`, and the user must click Retry. > - This pull request counts the claim's own record as untouched when the row still shows exactly that claim and nothing recorded since. > - The benefit is that a resumed run survives a later host sleep with no Retry. Every real takeover still fences the late owner out. ## Linked Issues or Issue Description Refs #20474 **What happened?** The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2, TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were Codex runs. In each run, a TM3 kill made restart recovery resume the run (`resume_dead_runner`, generation 2). The runner restored the conversation, reported `provider_turn_lost_on_restore`, and the same-conversation restart continuation started as designed. Then the TM2 fault aged every database time by one hour. The execution lease lapsed while its owner was alive. The run then failed with `native_session_lease_lost` (`adapter_failed`), and recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`. One run had already proposed its result and completed its turn. The Retry click worked every time. **Expected behavior** A run whose lease lapsed only because time jumped, and that no other actor claimed, continues with no Retry. This must also be true after a restart resumed that run. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` ## What Changed - `server/src/services/native-runtime/native-execution-lease.ts`: the "untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts the run's own local restart-recovery claim. All of these must be true: - the last `recovery_history` entry is that claim: same controller boot and generation as the row; - its disposition matches the current `recovery_state` (`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` → `awaiting_runner_reattach`, `bootstrap_incomplete` → `bootstrap_incomplete`); - `recovery_request_id` equals the request that claim recorded. Any later actor appends a history entry (shutdown suspension, failure settlement), moves the state (`awaiting_evidence`, `blocked`), or replaces the owner, boot or generation. Each of these still fences the late owner out. All other untouched checks are unchanged. - `native-execution-lease-reacquire.test.ts`: new cases for each claim kind, a requested claim, failure settlement under the claim, and eight fenced variants. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new rule. ## Verification - Red/green: `cd server && npx vitest run src/services/native-runtime/native-execution-lease-reacquire.test.ts`. Without the source change, 5 new cases fail (the three claim kinds, the requested claim, and failure settlement). With it, all 37 pass. - Related suites pass: `native-execution-lease.test.ts`, `heartbeat-restart-claim-lapse.test.ts`, `heartbeat-native-orphan-sweep-lease-race.test.ts`, `native-runner-restart-lease-renewal.integration.test.ts`, `native-runner-restart-preparation.integration.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: the same L7 wave (seed 9107, local) on this branch. The server log has 0 `native_session_lease_lost` (the baseline had 3). The two tasks whose cells failed this way in the baseline (codex C1 TM1, codex C1 TM3) now pass with no Retry. Two other tasks needed Retry for different causes, which this pull request does not change: codex B5 TM2 (a `runner.drain` timeout, then a same-run retry that may not continue a turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on `turn.start` until the retry budget ran out). The L6 wave (seed 9106) also ran on this branch: 5 of 6 passed. Its one failure is `execution_recovery_budget_exhausted` and is not related to leases. ## Risks - The rule widens when an owner may take back its own lapsed lease. It applies only to local runs, before a result, with the same owner token, attempt, boot, pid, process start and generation. A claim that any other actor changed afterwards does not match. - Rows saved by earlier releases carry the same history entry shape, so a run claimed before a deploy is handled the same way. A deploy changes the boot, so the new process never takes back the old process's lease. - No schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 14:31:15 -06:00
DottaandClaude Opus 5.5	fd6e71a7ff	fix(native): restore a warm backup when its source sandbox is idle (#20627) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes keep a warm sandbox and a verified harness backup so the next run continues the same provider session > - A run that starts while the previous run's workspace export is still saving gets a new (created) sandbox lease, because reuse skips an exporting sandbox > - When that run prepares the harness, it finds a backup but no recorded reusable lease, and it fails closed with `runner_harness_state_mismatch: backup_without_reusable_lease` > - #20278 accepted a destroyed source and a settled restore hold as retired backup sources, but not an idle warm source that is still `retained` for reuse > - This pull request accepts an idle warm source under the same proofs, claims it so it is never resumed again, and covers the resumed lease that the failed run leaves behind > - The benefit is that these recoveries continue the same session automatically instead of blocking the task ## Linked Issues or Issue Description Refs #20278 **What happened?** A monitor wake started about 90 seconds after the previous run on the same task finished. The previous run's warm sandbox lease was `retained` with policy `reuse_by_environment`, but its workspace export was still saving. Reuse skipped it and created a new sandbox. Workspace realization waited for the export (about 136 s) and restored the saved files. Then harness preparation found the previous run's backup (its lease stamp names this exact manifest, and no provider work followed it). The source lease was not destroyed and was not a restore hold, so the run failed with `native_session_recovery_failed: Error: runner_harness_state_mismatch: backup_without_reusable_lease` and the task was blocked. **What did you expect?** The run continues the same provider session. The idle source sandbox holds exactly the backed-up state, so restoring the backup into the new sandbox is safe. **Steps to reproduce** 1. Finish a warm native run on a remote sandbox. Its workspace export is still saving. 2. Start the next run on the same task before the export settles. 3. Reuse skips the exporting sandbox and creates a new lease. Harness preparation fails with `backup_without_reusable_lease`. 4. A retry resumes that new lease, which has no harness state, and fails with `resumed_unavailable`. ## What Changed - `native-retired-backup-source.ts`: an idle warm source is a third retired backup source. It must be `released` or `retained` with policy `reuse_by_environment`, its own run must be terminal and not the current run, and it must have no cleanup in flight, no restore hold or pending shutdown settlement, and no pending native workspace export. The backup stamp, single-owner chain, and later-provider-work checks still apply. - Before the restore, the idle source is claimed with a compare-and-swap on the checked row. The claim moves its `reusableSandboxLease` scope into a `reusableSandboxLeaseSuperseded` receipt. Reuse never resumes that older state again, and the next acquisition treats it as obsolete and destroys it. If a run resumed or changed the source first, the claim matches no row and the backup is not restored. - `environments.ts`: a cross-run reuse handoff now requires that the source row has no superseded receipt. The supersede claim and the handoff compete on the same row, so an acquisition that selected the source before the claim cannot resume it afterward. - The proof now rejects a backup whose source is the target allocation itself (`targetProviderLeaseId`). - `native-session-executor.ts`: a resumed lease whose sandbox has no harness state at all (`unavailable`) restores the backup under the same proof. This recovers the lease that the failed run left behind, so rows saved by older releases recover after deploy without a migration. - `doc/execution-semantics.md`: documents the idle-source rule, the supersede claim, and the resumed stateless lease. ## Verification - Red before the fix, green after: - `native-session-executor.test.ts` "restores a warm backup into a created lease only when an idle warm source finished its run and its export" failed with `backup_without_reusable_lease`. - "... a resumed lease never held the session and the warm source is idle" failed with `resumed_unavailable`. - `native-retired-backup-source.integration.test.ts` (real Postgres) "authorizes an idle retained/released warm source and claims it as superseded" failed (returned false). - Integration test "refuses a reuse handoff of a source superseded while its provider resume was pending" fails without the handoff guard and passes with it. - Safety rows stay refused: source run still running, workspace export pending, cleanup in flight, restore into its own allocation, stamp names another checkpoint, provider work after the backup, two host leases for one allocation, claim lost to another run, and a resumed lease that lost its own state. Existing destroyed-source and restore-hold rows are unchanged. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-retired-backup-source.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts` passes (1408 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 because a global CI input (`doc/execution-semantics.md`) changed. Required CI supplies the broad coverage. ## Risks - Previous-release story: no persisted shape changes. Lease rows saved by earlier releases carry forward. The new `reusableSandboxLeaseSuperseded` metadata key is additive. - The idle source is destroyed at the next acquisition for its workspace. Its state is the verified backup, which the restored lease now holds, so no work is lost. - Not covered: when the source cannot be proven retired or idle (for example a live owner or later provider work), the run still fails closed with the same named code. An automatic fresh-session fallback for that case is not in this PR. - The root trigger (reuse skips a sandbox while its export saves, then waits for that export) is unchanged. This PR does not change the reuse lookup. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 14:11:10 -06:00
DottaandPaperclip	76c787fffc	fix(runner): report monitor notes length limits (#20604) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use task monitors to resume a future check. > - The monitor schema limits notes to 500 characters. > - The field description did not state the limit. Rejection showed only the schema path. > - This pull request adds the limit to the description and to safe length errors. > - Agents can correct notes before they retry a rejected call. ## Linked Issues or Issue Description Related prior fix: #16868. **What happened?** A native `set_task_monitor` call with 501 characters in `monitor.notes` failed with a path that ended in `notes/maxLength`. The error did not state 500. The schema already had `maxLength: 500`, but the field description did not state it. **Expected behavior** State the limit in the field description and in the rejection. Accept 500 characters. Reject 501 characters before dispatch. Preserve a saved monitor when a replacement is invalid. **Steps to reproduce** 1. Supply a future `monitor.nextCheckAt` and 501 characters in `monitor.notes`. 2. Read the native provider error. 3. Before this fix, it shows the constraint path without the maximum. **Paperclip version or commit** Reproduced on `7368a0b15c2f02107778a961144d86e2eb59a952` in a Linux source checkout, with Node 25.9.0 and Rust 1.97.1. ## What Changed - Use one local value for the published notes limit and its description. The server test checks agreement with the persisted shared validator. Keep the standalone Runner package boundary. - Read the numeric maximum from the authorized JSON Schema validation error. Keep submitted values private and preserve the 512-byte feedback bound. - Test the published schema and real bridge dispatch path at 500 and 501 characters. Test saved contents and rejection without monitor changes with real PostgreSQL. - Refresh generated contracts and document the limit. - Sync two SDK proof lock identities after the reviewed master lock refresh in #20615. This fixes the remaining CI proof mismatch. Keep compiler, parser, Zod, and all rejection checks unchanged. ## Verification - Final head: `bac60234ab3a8fc7043a90c7f9c81b02e250428b`. - Red: the server regression failed because the field description omitted `500`. The Rust regression failed because the error omitted `maximum 500 characters`. - Local passes after the final base update: 44 PostgreSQL monitor/scheduler tests, 14 real-source Rust bridge tests, 11 cache/Docker guard tests, and 56 SDK proof tests. Runner TypeScript build and generated contract checks pass. - The focused Rust check compiles the actual source in a small Cargo workspace. The full Rust CI suites also pass on Linux and Windows. - Final-head [PR CI run](https://github.com/paperclipai/paperclip/actions/runs/42881633978) passes. This includes typecheck, build, Runner support, Runner Rust and Vitest, server tests, serialized tests, and all eight e2e shards. - Final-head [Windows Rust run](https://github.com/paperclipai/paperclip/actions/runs/42881633419) passes. - Greptile: 5/5 on the final head, with no open findings. There are 61 successful checks and five conditional skips. The separate Paperclip Review remains queued; its comment says work is paused. No CI or test check is pending or failed. - `pnpm verify` returned exit 2 and selected broader CI. The full local suite was not run. Final-head CI supplies that coverage. Earlier failed attempts were repaired and are not counted as passes. ## Risks - Low risk. The existing 500-character bound, scheduling semantics, and permissions stay in place. - Length feedback uses only a schema path and numeric bound. It does not print submitted text. - The Runner binary and tool contracts must reach an instance for the new guidance to apply. This PR does not deploy or merge. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The runtime does not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 13:59:58 -06:00
Nicky LeachandPaperclip	19e938415a	fix(runtime): omit Codex refresh handoffs from backups (#20614) ... ## Thinking Path > - Managed ACPX Codex retains its final refresh after provider exit so Core can save it. > - Shutdown saves a checkpoint before Core removes the live credential handoff. > - Checkpoint profiles must exclude the handoff while retaining resumable session history. ## Linked Issues or Issue Description Follow-up to #20580 and its credential-backup review finding. Refs #20457. ## What Changed - Exclude `auth-refresh.json` and `auth-refresh.json.tmp` from the shared Codex checkpoint profile, covering native and ACPX homes. - Extend real checkpoint-copy coverage to ACPX Codex, both handoff filenames, and native Codex. Keep the live files and session history intact. ## Verification - A real ACPX checkpoint-copy test failed before the exclusion fix; Grok passed. - Five targeted real-copy/profile tests pass, including after applying the change to current master. - `git diff --check` and local secret/PII diff review pass. - Full regression coverage runs in GitHub CI at the maintainer's request. No live provider credentials were used. ## Risks - Two entries added to an existing exclusion list; no runtime lifecycle, schema, persisted identity, or session format changes. Saved sessions remain readable. Existing backup restoration uses the same exclusion profile. - Provider credential cleanup still follows final writeback; excluding a backup entry does not delete the live handoff. ## Model Used - OpenAI GPT-6 through Codex, with tool-assisted implementation and verification. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 18:57:03 +00:00
Nicky Leach	17136a6c25	fix(runtime): persist managed Runner credential refreshes across execution targets (#20580)	2026-11-10 10:15:29 -08:00
DottaandPaperclip	85dec04677	fix(native): validate active controller durability during eligibility (#20601) ... ## Thinking Path > - Paperclip controls agents while their work runs outside the web server. > - A web replacement must retain each controller's durable recovery state. > - Modern controllers commit that state to the database before sending work. > - Their local controller file is only an external-persistence identity marker. > - Disk eligibility still required recovery metadata that can arrive after a run finishes. > - This pull request verifies the complete active database authority instead. > - It also bounds concurrent source reads so a large history does not require serial network waits. ## Linked Issues or Issue Description Refs: #20311, #20584, #20599. **What happened?** A live native run can have a complete database controller checkpoint and no terminal session snapshot yet. The disk-preservation check treats its local identity marker as a local-only obligation. The check also resolves selected sources one at a time. **Expected behavior** Accept a complete, bound active recovery authority. Refuse missing, partial, stale or foreign evidence. Keep all execution and enrollment guards. **Steps to reproduce** Use the new active-controller regression. Start a real Runner controller with the database checkpoint writer. Keep the business run active and omit its terminal snapshot metadata. The previous check returns a local obligation even after the full checkpoint commits. ## What Changed - Require the exact external-persistence marker and a complete decoded encrypted checkpoint for active runs. Match run, agent, task, session, Runner and workspace identities. For warm reuse, also permit an older marker only when its full older checkpoint matches that marker and its exact earlier successful run has a committed accepted result in the same company, agent, task, session, Runner and workspace. Keep the current live writer mandatory. Read the marker again after validation. - Accept that proof only with its fenced live writer and task binding, or the existing exact saved recovery snapshot. The saved-snapshot path still requires the full active journal. - Keep newer remote workspace authority strict. A malformed reference cannot fall back to an older prepared reference. Conflicting plugin task authority remains blocked. - Resolve selected sources at most eight at a time for read-only eligibility. Join every started reader before reporting an error. Keep source order, pagination, predicates and all write paths unchanged. Offline migration and prestaging still resolve sources one at a time. - Add real PostgreSQL regressions for active durability, partial and foreign state, expired ownership, previous prepared-reference state, post-result state and reader failure joining. ## Verification - Canonical Node24 preflight passed in74.2s. It built the current Runner inputs and published the cache. No shared Cargo target or hosted database was used. - Genuine active regression RED: the previous check returned true when false was required. The real checkpoint writer had committed the complete authority and replaced the local file with the Runner marker. Treatment passed, including the refusal controls. - Genuine source-reader RED: sixteen retained pointers used one reader at a time. Treatment used at most eight and joined all eight when the first reader failed. With a controlled20ms reader delay and real PostgreSQL selection, the small fixture changed from497.5ms to85.3ms. This is a bounded-concurrency proof, not a production speedup measurement. - All54 cases in the affected controller migration integration file passed in76.07s, with no skips. These include existing failed-controller, archived-controller, inline database and previous-session preservation cases. - Server TypeScript passed. The normal merge of current master kept both changed files byte-for-byte identical to the54-case proof. Final post-merge focused checks passed two of two in14.83s (52 other cases intentionally unselected); server TypeScript and module boundaries passed. Comment-only correction743 removed all new code comments required by CONTRIBUTING. A later warm-reuse regression reproduced the remaining false blocker through the real Runner rotateRunIdentity API. The bounded treatment uses one exact older-run lookup and full old/current checkpoint validation; it adds no Runner format, release or lineage service. Final normal-merge head `c62d6eb1755cf99b6def9563ba69854dba17de42` includes the new0459 baseline. All three focused PostgreSQL cases passed in21.37s (52 existing cases intentionally unselected); fresh server TypeScript and module boundaries passed. The prior54-case regression evidence is retained; no full matrix was repeated. - `git diff --check` passed. Final-head CI passed:56 checks succeeded and4 were skipped; the separate Paperclip Review advisory was still queued. All required test, build, typecheck and ci/verify gates passed on c62. [Final required workflow](https://github.com/paperclipai/paperclip/actions/runs/42868279612). Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/20601#issuecomment-6774537057) is5/5 on exact head `c62d6eb1755cf99b6def9563ba69854dba17de42`, with no open review threads. ```sh node scripts/agent-preflight.mjs --skip-rust-debug node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts --disableConsoleIntercept pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reports preservation evidence. It does not grant attachment, execution, replacement or enrollment authority. The normal live-owner and admission gates still apply. - A partial old or current checkpoint, stale local controller state, identity mismatch or uncertain owner stays blocked. Warm-marker proof also rejects a foreign agent, newer source, later completion, rejected result, per-turn scope or expired current writer. No source, checkpoint, session, lease, archive or provider state is changed by the check. - The eight readers add bounded database concurrency during an explicit eligibility request. There is no new background scan, cache, row filter or timeout change. Live enumeration time still needs deployed measurement. - The plugin task-lease branch uses the producer's environment lease row ID, as native-session-executor and existing diskless adoption coverage require. A direct active-eligibility positive for that plugin branch is a coverage limit; existing mismatch/conflicting-authority checks remain. The advisory found no must-fix defect, and this does not change the current tested revision. - Saved formats and terminal migration behavior remain unchanged. Older prepared workspace references remain supported. A fully saved result can still pass before the run-status update, but its entire active checkpoint must decode. - The observed staging run finished naturally and became valid on the existing terminal path during investigation. That observation is not deployed verification of the new active path. ## Model Used OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. This runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 12:02:22 -06:00
DottaandClaude Opus 5.5	1b7ec13acb	perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can run in remote sandboxes (Daytona), where each turn needs the task's workspace and a runner inside the sandbox > - With a default environment, every remote turn got a brand-new sandbox: the whole workspace was uploaded again and a new runner started, even for a follow-up comment > - In a paperclip-sized workspace that made a follow-up take 30 s to first token, against 8–10 s with a reusable warm sandbox > - This pull request makes an agent that never chose a lifecycle run warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude), so a follow-up keeps its sandbox, runner and workspace > - The default applies only where warm can work: a sandbox provider that keeps reusable leases, and a managed AI account that supports a warm session. Everywhere else the run is per-turn, exactly as before > - The benefit is fast remote follow-ups by default, without re-uploading a large repository every turn ## Linked Issues or Issue Description **What happened?** On Daytona, with a fresh paperclip clone as the project workspace, a follow-up comment took 30 s to its first token. The run log showed `environment: fresh/created` on every follow-up. It spent 13.3 s in `workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s in `runner.session.resume`. **Steps to reproduce** 1. Create a Daytona environment with the defaults: lifecycle inherited, lease reuse off. 2. Run a Codex task in a large repository, then post a follow-up. 3. The follow-up gets a new sandbox, uploads the whole workspace again and starts a new runner. **Expected behavior** A follow-up on the same task reuses the live sandbox, its runner and its workspace (plan rule 2: no full re-upload per turn). ## What Changed - **One rule for the default.** `defaultsRemoteSandboxRunToWarm` (`native-runtime/native-warm-capability.ts`) decides the remote-sandbox warm default for both the lease (environment view) and the runner (native session policy), in one vocabulary (runtime backend plus ACPX agent). It is applied only when no lifecycle was chosen anywhere: admitted session, environment `runnerLifecycleMode`, or agent `lifecycleMode`. It holds only when: 1. the harness is measured (Codex, OpenCode, ACPX Claude); 2. the sandbox provider keeps reusable leases (Daytona, CreateOS and boxd declare them; E2B, Modal and the built-in `fake` do not); and 3. with a managed AI account, that harness supports a managed warm session (`supportsManagedNativeWarmSession`: Codex). - **The harness resolves as the runtime resolves it.** `resolveWarmProviderSelection` uses `resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is Codex and ACPX without an agent is Claude, in both decisions. A config the runtime would refuse gets no default. - **Lease side** (`runner-environment-lifecycle.ts`, `environment-run-orchestrator.ts`): for an eligible run only, the orchestrator asks the provider before acquisition (`providerSupportsReusableLeases` in `environment-runtime.ts`: the acquisition rule, one registry lookup for a plugin provider, no worker-ready wait). With support, the run-scoped view sets `reuseLease: true` and forwards the warm idle window to the provider's auto-stop. The shared environment is never rewritten. Every other run resolves exactly as on master, without the probe. - **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the native policy applies the same rule to the acquired lease (`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If acquisition downgrades the lease anyway, the run falls back to per-turn; a default never produces `runner_warm_environment_requires_reusable_lease`. - **Idle window.** The remote default uses the window the environment view forwards to the provider: an inheriting environment's `runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes. The sandbox execution target now carries the environment's window (`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution target) so the runner and the provider auto-stop agree. An explicit warm choice and the local default are unchanged. - Unchanged: explicit agent choices, environments that pin a lifecycle, plugin-task environments, admitted sessions (`persistedPolicy`), and local behavior. - `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit --lifecycle default` sandboxes as retained, because they now default to warm. - `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the ranked waits, and the gate. ## Verification - Unit and integration tests, one per rule: - Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease → per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases → environment unchanged, no warm window), `environment-runtime-reusable-lease-probe.test.ts` (`fake`, an undeclared plugin, a declared plugin whose worker does not verify the reuse methods, and a plugin without a running worker read false), `runtime-selection.test.ts` ("defaults an agent without a lifecycle on ...": reusable → warm/keep_running; reuse configured on an unsupported provider → per_turn/destroy_after_turn; ephemeral → per_turn), `environment-run-orchestrator.test.ts` (probe result decides `reuseLease` and the forwarded window). - Rule 2 (explicit configuration wins): explicit per-turn agent, pinned environment, admitted session, configured `idleTimeoutMs`, managed Claude per-turn, managed Codex warm; the orchestrator does not probe when a lifecycle is explicit. - Rule 3 (omitted provider): `{}` resolves to Codex in the environment view; `runtime-selection.test.ts` "agrees with the environment view's lease for ..." feeds the environment view's lease into selection for Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without an agent, and Copilot (per-turn), and checks both decisions agree. A plugin-task target stays out of the default. - The two suites that failed on the previous head (`runtime-selection.test.ts`, `workspace-recovery-preparation-dispatch.test.ts`) pass locally. Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a nested worktree; they do not touch this change and pass in CI. - End-to-end check of this default on Daytona (before the gate was added; Daytona keeps reusable leases, so the gate does not change this path): a user-default environment (lifecycle inherited, lease reuse unset, `--daytona-inherit`), a fresh paperclip clone, built from the image's source revision with this change (branch measure/remote-warm). | | Before (per-turn sandbox) | After (this change) | |---|---|---| | Follow-up, Codex | 30.4 s | 7.6 s | | Follow-up, Claude | about 30 s | 11.0 s | | First follow-up after an idle K1, Codex | 45.9 s | 20.3 s | | First follow-up after an idle K1, Claude | 38.3 s | 18.3 s | The run-log reuse records show the follow-ups reusing the workspace, environment, runner and provider session, with no `workspace_sync.transfer`. First turns still upload the workspace once (14.8 s and 26.1 s). Every sandbox was cleaned up. - Daytona restart chaos wave (K1/K2/K4 with a mid-run comment, inheriting environment, agents with no lifecycle, runner image built from this branch): wave `w-20261110T171438-s20048`, seed 20048, on b7f6243d60 with image `paperclip-daytona-runner@sha256:76a33275…6fe29` (built from this branch by the `image_only` runner workflow; runner content id `ed503454…a479` matches every later commit here, which change only server, adapter-utils types and tests). `--daytona-inherit --lifecycle default --adapters codex,claude --workloads A1,C1 --triggers time,comment --faults K1,K2,K4 --tasks 6 --concurrency 3`. - **6/6 tasks pass** (R1–R6 and T), all 6 crossed a restart; cells: Codex A1 K2, Codex A1 K4, Claude A1 K4, Claude C1 K1, Claude C1 K4, Codex C1 K2 at the mid-run comment. **No task needed Retry** (R3 n/a on every task; one Claude task recovered automatically from `process_lost`). No open signatures, 0 speed breaches. - The default applied: the R6 sweep found 6 retained reusable leases (one per task, Daytona keeps them) and no orphan or leaked sandbox; the environment was deleted cleanly. - Timings (Daytona, first turns, n=3 each): TTFT p50 Codex 21.9 s, Claude 31.0 s. Restart recovery p50/p95: Codex 10.5 s / 29.6 s (n=3), Claude 8.8 s / 23.9 s (n=2). Warm follow-up TTFT has no clean sample in this wave (both follow-ups were crossed by a fault), so the follow-up comparison above (30.4 s → 7.6 s Codex, ~30 s → 11.0 s Claude) remains the speed evidence; no separate per-turn chaos wave was run. - An earlier attempt, `w-20261110T170357-s20048`, never started tasks: the first sandbox from the new image stayed `creating` past the harness's 300 s Codex qualification bound (Daytona building the new snapshot). It left one sandbox that the harness reported after deleting the environment; the rerun on the cached snapshot is the wave above. - The idle-window fix (9a74116082) landed after the wave; it changes only runs whose inheriting environment sets `runnerIdleTimeoutMs`, which this wave's environment does not. ## Previous-release story No persisted state changes shape. A session admitted per-turn before this change keeps its saved policy (`persistedPolicy` wins) until a new session starts. A run in flight at the cutover keeps its admitted lifecycle for recovery (`admittedLifecycleMode` wins in the environment view). ## Cost and opt-out - **Cost:** after a turn, the sandbox and its warm runner stay up for the warm idle window (the environment's `runnerIdleTimeoutMs`, else the agent's `idleTimeoutMs`, else 5 minutes). Then the controller suspends the runner, and the provider auto-stop follows the same window. One idle window per active task. - **Opt-out:** set the agent's lifecycle to "Turn by turn" (`lifecycleMode: per_turn`), or set the environment's `runnerLifecycleMode` to `per_turn`. Either wins over this default. ## Reuse scope, restart and saved work - **One lineage per sandbox.** The Daytona plugin keys a reusable lease by the agent plus the execution workspace (or issue). Another task never gets a sandbox holding a different task's files, and subtasks still get fresh workspaces from the project base, so they get their own sandbox. The workspace is adopted only after its stamp matches the host's. - **After a restart,** a warm sandbox is reattached or replaced through the existing warm recovery and owner fencing. A sandbox with a live owner is never claimed twice. The Daytona chaos wave above checks this. - **No lost work.** Each turn's workspace copyback (`workspace_finalize`) runs at the end of the turn, before any idle suspend, as for per-turn runs. ## Risks - **Cost:** an idle sandbox stays up for the warm idle window (5 minutes) after a turn, then suspends. Only on providers with reusable leases. - **Probe cost:** one plugin-registry lookup before acquisition, only for runs the default can apply to. No wait for a worker: a provider whose worker is not running yet reads as unsupported, so that run is per-turn. - **Known gap (pre-existing, not widened):** with an explicit agent warm choice on an inheriting environment that sets `runnerIdleTimeoutMs`, the provider gets the environment's window while the host uses the agent's, as on master. - **Restart behavior on warm remote sandboxes** uses the existing warm recovery paths. The Daytona chaos wave above checks K1, K2 and K4. - Sessions admitted per-turn before this change stay per-turn until a new session starts. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution, Daytona measurements). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 11:49:14 -06:00
DottaandClaude	c37eacbee3	fix(native): Retry re-finalizes a saved result that ran out of finalization retries (#20578) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save an accepted result. Then a finalizer applies that result to the task. > - Finalization has its own budget of 3 failed attempts (#20555). When the budget runs out, the coordinator goes to `terminal_failure` with `native_finalization_retry_exhausted`. > - For a provider-succeeded run, the task then rests `in_review` with a board recovery card. The task shows "resolved" and offers no Retry. The user is stuck. > - The ordinary Retry is also wrong here: it starts a new provider run, which repeats work the agent already finished. > - This pull request adds two things. First, contention failures on such a result get two longer automatic delays before the board gets the task. Second, when the state is still exhausted, the task offers Retry, and that Retry applies the same saved result again. > - The benefit: fewer tasks need a Retry, and when one does, Retry works and does no duplicate provider work. ## Linked Issues or Issue Description Refs #20555 (it names this gap under Risks). **What happened?** In restart-chaos wave `w-20261110T044338-s20385`, task t03 ended with a succeeded run, an accepted result, and a coordinator in `terminal_failure` / `native_finalization_retry_exhausted`. The issue stayed `in_review` with a board-owned `active_run_watchdog` card and `canRetry=false`. #20555 stopped the early exhaustion. A real exhaustion still leaves the same stuck state. Coordinators that older builds already left in this state are stuck the same way. **Expected behavior** Transient finalization failures recover by themselves when that is safe. When they do not, the task offers Retry. That Retry re-finalizes the saved result. It does not start a provider run and does not charge a provider attempt. **Steps to reproduce** 1. Let a native run save an accepted result and end `succeeded`. 2. Make each finalization of that result fail until the budget of 3 runs out. 3. Before this fix, the task presentation is `resolved` with `canRetry=false`, and the board card has no action that applies the result. The new test `server/src/__tests__/native-exhausted-finalization-retry.test.ts` reproduces this. **Paperclip version or commit** master at c58356003f (after #20555). ## What Changed - `native-run-finalizer.ts` (`recordRetryableFailure`): when the budget runs out on an accepted result of a provider-succeeded run, and the cause is contention (`status_cas_exhausted` or `side_effect_planning_failed`), the coordinator stays `retryable_failure`. It tries again after 2 minutes, then after 10 minutes, each with a fresh budget. Each delay is recorded once in `recoveryHistory` (`automatic_finalization_backoff`). After both delays, the old terminal path runs. Deterministic causes (for example `native_finalization_invalid`) and workspace-scope failures are unchanged. The existing deadline worker (`native-result-commit-work`) runs the retry, so no new timer or poll is added. - New `native-runtime/exhausted-finalization-retry.ts`: - `exhaustedNativeFinalizationCondition()`: one SQL predicate for "succeeded native run, accepted result, coordinator `terminal_failure` with `native_finalization_retry_exhausted`, no lease, no successor". - `rearmExhaustedNativeFinalization()`: under the task lock, reopens the same coordinator (`arbitrating` or `workspace_finalizing`), records `exhausted_finalization_retry` in `recoveryHistory`, resolves the board card (`handed_back`), and resets `finalizationFailureCount` to 0. It sets a 30 s fallback `nextAttemptAt`, so the deadline worker finishes the job if the process stops first. - `refinalizeSavedNativeResult()`: runs `finalizeNativeRun` for the same run with `preserveProviderAttempt: true`. A failure goes through the finalizer's normal bounded budget. - `task-interruption-presentation.ts`: the run read adds `nativeFinalizationRetry` from that predicate. A succeeded run with this flag shows `state: "failed"`, `canRetry: true`. It also adds `nativeFinalizationRetryPending` (`savedResultRetryPendingCondition()`: coordinator not committed or terminal, with an `exhausted_finalization_retry` entry). While it is true the task shows `state: "working"` with `recoveryStep: "finishing_recovery"`, so the user sees the Retry took effect. Done or cancelled tasks, user gates, and ordinary runs are unchanged. - `task-retry.ts` (`admitTaskRetry`): for a succeeded native run in the exhausted state, Retry re-arms the coordinator instead of creating a provider retry. It writes a `coalesced` `task_retry` receipt bound to the same run with strategy `finalize_result` (an existing strategy value), joins any unadmitted Retry intent to it, and logs `issue.retry_requested` with `strategy: "finalize_result"`. The idempotency key still returns the same receipt on replay. A later click with a new key, after an earlier saved-result Retry of the same run (still finalizing, or applied), joins that Retry with another coalesced `finalize_result` receipt and starts no provider work. - `routes/issues.ts` (`POST /issues/:id/retry`): after the receipt commits, it starts `refinalizeSavedNativeResult` for that run instead of the provider Retry dispatch. - `doc/execution-semantics.md`: documents the automatic delays and the re-finalizing Retry. ## Verification - `cd server && npx vitest run src/__tests__/native-exhausted-finalization-retry.test.ts` (4 tests): - Exhausted state reports `state: "failed"`, `canRetry: true`. Retry re-arms the coordinator, writes a `finalize_result` receipt and an `issue.retry_requested` activity row, and replays idempotently. A second click with a new key while pending, and a third after commit, join the same Retry; no provider Retry wake exists. While pending, the task shows `working` / `finishing_recovery`. Re-finalization commits, and the task shows resolved. The run count for the task stays 1, the coordinator `attempt` is unchanged, the run stays `succeeded`, and no board card stays active. - Old-shape row (no `finalizationFailureCount`, `attempt: 3`, already terminal): the same Retry works with no migration. - Re-finalization that fails again: the coordinator goes to `retryable_failure` with a fresh count of 1, then to `terminal_failure` after the budget, and the task again shows `canRetry: true`. Nothing is lost silently. - Contention failures (failpoint `status_projection`): two automatic delays (2 min, 10 min) with the task status unchanged, then the board Retry, which commits with no new run. - Red/green: with only the new module and test applied (finalizer, presentation, task Retry, and route reverted), all 4 tests fail (`canRetry` false; no automatic delay). - `npx vitest run src/services/task-interruption-presentation.test.ts` adds unit cases: the flag offers Retry; ordinary succeeded runs stay resolved; ordinary failed runs keep `canRetry: true`; done tasks stay resolved. - Also passing: `native-finalization-retry-budget`, `native-finalization-recovery`, `native-run-finalizer-telemetry`, `task-interruption-query`, `interrupted-preparation-restore-retry`, `native-result-commit-work`, `workspace-recovery-admission`, `workspace-recovery-preparation-gates`, `deferred-execution-receipt-work`. - `npx tsc --noEmit -p server/tsconfig.json` passes. - Not run: the full local suite (`pnpm verify` selects about 2,450 files for these shared modules; CI covers it), and no live restart-chaos wave. ## Risks - Previous-release story: coordinators saved by older builds in `terminal_failure` / `native_finalization_retry_exhausted` qualify by phase and code. The count is optional. Nothing is migrated. Older rows do not get the automatic delays; they get the working Retry. - Automatic delays add at most two extra finalization rounds of the same saved result, only for contention causes, only for provider-succeeded accepted results. During the delays the task keeps its prior status, as it does during the normal 30 s retries. Supersession by a newer run is still checked first on every attempt. - The re-finalizing Retry applies the agent's saved result after a person may have looked at the task in review. It uses the normal status arbiter and CAS. Done and cancelled tasks, other assignees, and newer runs refuse it. - No change before `turn.start`: all new work is on the finalization, presentation, and Retry paths. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, code editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 11:48:47 -06:00
DottaandPaperclip	1c78d023c6	perf(native): narrow backup eligibility to selected runs (#20599) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Server replacement checks whether retained native sessions still depend on local harness data. > - This check joins selected task sessions, runs, backup stamps, and durable backup records. > - PostgreSQL can start from backup leases and scan an issue's full run history before applying the selected-run condition. > - A measured instance spent 9.56 seconds in that selector and exceeded an 8-second statement limit. > - This pull request selects the exact current run identities first, then applies the existing lease and backup checks. > - The same receipt validation and eligibility decisions remain in place. ## Linked Issues or Issue Description **What happened?** Startup eligibility timed out while selecting native harness backup obligations. The plan scanned historical runs for each lease before checking the task session's selected run. **Expected behavior** The selector should use the selected run identities without scanning unrelated run history. It must return the same session, scope, company, and receipt flag. **Steps to reproduce** Use a database with retained native task sessions, many historical runs on the same issues, and version-two backup stamps. Run startup eligibility with an 8-second statement timeout. The lease-first plan can exceed that limit. **Paperclip version or commit** The local baseline is `d0c30e69a9cea912db7c5290c5d585fdd63ec228`. **Deployment mode** Self-hosted server with PostgreSQL. ## What Changed - Keep `AS MATERIALIZED` as an intentional join-order fence: build selected task-session and last-run identities before joining backup leases. - Keep only narrow identity columns in that materialized set. Read the JSON receipt flag through the selected run's primary key after the lease and current-backup filters, so profiles for unrelated identities do not widen the intermediate rows. - Preserve company and agent bindings, exact normalized-session text comparison, current-backup exclusion, scope deduplication, ordering, and the 100-row page. - Add real PostgreSQL regressions for full ordered-row equivalence, nullable JSON receipt flags, malformed and uppercase session bindings, cross-company rows, duplicate stamps, page tails, and invalid receipts. - Existing saved state carries forward under the same validation rules. No schema or persisted identity changes are required. ## Verification - The two focused PostgreSQL regressions passed locally. Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-obligations.integration.test.ts --silent`. - Worktree preflight passed. The focused regressions (2/2) and `node server/scripts/typecheck-layers.mjs` passed on final head `c739353b22` after rebase. - One bounded read-only pair on PostgreSQL 17.11 returned the same 54 complete ordered rows: baseline 9,564 ms, candidate 298 ms (32.1 times faster). The plan changed from lease-to-issue-history scans to selected-run primary-key lookups and a hash join. - A synthetic PostgreSQL 18.1 fixture already chose an efficient baseline plan. Across five alternating warm pairs, its median was 104.6 ms before and 109.4 ms after. This neutral control used 22,800 runs, 14,100 leases, 3,173 task sessions, and 5,460 backups; all 54 complete rows matched. The empty control also matched. - These measurements cover the selector. A later read-only assembled-call probe on a newer serving release substituted only this selector: the selector took 544 ms for 54 rows, while the full eligibility call took 56.1 seconds and still reported a controller migration blocker. Other checks remain outside this PR; this PR does not claim to bring the whole request within its deadline. - Required CI and fresh review results will be recorded here before handoff. ## Risks - Materialization adds a selected-identity pass. Databases that already choose an efficient plan may see little benefit, as the local neutral control shows. - Pagination and receipt validation must stay exact. The regression crosses the 100-row boundary and checks that an invalid final scope still blocks eligibility. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are unavailable in this session. The agent used reasoning, code editing, shell execution, PostgreSQL diagnostics, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 11:37:13 -06:00
DottaandClaude Opus 5.5	be0d100a8b	fix(native): settle cleanup-gate refusals after a completed turn without a dead-end Retry (#20572) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute a provider turn, then close the provider session and prove it stopped > - When close cannot prove that the old runner, provider and tools stopped, the runner holds a cleanup gate (`native_session_cleanup_quarantined`) that refuses the next session on that task > - A run finished its provider turn and saved its work. A server restart had left one tool callback without a saved result, so close could not prove the drain, and the gate refused the run's own same-run recovery 30 seconds later > - The task then showed "Couldn't finish this attempt" with a Retry button, but the retry route refuses this error code with 409. The saved error text told an operator to act > - This pull request makes the task projection, the execution blocker, the executor and the retry route agree, and replaces the operator text with plain wording > - The benefit is that finished work no longer looks like a failed attempt, and the UI never offers a Retry that the server refuses ## Linked Issues or Issue Description **What happened?** A remote native run completed its provider turn and saved its comments and documents. A server restart during the turn left one semantic tool callback without a saved result. Session close then reported `native_session_settlement_incomplete` (runner suspended, provider drain unproven), and the runner kept an operator-required cleanup gate. The same-run disposition recovery was refused at admission, and the run ended `failed` with `native_session_cleanup_quarantined`. **Expected behavior** The task should show its real state (here, waiting on a child task). It should not show a failure banner or a Retry that the server refuses. No text should ask an administrator to act. **Steps to reproduce** 1. Start a remote native run and restart the server while a tool callback runs. 2. Let the resumed turn complete without a semantic result, so a same-run recovery is scheduled. 3. Observe the refusal at admission, the "Couldn't finish this attempt" banner, and a Retry that returns 409. **Paperclip version or commit** master at d0057ee384 (remote native runner, Daytona sandbox). ## What Changed - Add `server/src/services/native-runtime/completed-turn-cleanup.ts`. It holds one predicate for "a cleanup-gate refusal after this run's own provider turn completed" and the plain user wording. It reads only the saved `errorCode` and `runnerProfileJson.sessionCheckpoint`. - `task-interruption-presentation.ts`: the projection never sets `canRetry: true` for a cleanup-quarantined run, which matches `routes/agents.ts` and `canRetryStoppedRun`. A quarantine after a completed turn projects as `resolved` only while the task waits on unresolved blocker tasks, so the task shows that wait. Without such a wait, the missing task disposition stays visible. Other quarantined runs show a plain settling message without Retry. The list query computes the completed-turn fact in SQL. - `native-session-executor.ts`: the refusal keeps its permanent, named code, its hold and its ownership, and does not replay provider work. The next action uses plain wording, and names the saved work after a completed turn. - `execution-blocker.ts`: show plain wording for this code instead of the stored error text, also for rows that earlier releases saved. - Runner error message and retry-route conflict message use plain wording. - `doc/execution-semantics.md`: document how the cleanup gate settles and how the task shows it. ## Verification - Red before the fix, green after: - `server/src/services/task-interruption-presentation.test.ts` "native cleanup quarantine presentation" (2 failed before: `canRetry: true`, `state: failed`). - `server/src/__tests__/task-interruption-query.test.ts` "reads a quarantine saved after a completed turn as settled work" (real Postgres, row in the previous release's saved shape; before: `state: failed, canRetry: true`). - `server/src/services/native-runtime/native-session-executor.test.ts` "settles a cleanup refusal after a completed turn" (before: operator next action and board ownership). - `server/src/services/explicit-native-continuation.test.ts` "shows plain wording for a cleanup refusal saved with the earlier operator text" (before: stored operator text). - The four files pass in full: 1851 tests. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - `pnpm verify` returns exit 2 (a global doc input changed), so required CI supplies the broad coverage. - No UI files changed, so token gates and UI tests do not apply. ## Risks - Previous-release story: no persisted shape changes. Runs that earlier releases quarantined carry forward. The new rules read existing fields, so these runs show the new projection without a migration. - The code stays permanent and named. No automatic provider replay is added. - A quarantine whose turn did not complete keeps its board-owned recovery. Only its text and its Retry visibility change. - The runner still keeps the in-process cleanup gate until environment release proves the stop. Existing lease termination receipts (`acknowledgeRemoteStop`) clear it. This PR does not change the runner drain rule for a callback that a restart orphaned. That is a separate follow-up. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 11:09:59 -06:00
DottaandClaude	84007892e9	refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` is the instruction file that each human and AI contributor reads before a change > - Pull request #20557 shortened the file, but several sections still held detail that applies to only some changes > - The scoped-rules section told agents to read each matching rule in full, and most changes match five rules > - The core engineering rules mixed three product invariants with process conventions > - This pull request moves that detail to the docs and to scoped rules, and adds a section on how to fix a bug > - The benefit is a 211-line file that states what applies to every change and points to the rest ## Linked Issues or Issue Description Refs #20557 **Issue type** Outdated or unclear contributor documentation. **Where is the issue?** `AGENTS.md` at the repository root, and `rules/`. **What's wrong?** - The scoped-rules section made agents read the full body of each matching rule before a change. The rule bodies total about 47 KB. - Sections 4, 6, 8, 9 and 10 repeated detail that belongs in `doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and `CONTRIBUTING.md`. - Section 4 had a database reset command for `data/pglite`. That directory does not exist. - Section 5 had seven rules. Four of them were process conventions or path-specific detail. - The file did not say how to fix a bug. **Suggested fix** Make the rule headers the index. Move section detail to the document or rule that owns it. Keep only the three invariants in section 5. Add a bug-fixing section. ## What Changed - The scoped-rules section now tells agents to list the rule headers with one `grep` command. Agents read a rule body only for a design decision in its area, when the description is not sufficient, or when a reviewer cites it. - `rules/README.md` says that each `description` must state the required behavior in one sentence. - `rules/pull-request-review.mdc` has a description that states its requirements. - Section 4 (dev setup) and section 11 (pull requests) now point to `doc/DEVELOPING.md` and `CONTRIBUTING.md`. - The schema change workflow moved from section 6 to `doc/DATABASE.md`. - The API and auth expectations moved from section 8 to the new `rules/api-route-expectations.mdc`. - Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer tells contributors to run `pnpm check:token-gates`. - Added section 10, "Fixing a Bug": reproduce the error, write a failing test, write the fix, see the test pass, and verify the fix. - Section 5 now has three rules: company scope, contract synchronization, and the control-plane invariants. - The company-scope exceptions moved to the new `rules/company-scope-exceptions.mdc`. - Removed the strategic-docs rule. The plan-doc convention moved to the repo map. The artifact rule moved to section 7. - Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now cite `rules/data-paths.mdc`. - Regenerated the SDK public source pin, because one of those comments is in `packages/shared/src/crash-points.ts`. - Added both new rules to `.greptile/files.json`. - The API route rule applies only to routes that read or write company data. Its status code list is a list of common codes. - The company-scope rule lists the sidebar company order as an instance-wide exception. - The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for the Telemetry workflow. ## Verification - Open `AGENTS.md` and confirm that each link resolves. A local script checked this. - Run `grep -E '^(description|globs|alwaysApply):' rules/*.mdc` and confirm that each rule prints a description. - Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`. It passes locally. - `packages/shared/src/crash-points.test.ts` passed locally before the rebase onto `master`. - Five source files changed in comments only. No other test suite ran locally. ## Risks - An agent that reads only rule descriptions can miss a detail in a rule body. Reviewers still apply the full rules. - Pull requests and the definition of done are now sections 11 and 12. Sections 1 to 9 keep their numbers. - Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`. They are historical records. - Pull request #20565 also changes `AGENTS.md`. The second one to merge needs a conflict fix in section 1 and section 2. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 11:00:31 -06:00
Nicky LeachandPaperclip	2e3cbe57a4	fix(native): preserve parked Runners across controller restarts (#20586) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution parks a released sandbox Runner for a later turn. > - The warm idle timer detaches Core and leaves Runner retention to the host. > - Controller shutdown instead stops recently parked Runners through checkpoint cleanup. > - This makes reuse depend on whether the idle timer fired before shutdown. > - This pull request uses the same detach operation for idle expiry and shutdown. > - The benefit is consistent Runner retention without a new retention or recovery mechanism. ## Linked Issues or Issue Description Fixes: #20012. Related: #20005 unified cached and adopted process reuse. Closed #19975 included this shutdown change within a broader retention proposal. ## What Changed - Detach confirmed released entries during controller shutdown. Keep busy, unreleased, and close-marked entries on the existing cleanup path. - Share the detach operation with idle expiry and cached-owner handoff. - Record that a local handle has started detaching. Later cleanup cannot fall back to stopping its Runner after a partial detach failure. - Preserve explicit environment deletion, release proof, admission, and process-reuse checks. - Add Codex and Claude restart, detach-failure, and environment-deletion regressions. Add real Runner adoption cases after the restart sweep, including managed GitHub rebinding. - Document the lifetime rule and make the shutdown warning cover both detach and checkpoint failures. ## Verification - Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64, Node 26.12.0. - New controller-restart and detach-failure tests fail against unchanged master: 4 expected failures. Old shutdown calls close or reports success instead of reporting detach failure. - Focused executor tests: 22 passed. - Full native session executor suite: 1,328 passed. - `cd server && node scripts/typecheck-layers.mjs`: passed all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Manual diff review and added-line credential/PII scan: passed. - `cd server && node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/plugin-runner-release-resume.integration.test.ts --bail=1 --reporter=verbose`: 14 passed on this revision. This includes new Codex and Claude restart cases and managed GitHub rebinding. - Failure history: an earlier concurrent run had 4 passes and 10 failures. Its first failure was the existing Claude cached case exceeding 60 seconds. Later cases encountered a leftover prototype spy, recursive calls, and startup timeouts. No source changed before the passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`) passed all 11 original cases. The first timeout's exact cause remains unknown; the later pass does not establish that cause. - Standard Greptile review: 5/5 on `e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings. - `node scripts/local-verify.mjs --explain` returned exit 2 because the execution-semantics document requires broad CI. Required CI will supply broad typecheck, test, and build coverage. - A hosted deployment and an actual controller process replacement have not been run. The added integration cases exercise the shutdown sweep and a new adopting transport against a real Runner. ## Risks - SIGINT and SIGTERM preserve confirmed parked Runners, including ordinary service stops. The host still owns eviction and retention. - Detach failure retains a fenced local owner and reports failure. Core does not infer permission to stop the Runner from that failure. - Reuse remains conditional on current admission, process configuration, and host availability. - No persisted schema, checkpoint format, launch identity, or migration changes. State from the previous release uses the existing adoption or verified restore paths. ## Model Used - OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use, and local test execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 16:48:30 +00:00
DottaandClaude Opus 5.5	96ab5e3d38	fix(recovery): read a retained Runner from its own allocation and continue unverifiable no-replay holds from the last save (#20567) ... <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native warm runs on a remote sandbox continue a saved session in the next run. The next run first reads the retained Runner lifecycle from the sandbox. > - #20346 abandons the workspace epoch of a run that died during preparation. The next run then restores the last save into a new directory of the same sandbox. > - The retained-sandbox binding check required the old and the new directory to be the same. The warm follow-up failed with `native_controller_recovery_missing_evidence:retained_sandbox_binding` before provider work. > - The failed run's no-replay hold then refused its files with `allocation_ownership_ambiguous`. The task stayed blocked. The text said that Retry "succeeds only after Paperclip is updated", so no click could help. > - This pull request binds the Runner read to the directory that the retained lease recorded, and lets the existing retained-source fallback settle these holds from the last save. > - The benefit is that a moved workspace continues its session, and a hold that cannot be verified continues automatically instead of dead-ending. ## Linked Issues or Issue Description Refs #20346 **What happened?** On staging, a warm native continuation ran about 10 minutes after a deploy. The run before it died during preparation (database deadlock), so its workspace epoch was abandoned. The continuation restored the last save into a new allocation directory of the same Daytona sandbox. It then failed with `native_controller_recovery_missing_evidence:retained_sandbox_binding`, because the retained lease of the committed source run recorded the old directory. The failure recorded a no-replay hold (`native_continuation_requires_reconciliation`, resolved with outcome `blocked`). Workspace recovery refused the source with `allocation_ownership_ambiguous`, because the sandbox's reusable-lease chain had predecessors in another directory and a non-native preparation-only run. The task stayed blocked with "Retry runs the check again and succeeds only after Paperclip is updated." **Expected behavior** The continuation reads the retained Runner state from the directory where that Runner wrote it, and continues. When retained files cannot be verified, the task continues from the last committed save without a person. **Steps to reproduce** 1. Run a warm native session on a reusable remote sandbox and let it commit. 2. Fail the next run during preparation, so that its epoch is abandoned (#20346). 3. Start the next warm continuation. It restores the last save into a new allocation directory and fails with `retained_sandbox_binding`. 4. See the hold refuse its source with `allocation_ownership_ambiguous` and stay blocked. **Paperclip version or commit** Staging `c874cfa4dacc` (includes #20346). ## What Changed - `assertNativeRetainedSandboxBinding` now returns the directory that the retained lease recorded. It accepts a different target directory only when both are canonical sibling allocations of the same execution workspace under the sandbox's `.paperclip-workspaces` (`<workspaceId>` or `<workspaceId>-<epoch>-<leaseId>`). Same sandbox (`providerLeaseId`), environment, run, issue and committed owner are still required. Other directories stay refused. - `restoreRemoteSuspendedRunnerCheckpoint` reads the Runner state from that recorded directory. A moved source must be suspended and settled. A Runner left in the old allocation is never adopted live. - The retained-source fallback (`continueFromLastSavedWorkspace`) and its discovery now also cover the native no-replay hold (`native_continuation_requires_reconciliation`, active or resolved with outcome `blocked`, replay blocked for the same run). Rows saved by earlier releases are discovered without migration. - The fallback owns the Blocked status of such a hold only while the recorded `nativeFailureBlock` status version still matches the task. Otherwise it keeps the newer block and starts no continuation. - The refusal text no longer says "succeeds only after Paperclip is updated", and it promises no fallback. - `doc/execution-semantics.md` describes both rules. ## Verification - Red (unfixed code): `native-controller-recovery.integration.test.ts` "binds a retained Runner read to the source's directory after the workspace moved within the same sandbox" failed with `native_controller_recovery_missing_evidence:retained_sandbox_binding` at `native-controller-recovery.ts:145`, the staging error. - Red (unfixed code): `workspace-recovery-fallback.test.ts` "continues a resolved-but-blocked native no-replay hold from the last save, discovered by the worker" failed: the worker found no due work for the exact staging row. The newer-block safety case and the message case in `system-recovery-state.test.ts` also failed. - Provider boundary: a new test drives `restoreRemoteSuspendedRunnerCheckpoint` through a sandbox runner that refuses paths outside the sandbox workspace base. The suspended case reads `runner-state.json` from the source directory, never the moved root. The ready case is refused with `runner_not_settled` and caches nothing. - Green: `native-controller-recovery.integration.test.ts`, `workspace-recovery-fallback.test.ts` and `system-recovery-state.test.ts` pass (45 tests). The neighbouring suites `workspace-recovery-retry-copyback`, `workspace-recovery-preparation-gates` and `workspace-recovery-stop-proof` pass (135 tests). - Safety cases: another workspace, another parent, a non-canonical path or name (`-0-`, trailing text), `/unrelated` and another sandbox stay refused. A hold whose task moved to a newer Blocked decision keeps the block and starts no continuation. Exactly one continuation is delivered. Native replacement is denied for the abandoned source, so no second executor starts. - `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` returns exit 2 (a global doc input changed), so required CI supplies the broad coverage. ## Risks - Previous-release state: rows saved by earlier releases carry forward without migration. The binding reads existing lease metadata. The fallback discovers existing resolved-but-blocked holds by their existing evidence fields. - The binding now accepts a sibling allocation directory. The read still binds to the exact sandbox, run, environment and committed owner, and it reads only the source's recorded directory. A non-suspended Runner in a moved allocation is refused, not adopted. - The fallback abandons the unverified epoch and continues from the last committed save with a fresh session. In the staging shape the run did no provider work after it restored that save, so no work is lost. Unverified files are attached to the task when they can be read, as before. - Not verified live: a real warm Codex resume after a moved allocation on Daytona. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only browser access to staging). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 10:34:44 -06:00
DottaandClaude Opus 5.5	e5abcbf7a9	fix(native): release a restart claim that lapsed during a failed preparation (#20529) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, native restart recovery claims an unfinished run and prepares it again on the same heartbeat run. > - The restart claim holds a lease on the run's native coordinator row. For a remote claim with durable authority, the lease is 60 s. > - When that preparation fails, the failure path first preserves the remote workspace. Then it settles the run. > - When preparation outlived the claim lease, preservation took the run's own lapsed claim for another owner. It threw `NativeWorkspaceFinalizationBusyError`, and the execution returned without settling. The run stayed `running` with no event until a later scan reclaimed it (4.5 min in one wave, never in another). > - This pull request releases exactly that lapsed claim and asks restart recovery to decide on the run right away. It also logs the cause when the failure path defers. > - The benefit is that a failed restart-claim preparation is reclaimed or blocked visibly within seconds, instead of hanging. ## Linked Issues or Issue Description Refs #20514 (the same restart-chaos investigation). **What happened?** In restart-chaos wave `w-20261110T101517-s11936` (Daytona R1, K2, ST1), run `9d39550a` got a `bootstrap_incomplete` restart claim (generation 3). Its `heartbeat.prepare.run_output` step waited 141 s through an ST1 storage outage. The 60 s claim lease lapsed meanwhile, and the begin failed with `run_log_writer_authority_lost` when storage came back. The inner failure path then called `preserveNativeFailedWorkspace`. Its `ownsClaim` check required an unexpired lease, so the run's own claim counted as another owner and the call threw `NativeWorkspaceFinalizationBusyError`. `preserveNativeWorkspaceOnFailure` set `nativeOwnershipHeld` and the execution returned before `runCompletion.failRun`. No failure was recorded, and nothing was logged. The run stayed `running` until a later scan reclaimed it 4.5 minutes later. In a second wave no such scan came, and the run stayed `running` for 30 minutes. **Expected behavior** A failed preparation must settle or retry, or be claimed promptly by its next owner. It must not stay `running` with no record. **Steps to reproduce** 1. Claim a remote native run for restart recovery. 2. Keep its preparation busy past the 60 s claim lease, for example with a storage outage during `runOutput.begin`. 3. Let preparation fail. The run stays `running` with no failure event. **Paperclip version or commit** `master` at 23fa37f5bb. ## What Changed - `native-execution-lease.ts`: new `releaseLapsedNativeRestartClaim`. It is one conditional UPDATE that releases the claim only when the run is `running`, the coordinator is `observed` with no result, and it still names this exact `leaseOwner`, `controllerGeneration` and `attempt` with an expired lease. A successor claim, a result, or a live lease leaves the row unchanged. - `heartbeat.ts`: when failed-workspace preservation defers on a restart-claim execution, the failure path tries that release. If it succeeds, the execution schedules a restart-recovery pass for the run (`scheduleNativeSessionResumeDispatch(..., "restart")`) after it leaves `activeRunExecutions`. Recovery then reclaims the run, or blocks it with a named reason, right away. The path also logs the redacted cause, the claim kind and whether the claim lapsed. Before, this return was silent. - `preserveNativeFailedWorkspace` is unchanged. It still refuses an expired claim (`refuses a stale expired recovery claim`). - New `heartbeat-restart-claim-lapse.test.ts`. A restart-claim execution whose preparation fails after its claim lease lapsed, on a run with a prepared remote workspace, gets an immediate new restart-recovery decision. It failed before the change: the run stayed `running` with the lapsed claim and recovery answered `controller_still_alive`. Not in this PR: renewing the claim lease during preparation for `bootstrap_incomplete` claims (recovery.ts). Another restart-chaos fix owns that change. This PR makes sure a lapsed claim's failure is handed back at once either way. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-restart-claim-lapse.test.ts`: passed. Without the heartbeat change it times out. - `cd server && npx vitest run src/services/native-runtime/native-failed-workspace-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts src/services/native-runtime/native-execution-lease.test.ts`: passed (195 tests across the four files). - `cd server && pnpm typecheck`: no errors. - Live Daytona proof uses `--native-trigger` from #20538. Results follow in a comment. ## Risks - Low risk. The release matches only the exact lapsed claim of a running, result-less, `observed` run in one statement. A takeover between the check and the release changes the owner or generation, so the release does nothing. - The run is not settled by this execution. Restart recovery makes the decision under its own rules, which is the same path a later sweep would take. - Previous-release story: no persisted shape or identity changes. Rows saved by the previous release are read and released the same way. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 09:55:21 -06:00
DottaandClaude	c58356003f	fix(native): give run finalization its own retry budget (#20555) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save an accepted result. Then a finalizer applies that result to the issue. > - After a control-plane restart, recovery can resume the same run. Each resume uses one provider attempt (`coordinator.attempt`). > - The finalizer used the same `coordinator.attempt` value as its own retry budget. After three resumes, `attempt` was 3. So the first transient finalization failure made the coordinator `terminal_failure`. > - The result was then lost: the issue stayed `in_review` and no Retry was available. This breaks the rule that a restart must not leave a task stuck. > - This pull request gives finalization its own retry count. Provider attempts no longer use up finalization retries. > - The benefit: a resumed run that finishes its turn also finishes its task, and finalization stays bounded. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave `w-20261110T044338-s20385`, task t03 (local, Codex, three K1 SIGKILL restarts, `--db-pressure 4`), recovery resumed run `d5d0e839` three times (`providerAttempt: 3`). The resumed turn completed and the server logged `run.result.accepted`, and the run ended `succeeded`. The first finalization hit `NativeStatusRaceError` three times under row contention. `recordRetryableFailure` then compared `coordinator.attempt` (3) with its limit of 3 and wrote `terminal_failure` / `native_finalization_retry_exhausted`. The run's `execution.nextAction` is "Reassess against the latest authoritative issue status version.", which only the `status_cas_exhausted` branch writes. The second finalize call 2 s later got `native_finalization_terminal_failure` from `claimCoordinator`. The issue stayed `in_review` with `canRetry=false`. **Expected behavior** A transient finalization failure must stay retryable. The finalizer must apply the saved result without another provider turn. Finalization must still have a limit. **Steps to reproduce** 1. Resume a native run until its coordinator `attempt` is 3, for example with three restarts. 2. Let the resumed run save an accepted result. 3. Make the first finalization fail with a retryable error. 4. Before this fix, the coordinator goes directly to `terminal_failure`. The new test `server/src/__tests__/native-finalization-retry-budget.test.ts` reproduces this order. **Paperclip version or commit** The wave ran at e2b0290213. The bug is still on master at 902f376796. I found no later merged PR that changes this check. ## What Changed - `native-run-finalizer.ts`: `recordRetryableFailure` counts failed finalizations of the saved result in `heartbeat_runs.result_json.finalizationFailureCount`. It ends retries after 3 failures. It no longer reads `coordinator.attempt`, which counts provider attempts. The count goes on the run because each claim clears the coordinator's `failure_detail`. Workspace-scope failures keep their existing `workspaceFinalizeAttempt` budget. - New regression test `native-finalization-retry-budget.test.ts`: a coordinator saved at `attempt: 3` with no count stays `retryable_failure` after the first failure. It goes to `terminal_failure` only after the third failure. ## Verification - `cd server && npx vitest run src/__tests__/native-finalization-retry-budget.test.ts src/__tests__/native-finalization-recovery.test.ts`: 37 tests pass. - The new test fails on the unchanged finalizer (`terminal_failure` received, `retryable_failure` expected). It passes with the fix. - `pnpm verify` (selected tests). - I did not run a live chaos replay. The evidence is the wave's run events, its server log, and the code path. ## Risks - Low risk. The only behavior change: runs that used provider attempts now get their full 3 finalization retries. - Previous-release story: rows that an older server saved have no `finalizationFailureCount`. Their count starts at 0, so they get at most 3 more finalization tries, with no provider work. A coordinator that an older server already set to `terminal_failure` stays terminal. This PR migrates nothing. - If another writer replaces `result_json` without spreading it, the count can reset. That allows only a few more cheap retries, 30 s apart. It cannot cause an unbounded loop unless the overwrite repeats. - Not addressed here: when finalization really runs out of retries for a succeeded run, the issue still rests `in_review` with a board recovery card and no Retry. A safe Retry for that case must re-finalize the saved result, not start a new provider run. That needs a separate change. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, code editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 09:45:39 -06:00
DottaandClaude Opus 5.5	bd30ef8510	perf(chat): authorize the waits of an issue page with one read (#20551) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can work on tasks that start from an external chat, such as Slack. When the agent waits for the next chat message, the task shows "in review" with an external response wait. > - Issue lists and inbox attention show these waits. To show a wait, the server checks that the chat conversation is still authorized: the messages, the destination, the endpoint, and each person who wrote. > - The server did this check one time for each waiting task, and it started all the checks at the same time. A page with 19 waits sent 19 copies of the message-link statement at the same moment, each on its own pool connection, and then 19 copies of each policy read. > - On a hosted database, 19 identical copies of this statement were active at the same moment, about 670 ms each. They were 19 of the 20 active queries. > - This pull request reads the messages of all waits of a page with one statement and shares the policy reads between the waits. Each wait gets the same authorization result as before. > - The benefit is fewer statements and fewer connections for each issue page, so one page cannot fill the pool with copies of one read. ## Linked Issues or Issue Description Refs #20400 (shares wait reads between the callers of one request; this change removes the per-wait fan-out inside that shared read). Refs #18966 (reuses the prepared message-link statement; this change keeps that statement for single runs). **What happened** `listExternalReviewWaits` ran `authorizeChatConversationForBoundRun` one time for each waiting task, inside `Promise.all`. Each call sends the message-link statement, then a resource read, and three reads for each person. A page with N waits sent N copies of each statement at the same time, on up to N pool connections. **Expected behavior** One issue page sends one message-link statement for all of its waits. A resource or a person that is common to many waits is read one time. The authorization result of each wait does not change. **Steps to reproduce** 1. Put many tasks of one company in review with an external chat response wait. 2. Open the issue list or the inbox. 3. Look at the active statements of the database. One copy of the message-link statement runs for each wait. ## What Changed - `server/src/services/native-runtime/chat-attachment-reuse.ts`: - Split `authorizeChatConversationForBoundRun` into steps: read the bound provider and wake comments, read the message rows, and apply the policy to the rows (`authorizeBoundRunLinks`). The single-run path uses the same prepared statement, the same lock modes and the same policy code as before. - Move the message-link join and its policy filters to one query builder, `boundRunLinksQuery`. The single-run statement and the batch statement both use it. - Add `authorizeChatConversationsForBoundRuns(db, companyId, runs)`. It is read-only and takes no locks. It reads the messages of all runs with one statement, then gives each run only the rows that the single-run statement returns for that run (same issue, agent, provider and comments). The runs share each distinct resource and principal read. Each run gets its conversation, or `null` for a policy denial. Other errors are thrown, as before. - Add `isBoundChatAuthorizationDenial` for the three policy denial errors. - `server/src/services/native-runtime/external-review-wait.ts`: `listExternalReviewWaits` uses the batch authorization. The inherited GitHub review-wait reads do not change. - `server/src/__tests__/batched-chat-authorization.integration.test.ts`: new tests on real PostgreSQL. For each case, the batch result must equal the result of separate single-run reads. The cases include each denial: a missing or foreign comment, no comments, no execution marker, a wrong provider, agent or issue, a person without a link or without a membership, a viewer member, a revoked link, a disabled or unavailable destination, a paused endpoint, a closed conversation, an unprocessed delivery, a delivery without a person, and a verifying endpoint. A test also makes sure that one person's decision never applies to another person in the same batch, and that a batch sends one message-link statement. - `scripts/benchmark-chat-bound-authorization.ts`: a local benchmark that loads the pre-fix modules from a baseline commit, checks that both versions return the same waits, and measures them. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/batched-chat-authorization.integration.test.ts src/__tests__/prepared-chat-authorization.integration.test.ts` passes (10 tests). - Mutation check: when I removed the issue filter from the per-run row selection, or shared one principal decision for all people of an endpoint, the new tests failed (1 and 3 failures). - Related suites pass: `heartbeat-reviewed-chat-binding`, `external-chat-wait`, `preserved-native-response-wait`, `chat-attachment-reuse` and `external-review-wait` (392 tests). - `pnpm verify` asks for broader verification because the change adds a file under `scripts/`. Required CI supplies that coverage. - Benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-bound-authorization.ts --baseline-ref=f4337bb5fb` (the measured pre-fix commit) Benchmark fixture (synthetic, local PostgreSQL 18.1, pool of 10): one company, one Slack endpoint, 20 tasks waiting for a chat response from 5 people, 4 of those waits denied (the person is a suspended member), and 2,000 earlier conversations with 50,000 processed deliveries (6 KiB events, 419 MB). Both versions return the same 16 waits. 15 alternating pairs after warm-up: | One issue page (20 waits) | Before | After | | --- | --- | --- | | Statements sent | 101 | 18 | | Message-link statements | 20 | 1 | | Server time of the message-link statements | 0.50 ms (sum of 20) | 0.13 ms | | Page time, median (min to max) | 8.86 ms (6.94 to 12.05) | 4.05 ms (3.62 to 5.54) | | 8 pages at the same time, median (min to max) | 54.18 ms (48.36 to 89.24) | 14.99 ms (12.60 to 19.59) | ## Risks - The batch must give each wait the same result as a separate read. The batch statement uses the same join and policy filters. It then selects each run's rows with the equality filters of the single-run statement. The policy code is the same function for both paths. The tests compare both paths for each denial case. - Shared reads: the batch reads each resource and each person one time. All reads are plain reads with no locks, as before. A policy change that occurs during one page read can now apply to all waits of the page at the same time, instead of to some of them. - An invalid wake comment id in one wait makes the whole page read fail. This was already true: the failure of one wait rejected the whole `Promise.all`. - The single-run path (attachment reuse, status decisions, run preparation and others) keeps its prepared statement and lock modes. The order of the conditions in its WHERE clause changed; the conditions are the same. - Not proven: why each copy took about 670 ms on the hosted database. Locally one copy takes about 0.03 ms with the expected index plan. The likely causes are the burst of copies on a small compute and cold pages. Verify after deployment that the statement runs at most one time for each issue page request. - No migration and no schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 09:25:23 -06:00
DottaandClaude Opus 5.5	db0d523bc8	fix(recovery): start a Retry after a restart cut off a relaunch whose earlier Runners were proven stopped (#20537) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run fails, a person presses Retry and expects a new run to start > - Before a native Retry starts, Paperclip must prove that the previous run's Runner and provider stopped, so two providers never work on the same task > - When the server is killed after it asked for a Runner launch but before it recorded the Runner's process, there is no PID to check. Paperclip then proves the stop from two facts: no Runner process for the run exists on this host, and the run never saved Runner state > - A run that already crossed restarts did save Runner state, from its earlier Runners. Those Runners were proven stopped, with stop receipts. The proof still refused, because it only asked whether any state belonged to the run > - The Retry returned 202, but the task then waited for a board operator to confirm the stop (`confirm_run_stopped`, `process_identity_missing`), until the task timed out > - This pull request lets the proof accept Runner state that only the earlier, proven-stopped Runners could have written > - The benefit is that a Retry after a restart-interrupted relaunch starts right away, with no operator step ## Linked Issues or Issue Description **Subsystem affected** Server native-run stop proof for a Retry or a new message (`server/src/services/native-local-process-stop.ts`, `unidentifiedLocalRunnerProvenAbsent` in `server/src/services/native-runtime/native-session-executor.ts`). **Problem or motivation** Restart-chaos wave `w-20261110T111245-s701`, task t03 (local, codex, C1, K1/K2, `--storage s3 --storage-faults ST7`): run `f1ffeed3` crossed three restarts. Its event history is: 1. Launch request, process identity, stop receipt (first Runner, killed by K1) 2. Launch request, process identity, stop receipt (second Runner, killed by K1) 3. Launch request at 11:13:54.999, then a K1 kill before the Runner's identity was recorded 4. A fourth controller resumed the run, a K2 drain interrupted its preparation, and the run failed (`adapter_failed`, fixed separately by #20514) The person's Retry at 11:14:37 returned 202. Admission reached `recordUnidentifiedLocalRunnerStop`. No Runner process for the run existed, but the Runner state root held `control-plane-state.json` with this run's id, written by Runners 1 and 2. The proof refused, the issue was saved as `waiting_for_user` / `confirm_run_stopped` / `process_identity_missing`, and the wake stayed deferred until the task timed out about 15 minutes later. The harness graded `R3:retry_no_new_run`. **Proposed solution** A Runner persists its command journal (`runner/runner-state.json`) before any command takes effect, and a provider starts only from a command. Its `runner/process-lineage/` writer marker is written before any supervised process starts. So a launch whose Runner never wrote either file after the launch request cannot own a provider. - The launch request event now records `requestedAt`, taken on the launching host just before any backend may spawn. - `recordUnidentifiedLocalRunnerStop` passes that time on only when the latest process event before the request is a stop receipt (every earlier Runner of the run was proven stopped) or there is none. - `unidentifiedLocalRunnerProvenAbsent` then accepts saved state of this run only when `runner/runner-state.json` (if present) and every `runner/process-lineage/` entry were last modified before `requestedAt`. A file time that falls on a whole second (a coarse-timestamp filesystem) must predate the request by 2 s. The existing checks stay: no `--run-id` Runner process on this host, the launching host matches, and no remote lease. **Previous-release story** A launch request saved by the previous release has no `requestedAt`. It keeps today's proof: it is accepted only without saved Runner state of the run, otherwise the board shows the operator confirmation. Nothing is migrated or rewritten. **Alternatives considered** - Treating a proven-gone controller as proof that its Runner stopped. Local Runners outlive a controller restart (graceful drains reattach them), so a gone controller says nothing about its Runner or provider. - Running `paperclip-runnerd reap-lineage` for each writer marker. It needs a Runner PID and binary on the hot Retry path, and a Runner whose journal write failed has no marker to find. ## What Changed - `native-session-executor.ts`: the `native.process_start_requested` payload adds `requestedAt`. `unidentifiedLocalRunnerProvenAbsent` takes `earlierRunnersStoppedBefore` and accepts this run's saved state only when the Runner journal and lineage entries predate it (`runnerStateUnchangedSince`). - `native-local-process-stop.ts`: `recordUnidentifiedLocalRunnerStop` reads the last two process events and passes the request time only after a stop receipt (or no earlier launch). - `explicit-native-continuation.test.ts`: new embedded-Postgres case. A native run with a proven-stopped Runner, saved Runner state, and a later unidentified relaunch: the Retry is admitted and the stop is recorded. It stays blocked (`process_identity_missing`) when the Runner journal or a lineage entry changed after the request, when the request has no `requestedAt` (previous release), when the earlier Runner has no stop receipt, when a whole-second file time is within 2 s of the request, or when a `--run-id` Runner process is alive. A whole-second time 3.5 s before the request is admitted. ## Verification - Red/green: `cd server && npx vitest run src/services/explicit-native-continuation.test.ts -t "proven stopped"`. On master the `proven` case fails (admission returns null); with the fix all 8 cases pass. - Related suites: `explicit-native-continuation.test.ts`, `issue-confirm-run-stopped-routes.test.ts`, and the unidentified-Runner and stop cases of `native-session-executor.test.ts`. - Live, the s701 seed with the original wave's flags (`--env local --adapters codex,claude --workloads A1,C1 --faults K1,K2 --triggers time,result --concurrency 4 --tasks 4 --seed 701 --storage s3 --storage-faults ST7 --keep`): - `w-20261110T131243-s701` (final head) and `w-20261110T125259-s701`, master plus this fix plus #20514 merged locally: 4/4 pass each, every task crossed a restart, no R3 signature, 0 speed breaches. - `w-20261110T125616-s701`, master plus this fix without #20514: t01 and t02 pass. t03 and t04 hit the drain-during-preparation failure that #20514 fixes (`R3:retry_needed`). The Retry started a new run each time and both tasks finished (`retry_no_new_run` is gone). New launch requests carry `requestedAt`. - The restart timings of these replays did not land in the exact gap between a launch request and its process identity. The red/green test covers that window. ## Risks - Low. The change only widens one fail-closed proof, and only for launch requests saved by this release, after a stop receipt, with Runner files older than the request. The proof compares file times with a time taken on the same host. A backwards wall-clock step between the request and a Runner's first write could hide that write. The `--run-id` process check still has to pass, so that Runner must also be gone. - The Retry path adds one `lstat` per file in a small directory. It adds no wait and no work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 09:08:23 -06:00
3 people	faae6b919f	fix(runner): explain task scope on missing tool inputs (#19774) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read saved task documents through native tools. > - Active-task tools use the task bound to the run. Other task tools require a task UUID. > - Agents sometimes call the second set with only a document key. > - The validator rejects the call, but gives no instruction about the active-task tool. > - This pull request adds scope instructions to the tool schemas and returns them for missing required fields. > - Agents can correct a rejected call. Task selection and permission checks stay in place. ## Linked Issues or Issue Description **What happened?** Two observed document reads sent only `{"key":"sentry-case"}` to `read_task_document`. Each returned `Invalid tool input: / must have required property 'taskId'`. Each later retry with a task UUID returned without an error. Pattern: [Task reads fail when agents omit taskId](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/703d8c29-6a40-4aa2-b2c6-e2c5c36b5657). Related PR: #18824 changed document key validation. It does not address task scope. **Expected behavior** The tool contract must state which task it reads. A missing-field error must include the schema owner's recovery instructions. A retry must still pass validation and task permission checks. **Steps to reproduce** 1. Call `read_task_document` through the MCP bridge with only a document key. 2. Inspect the error sent to the caller. 3. Retry with an authorized task UUID. Confirm the returned document belongs to that task. ### Hypothesis and thesis for a fix Hypothesis: generic tool descriptions contribute to confusion between active-task and explicit-task tools. An agent can also ignore a correct schema. These tests do not measure the rate of model mistakes. Thesis: Paperclip owns task scope instructions. Its five explicit-task tools now describe the required UUID and the active-task alternative. The two MCP transports return direct missing-property descriptions from the admitted schema. The transports do not infer task IDs or apply Paperclip task policy. Proof: the same HTTP MCP regression fails on baseline `f673bd3cb5b5b65ec51781e37355e168c3fe8ae4` and passes with the fix. It still rejects before handler dispatch. An explicit retry reaches the requested task. An isolated real API and PostgreSQL test checks the saved revision, active-task selection, malformed UUIDs, and cross-company denial through both transports. ## What Changed - Add UUID and active-task guidance to `get_task`, `comment_on_task`, and the three explicit-task document tools. - Return bounded, distinct descriptions for up to three direct missing required fields. Keep ordinary validation errors unchanged. - Share this formatter between the Runner and OpenCode MCP bridges. - Add transport regressions, real route and database checks, and tool documentation. ## Verification - Baseline red: both transport regressions return only the required-property error and fail the recovery instruction assertion. - Candidate green: 58 Runner and OpenCode transport and error formatter tests pass on Linux with Node 24.22.2. - Final-head server verification passed all 115 tests: 113 document-key checks and two real API integration cases. - The two local integration cases use real MCP HTTP, agent authority, API routes, and disposable PostgreSQL. They read a selected other task and its saved revision, read the active task separately, reject missing IDs for all five tools, reject malformed UUIDs, and return 404 for a foreign company task. - Runner TypeScript typecheck and build pass. - The full server typecheck passed on head `03a37e9e50896416cfd45be33bf6dbe3e13169df` with `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 node scripts/typecheck-layers.mjs`. The first run ended without a compiler diagnostic. Final-head CI passes and Greptile is 5/5. - Full preflight stops because cargo is absent. Rust checks were not run locally. - One existing delayed-body fetch test failed with a Node 25 fetch runtime error. All transport tests pass on Node 24. The first integration attempt used the wrong fixture return shape; the test was corrected before the passing run. - CI follow-up: the Runner test program reported four TS18046 errors because `Response.json()` returns `unknown` under CI typings. Both tests now declare the response type. `pnpm exec tsc -p tsconfig.test.json --noEmit` passes, and both transport suites pass again (56 tests). Final head `ffccebca19fe9e21a097829adc020679ffa7d86c` passes CI and Greptile 5/5. - Review follow-up: the foreign task now has a saved document with a distinct body. Both integration cases passed again after this test-only change. The read must return 404 and must not return that body. Current-head CI passes and Greptile is 5/5. - Final CI: https://github.com/paperclipai/paperclip/actions/runs/42352822739 . All required jobs passed for `ffccebca19fe9e21a097829adc020679ffa7d86c`. Greptile is 5/5 with no unresolved review thread. - Real local API behavior is verified. External model behavior and deployed recovery are not verified. No merge or deployment is included. ## Risks - Missing-field errors now contain schema descriptions. The text is limited to direct properties in the admitted schema, at most three distinct descriptions and 1,000 characters each. - Nested schemas and references do not get recovery text. Their validation errors remain intact. - Clearer instructions can help callers recover. They do not guarantee that a model will choose the correct tool. ## Model Used - OpenAI GPT-6 via Codex, with tool use and code execution. The exact deployment model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 09:07:11 -06:00
DottaandClaude Opus 5.5	902f376796	fix(recovery): keep a remote restart claim's lease through slow preparation (#20540) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, a remote run resumes on the same heartbeat run under a restart claim: a lease on its native coordinator row, 60 s for a remote claim (`REMOTE_CONTROLLER_LEASE_MS`) > - Preparation then runs before the native executor takes its own lease renewal. It includes the run-log begin and the workspace restore, which can each wait out a storage outage for minutes (#20299) > - Preparation renewed the claim only for `reattach_remote_runner`. A `bootstrap_incomplete` or resume claim kept its 60 s lease with nothing renewing it > - So a preparation wait longer than a minute let the lease lapse. The run failed at its next writer-authority check (`run_log_writer_authority_lost`) the moment storage came back > - This PR renews every remote restart claim during preparation. When the same owner's executor claims at the next attempt, it takes over that renewal > - The benefit is that a resumed remote run survives a long storage outage during preparation instead of failing as soon as storage returns ## Linked Issues or Issue Description **What happened?** In restart-chaos waves on Daytona R1 with `--storage s3 --storage-faults ST1` and K2 at +110 s, a resumed `bootstrap_incomplete` run's `heartbeat.prepare.run_output` ended `failed` right after the outage ended: - `w-20261110T101517-s11936` t01, run `9d39550a`. The claim was at 10:19:55 and the outage ended at 10:22:18.15. `run_output` failed at 10:22:18.17, after 141 s. - `w-20261110T081647-s11936` t02, run `9da10424`. The claim was at 08:21:29 and the outage ended at 08:23:30. During the outage the journal claim fails on its head `GET` before it checks authority. The first attempt after the outage reaches `claimFrom`'s `assertAuthority` (`run-log-authority.ts`), which requires a live coordinator lease. That lease expired 60 s after the claim. **Why not just widen the condition?** For a `bootstrap_incomplete` claim, the executor's own claim moves the coordinator attempt to N+1 (`nextNativeProviderAttempt`). `startNativeSessionExecutionLeaseRenewal` shares one renewal per lease owner and threw `native_session_renewal_binding_changed` when the attempt differed. That is presumably why preparation renewal was limited to reattach claims. **Related:** #20460 (live status while a run waits out a storage outage). #20514 settles a failure that this lapse used to leave silent. ## What Changed - `heartbeat/recovery.ts`: start the preparation lease renewal for every remote restart claim (`scope.remote`), not only `reattach_remote_runner`. - `native-runtime/native-execution-lease.ts`: - When the same owner's executor starts its renewal at a higher attempt, it takes over the shared renewal, rebinding the attempt and `onLost`. A lower attempt or any other binding change still throws. - A renewal that failed under the old binding is not counted as a loss. - A failed renewal is not a loss when the coordinator shows the same owner already at a higher attempt (the executor's claim committed while the renewal was in flight). The renewal deadline still ends ownership if no executor takes over. - Tests: one recovery case and three lease-renewal cases (below). ## Verification - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/heartbeat/recovery.test.ts src/services/native-runtime/native-execution-lease-reacquire.test.ts src/services/native-runtime/native-runner-restart-lease-renewal.integration.test.ts src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`: 143 pass. `npx tsc --noEmit`: pass. - Red/green, recovery test "keeps a remote bootstrap restart claim through preparation longer than its first lease, then hands it to the next attempt": the claim's first lease ends 4 s in. Preparation runs past it and the run-log writer authority still passes. Then the executor claims attempt 2 and its renewal keeps extending the lease with no detach. - Red with master's `recovery.ts`: the lease is never extended. - Red with only the `recovery.ts` change: `native_session_renewal_binding_changed`. - Green with both. - Red/green, lease unit tests: handover to the next attempt (and refusal of a lower one); a failure under the old binding is not a loss; a failure while the coordinator shows the same owner at a higher attempt is not a loss. All three fail on master. - Live wave: results in a comment below. ## Risks - Medium-low. - The preparation renewal now runs for remote `bootstrap_incomplete` and `resume_dead_runner` claims too. It renews the same coordinator row under the same CAS conditions reattach claims already use. - If preparation truly loses the lease (another owner takes it), its `onLost` still runs (`detachNativeSessionsForRestart`) and authority checks still fence writes. - The new non-loss case applies only when the coordinator names this exact owner at a higher attempt. The renewal deadline still bounds it. - Previous-release state: this persists nothing new. A claim saved by the previous release resumes unchanged; its preparation is simply renewed now. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 08:07:16 -06:00
3 people	f4337bb5fb	test(evals): land the chat catalog eval harness on master (#19614) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 08:02:11 -06:00
DottaandClaude Opus 5.5	60b9147e7c	fix(native): accept harness diagnostics in a remote session-opened bootstrap proof (#20543) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) keep their controller state in a Postgres checkpoint. The controller writes every command to that checkpoint before it sends the command. > - After a restart, recovery uses that record to prove that a drained remote bootstrap reached no provider turn. If so, it bootstraps the run again on the same heartbeat run. > - A graceful restart seconds after native start often catches a runner that opened its session and emitted ACPX `harness.diagnostic` events, but was never sent a turn. > - The remote no-turn proof did not accept `harness.diagnostic`, so recovery blocked these runs as `ambiguous_remote_recovery_evidence` and the task needed a manual Retry. > - This pull request accepts `harness.diagnostic` in that proof. The proof's command check still requires that no turn command was ever issued. > - The benefit is that a restart just after native start re-bootstraps the run automatically. ## Linked Issues or Issue Description Refs #20538 (the harness trigger that reproduced it) **What happened?** Restart-chaos wave `w-20261110T091639-s11936` t02, run `c4fc7f03` (Daytona R1, K2): a K2 landed 13 s after native start. Recovery returned `ambiguous_remote_recovery_evidence`, and the run blocked with `native_restart_recovery_blocked` (R3 retry_needed). I reproduced it with a kept database in `w-20261110T132656-s11936` t02, run `ecd98d55`, using `--native-trigger native-start+5` from #20538. Decoded controller checkpoint: - commands: `run.prepare`, `session.open`, `session.goal.get` (no turn command); - committed events: `session.started`, `session.capabilities.updated`, `session.goal.snapshot`, `harness.diagnostic`, `harness.diagnostic`, `session.goal.snapshot`; - `ackedSourceSeq` 6, no `sessionCheckpoint`, no projected provider events. `nativeRunnerCheckpointRemoteBootstrapProof` returned `null` only because `harness.diagnostic` was not in `SESSION_OPEN_EVENT_TYPES`. **Expected behavior** Recovery classifies the run `bootstrap_incomplete` (`remote_session_opened_without_turn`) and bootstraps it again. **Steps to reproduce** 1. Run a Daytona R1 codex wave with `--faults K2 --native-trigger native-start+5`. 2. The suspended run is blocked with `ambiguous_remote_recovery_evidence`. **Paperclip version or commit** `master` at c874cfa4da. ## What Changed - `native-bootstrap-proof.ts`: add `harness.diagnostic` to `SESSION_OPEN_EVENT_TYPES`, which `runnerdStateHasNoProviderTurn` uses. The proof still requires every command to be `run.prepare`, `session.open` or runner maintenance (`session.goal.get` and similar), every committed event to be accounted for in order, and no adopted state or warm transition. A turn command or any turn event still fails it. The local opened-session proof already accepted `harness.diagnostic` through `PRE_TURN_EVENT_TYPES`. - `native-runner-restart-recovery.integration.test.ts`: a case built from the observed record (goal read plus two diagnostics). It is classified `bootstrap_incomplete` with `remote_session_opened_without_turn`, and the executor's restored-checkpoint proof agrees. A diagnostic followed by `turn.started` stays blocked. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 163 passed, 14 skipped. The new diagnostics case failed before the change with `{ kind: "blocked" }`. - Live: a Daytona R1 wave with `--native-trigger native-start+5` follows. Results will be in a comment. ## Risks - Low risk. `harness.diagnostic` is a harness report, not provider input. Provider work needs a turn command, which the controller writes to this record before sending, and the proof still rejects any turn command or turn event. - Previous-release story (upgrade-safe run state): no persisted shape changes. A controller checkpoint saved by the previous release with diagnostics and no turn is now re-bootstrapped instead of blocked. A record with a turn keeps the same named refusal. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 08:00:45 -06:00
DottaandClaude Opus 5.5	c9226de30b	fix(heartbeat): leave a preparing run to the next controller after a graceful drain (#20514) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat service runs each agent task. A graceful shutdown (SIGTERM) drains running native runs and hands them to the next server. > - The drain detaches the native runner session and releases the run's lease with the recovery state `awaiting_runner_reattach`. > - An in-process execution can still be in run preparation at that moment, for example while the run-log journal claim waits on a slow or unavailable object store. > - When that execution then lost writer authority (`run_log_writer_authority_lost`), the generic failure path settled the run as `adapter_failed`. The next server then saw `native_continuation_requires_reconciliation`, and replay stayed blocked. > - This pull request makes a preparing execution that a restart detached stop quietly, the same way a detaching native session already does. > - The benefit is that a slow preparation step plus a graceful restart no longer turns into a failed run that needs a manual retry. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave `w-20261110T092411-s701` (local, A1/C1, K1/K2, ST7 read-after-write lag), three of four tasks failed with R3 `retry_needed`. For run `218a0425`, the K2 SIGTERM drain wrote `native.recovery.transition` with `awaiting_runner_reattach` and released the lease. About 3 s later, the old server's in-process execution, which was still in `runOutput.begin`, got `run_log_writer_authority_lost`. `runCompletion.failRun` then settled the run `adapter_failed` with a durable settlement. The next server found `native_continuation_requires_reconciliation` and did not replay the run. **Expected behavior** The drain hands the run to the next server. The old execution must not settle it. A real authority loss outside a drain must still fail the run. **Steps to reproduce** 1. Start a native run recovery, and hold it in preparation (slow store, ST1 outage wait, or ST7 lag). 2. Send SIGTERM so `drainRunningRunsForShutdown` suspends the run. 3. Let preparation continue and check writer authority. The run settles `adapter_failed`. **Paperclip version or commit** `master` at 914e3ae14f. ## What Changed - `native-session-executor.ts`: add `watchNativeRestartDetach(runId)`. It records the run's restart-detach marker when the execution starts. It reports `true` only when a later restart detach for that run happens. A marker left by an earlier detach does not count. - `heartbeat.ts`: the execution creates the watcher at start. In the inner failure path (before `runCompletion.failRun`) and the setup failure path (before `runCompletion.failRunSetup`), if a restart detached the run and no provider dispatch started, the execution logs the handoff and stops without settling. It uses the same teardown flag (`nativeSessionResumeScheduled`) as `NativeControllerDetachedForRestartError`. - `heartbeat-native-restart-suspension.test.ts`: add a red-before/green-after regression. A recovery execution is drained during preparation and then checks the real run-log writer authority. With the drain, the run stays `running` with `awaiting_runner_reattach`, and the next controller resumes it. Without the drain (the lease only expired), the run still fails `adapter_failed`. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 98 passed. Before the fix, the new drained case failed with `status: failed, errorCode: adapter_failed`. - `cd server && pnpm typecheck`: passed. - Live restart-chaos wave: see the comment below. ## Risks - Low risk. The new branch applies only when this process's own restart detach for this run happened after the execution started, and before any provider dispatch. Typed native outcomes (startup cleanup, ownership unverified, cancellation pending, resume scheduled) keep their earlier handling because they are checked first. - Previous-release story (upgrade-safe run state): no persisted shape, launch identity or pack identity changes. The marker is process-local. State saved by the previous release carries forward unchanged. The deploy-lands-mid-run case is the regression test itself: a drain lands during recovery preparation and the next controller resumes the same run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 08:00:09 -06:00
DottaandClaude	bf45700ca3	fix(native): make Retry work after a turn start cut off once Codex recorded the turn (#20516) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run sends `turn/start` through its runner. A control-plane restart can kill the runner and Codex while that request is in flight > - #20507 lets the runner resume by itself when Codex's thread history proves the turn was never created. When Codex did record the turn, the turn may have done work, so the runner still fails closed and nothing replays it > - That branch must still end in a Retry that works. The next run on the same conversation is refused at the runner-state fence before it asks for a runner. #20507 let Retry continue after that run by matching the refusal's error text > - Driving that branch end to end with the real runner exposed a second dead end. When the owner's runner dies, its provider-session close cannot prove suspension, so this server process can keep an operator-required cleanup quarantine for the task. Retry was then refused outright ("Send a new message") > - This pull request replaces the text match with a server-authored run-log event, lets Retry take the same verified-stop path as a new message, and adds a real-runner test of the whole branch: the cut-off owner fails closed, the follow-up is refused with structured proof, Retry is admitted, and the fresh Retry run completes > - The benefit is that this restart window cannot strand a task, and the Retry gate no longer depends on error wording ## Linked Issues or Issue Description **What happened?** Follow-up to #20507 (restart-chaos wave `w-20261110T033426-s108`, t03). #20507 left one item open: prove that a person's Retry produces a working run when the restart cut off `turn.start` after Codex recorded the turn. It also matched the fence refusal by its error text. **Expected behavior** After that cut-off, the owner stays failed closed and the turn is never started again by itself. A person's Retry starts a new run that completes. It never fails with `quarantine_owner_unverified` and never returns 202 with no run. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts`. 2. Without the new event, the test fails: the refused follow-up has no proof that it never launched, so Retry after it would wait on `process_identity_missing`. ## What Changed - `native-session-executor.ts`: when the runner-state fence refuses a run before launch (`prior_owner_*` or `legacy_prior_owner_*`), the executor records the server-authored event `native.prior_owner_fence_refused` with the closed refusal `reason`, then fails the run as before. A failed write leaves the run held (the safe side). - `native-local-process-stop.ts`: `nativeRunNeverLaunched` accepts a failed run that has this event (and still no launch, identity, `adapter.invoke` or runner-sourced event). It no longer reads the error text. - `explicit-native-continuation.ts`: when the task's in-process cleanup quarantine is operator-required, a Retry no longer returns `cleanup_quarantined` at once. Like a new message, it must first pass `verifyStoppedNativeSessionForContinuation`, which releases the quarantine and records `native.stopped_conversation_verified`. A native Retry always starts a fresh conversation (`forceFreshSession`), so it never resumes the interrupted turn. If the proof fails, Retry still returns `cleanup_quarantined`. - `verifyStoppedNativeSessionForContinuation` (`native-session-executor.ts`), three narrow additions: - a run with `native.prior_owner_fence_refused` may stand in for the saved state's owner, like a run refused by the cleanup quarantine (both were refused before launch); - when the session root is gone, it reads the fence's single `identity_indeterminate` quarantined copy of the owner's state (several copies stay ambiguous); - a provider startup attempt in phase `initialization_failed` whose direct child exited and whose process and process group are gone no longer blocks the proof. A same-run restore that fails closed leaves exactly this record. Those PIDs are also re-checked before the quarantine is released. - New real-runnerd integration test `native-cut-turn-start-retry.integration.test.ts` (real runnerd and fake Codex with `--hang-turn-start after`): 1. The owner run's `turn/start` hangs after Codex recorded the turn; the runner and Codex are killed. The owner's provider state keeps the ambiguous start, and the thread history keeps the turn. 2. The same-run resume restores the runner and fails closed with "cannot safely recover an ambiguous Codex turn start". Codex saw exactly one `turn/start`. 3. A follow-up on the same conversation is refused with `prior_owner_*`, never `quarantine_owner_unverified`. It records `native.prior_owner_fence_refused`, launches nothing, and `nativeRunNeverLaunched` is true for it. 4. The owner's operator-required in-process cleanup quarantine is pinned (whether the real close lands there is timing-dependent; about half of the local runs did). 5. Retry of the refused follow-up passes `admitExplicitNativeContinuation` (not a dry run) with no block reason, releases the quarantine and records `native.stopped_conversation_verified`. 6. The Retry run, prepared with a fresh conversation as task Retry does (`forceFreshSession`), uses a different runner-state root and completes with exit code 0. Codex saw one new `thread/start` and one new `turn/start`; the cut-off turn was never started again. - `explicit-native-continuation.test.ts`: the fence rows now insert the event. A new row shows that the refusal text alone keeps the run held. The "worker loss (retry)" case now expects a verified Retry to be admitted like a new message. - `native-recovered-owner-followup.test.ts`: a fixture event moves to a free sequence number, because the refused follow-up now writes its own event. - Docs: `doc/run-log-events.md` (new event) and `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Red/green: - Without the event write, the integration test fails (no fence event, so the follow-up is not proven unlaunched). - Without the gate and proof changes, it fails with `expected [ 'cleanup_quarantined' ] to deeply equal []`: Retry is refused while the owner's quarantine is held. - With all changes, it passes. - `vitest run` of `native-recovered-owner-followup`, `explicit-native-continuation` (427 tests), `quarantined-task-retry`, `issue-confirm-run-stopped-routes`, `native-finalization-recovery`, `task-interruption-query`, `execution-projection`, `chat-run-publications`, `closed-native-receipt-accounting`, `agent-live-run-routes` and the new integration test pass. - Server `tsc --noEmit` passes. - Runner code and the pack `contractDigest` do not change. ## Risks - Low-medium risk. The new event is written only on the fence refusal path, which already fails the run. It is local run-log data (no Telemetry or OpenTelemetry). - Retry after an operator-required cleanup quarantine is a behaviour change: it was always refused, and now it is admitted only after the same exact stop proof a new message needs. That proof checks the owner's runner stop record, every recorded provider and startup PID and process group, and the unchanged state fingerprint before it releases the quarantine. - A run refused by a build between #20507 and this change has only the error text. It stays held on `process_identity_missing` until a person sends a new message or the board confirms the stop. #20507 has not shipped in a release yet. - The test models the restart in-process: it kills the runner and Codex and calls the executor again. It does not drive the HTTP Retry route; the route's own behaviour (fresh conversation, admission gate) is covered by existing tests and by `nativeRunNeverLaunched` here. - Open decision, unchanged: a turn Codex recorded before the restart is never replayed or adopted. Continuing is a person's decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20507 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 06:16:42 -06:00
DottaandClaude Opus 5.5	c0593b69f1	fix(accounting): close a remote-Runner turn completed across a session restore as a lower bound (#20524) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs settle their charge once the final usage receipt is complete. > - #20517 fixed one case for the local Runner. If a restart stops the provider before a turn's final usage report, and the restored Runner then reports that turn completed, the turn now closes as a lower bound instead of waiting 10 minutes. > - Remote Runners deliver usage through a separate producer-owned receiver. That receiver dropped `session.resumed`, so the same turn there still waited for the 10-minute closure timeout. > - This pull request applies the #20517 rule to that receiver. > - The benefit is that remote ACPX and OpenCode runs settle promptly with an honest lower-bound label after a mid-turn restore. ## Linked Issues or Issue Description Refs #20517. **What happened?** After #20517, `session.resumed` reached turn accounting only through the local Runner's fallback recorder. The remote producer receiver (`native-usage-delivery.ts`) and the session-authority filter (`isNativeUsageAccountingEvent`) both dropped it. A remote ACPX or OpenCode turn that was open across a session restore, and then reported terminal without a complete report, left the receipt incomplete until the closure timeout. **Expected behavior** Same rule as the local Runner. Such a turn closes as a lower bound right away. A complete report that arrives after the restore is still used, and a turn that starts after the restore stays pending until its own report is complete. **Steps to reproduce** `native-usage-delivery.integration.test.ts > a turn restored mid-turn on a remote Runner > closes the turn as a lower bound, also after the controller restarts between restore and terminal` fails on master: the receipt stays `accountingReceiptReady: false`. ## What Changed - **Receiver (`createNativeUsageReceiver`).** It now accepts `session.resumed` for turn-reporting providers (ACPX, OpenCode) while a turn is open. It journals the restore as a receipt like any other producer event, so turn accounting marks the open turn restored. A replacement controller rebuilds the same state from that journal. - **Restores never fail delivery.** A restore from another run or source, or one with no open turn, is ignored, not rejected. An unacknowledged lifecycle event would otherwise replay without end. - **Session-authority filter.** `acceptAccountingEvent` now uses a new `isNativeUsageDeliveryEvent`: the usage events plus `session.resumed`. A restore for another run is ignored there too. - **Recovery routing is unchanged.** `acceptRecoveryAccountingEvent` and `onRecoveryAccountingEvent` keep the usage-only `isNativeUsageAccountingEvent`, so a source run's restore never requires a proven recovery source. - **Turn accounting** gets a `hasOpenTurn()` helper. - **Docs:** `doc/SPEC-implementation.md` now covers both Runners. ## Verification - `native-usage-delivery.integration.test.ts`, describe `a turn restored mid-turn on a remote Runner` (real PostgreSQL): | Test | On master | Here | |---|---|---| | `closes the turn as a lower bound, also after the controller restarts between restore and terminal`: the restore is journaled by one receiver and the terminal arrives at a rebuilt one; replaying everything again is idempotent | fails | passes | | `keeps the turn's own final report when it arrives after the restore` (guard) | passes | passes | | `keeps a turn started after the restore pending, and journals nothing for a restore with no open turn` (guard) | passes | passes | - `native-session-executor.test.ts > binds retained warm transport usage to the successor run authority` covers routing through the session-authority filter: - a current-run restore reaches the successor's capture; - a draining-run restore reaches the drained capture; - an unrelated-run restore resolves with no capture; - recovery routing ignores a source restore. It fails when the filter is reverted to `isNativeUsageAccountingEvent`. - Related suites pass: `native-usage-delivery.integration` and `native-usage-accounting` (46 tests), and the usage, accounting, receipt and authority cases in `native-session-executor.test.ts` (224 tests). Server typecheck passes. - **The live remote path is unproven.** No Daytona wave ran because the Daytona Runner image has not been rebuilt yet. The local Runner counterpart (#20517) passed restart-chaos L1, seed 20386, 4/4 (`w-20261110T104332-s20386`). ## Risks - **Upgrade safety.** Usage state saved by current master has no journaled restore and rebuilds as before. A receiver that predates this change replays a journaled restore as an ignored event, which is the behaviour on master. A restore that an older controller already acknowledged is not replayed, so that run waits for the closure timeout as it does today. - **One extra receipt per mid-turn restore**, only for ACPX and OpenCode and only while a turn is open. Nothing is added before `turn.start`: a restore with no open turn returns immediately. - **Codex and other providers are unaffected.** They do not use turn accounting. - No schema or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 05:42:52 -06:00
DottaandClaude Opus 5.5	37808f2a4e	fix(accounting): close a local-Runner turn completed across a session restore as a lower bound (#20517) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs record usage receipts, and accounting settles a run's charge once its final receipt is complete. > - ACPX Claude reports provisional zero usage until a turn ends, then one final report. > - A restart can stop the provider after the agent's finish call and before that final report. The restored Runner then resumes the session and reports the same turn completed with no complete report. > - The receipt stayed incomplete, so the accepted run waited for the 10-minute closure timeout and then settled as unpriced. Its spend was missing from company totals in the meantime. > - This pull request closes such a turn as a lower bound, the same way a turn the Runner fails as lost on restore is already closed. > - The benefit is that an accepted run settles right away with its observed usage and an honest lower-bound label. ## Linked Issues or Issue Description Refs #20484 (restart-chaos L1 contention follow-up). **What happened?** After #20484 fixed the database contention, restart-chaos L1 (`--seed 20386 --db-pressure 4`, codex and claude, K1) reached 4/4 tasks done. Each task still failed the T-oracle `company_spend_settled` with `pending_runs=1`. A replay on master with `--keep` (`w-20261110T100747-s20386`) reproduced it, and I read the kept database: - The pending run (`71f71d10`, ACPX Claude) succeeded and had its result accepted. Its usage had `accountingReceiptReady: false`, zero tokens, `costStatus: unpriced`, and no cost event. - Its only turn started before K1, and every usage report in it was provisional (`runDeltaComplete: false`). K1 struck right after the agent's `paperclip_finish` call (source seq 118), before the turn's final report. - On restart, recovery verified the local process stopped and resumed the session (`session.resumed`, seq 119). The restored Runner then reported the same turn `turn.completed` (seq 124) with no usage report. - The other Claude run in that wave lost its turn as `provider_turn_lost_on_restore`, ran a new turn live with full usage, and settled at once. **Expected behavior** An accepted run whose open turn's final report was lost with the provider process settles promptly. It records its observed usage as a lower bound, like a turn lost on restore. A complete report that does arrive after the restore is still used. **Steps to reproduce** `server/src/services/native-runtime/native-usage-accounting.test.ts > closes a turn that completed across a session restore without its final report, as a lower bound` uses the event sequence from the kept database. It fails on master because the snapshot stays incomplete. ## What Changed - `createNativeTurnAccounting` observes `session.resumed` and marks every open turn as restored. A terminal event for a restored turn without a complete report marks it lost. The snapshot then closes it as a lower bound (`accountingUsageLost`), through the existing lost-on-restore path. A new `turn.started` clears the mark. - A lost terminal turn keeps its identity in the snapshot, so the executor's terminal-turn match certifies the final checkpoint as complete. This came from review. - The native session executor includes `session.resumed` when it rebuilds turn state from the run's event log after a controller restart. - `doc/SPEC-implementation.md` documents the case next to the existing lost-on-restore rule. ## Verification - Red on master, green here: `native-usage-accounting.test.ts > closes a turn that completed across a session restore without its final report, as a lower bound`. It also checks that replaying the history twice gives the same snapshot. - Red on the first revision, green after the review fix: `native-session-executor.test.ts > certifies a turn completed across a session restore as a lower bound (history replay: false|true)`. It drives the restored sequence through executor completion, both live and from the event history. - New guards that pass before and after: - `keeps a restored turn's own final report when it arrives after the restore`: a complete report after resume is used, not marked lost. - `keeps an incomplete turn pending when the session resumed before it started`. - The existing `keeps an incomplete provider report pending instead of trusting placeholder zeroes` still passes. - Related suites pass: `native-usage-accounting`, `native-usage-delivery.integration`, `closed-native-receipt-accounting` (107 tests), and the usage, accounting and receipt cases in `native-session-executor.test.ts` (151 tests). Server typecheck passes. - Live, restart-chaos L1, seed 20386, `--db-pressure 4`, codex and claude, K1, head `a6e521f06c`, clean worktree: `w-20261110T104332-s20386` passed 4/4. R1–R6 and the T-oracle passed for every task. `company_spend_settled` reported `pending_runs=0` (summary 85.30823 = cost events 85.30823), with zero serialization failures under about 397k pressure updates. On master the same seed failed every task on `company_spend_settled pending_runs=1` (`w-20261110T100747-s20386`, among others). ## Risks - A restored turn without a complete report now settles at its observed usage, flagged as a lower bound, instead of after 10 minutes as "closure timed out". In both cases the final report never arrived. The money recorded is the same observed usage; only the timing and the provenance label differ. - Scope: local Runner only. The executor fallback-recorder path sees `session.resumed`. The producer-owned delivery path for remote runners (`native-usage-delivery.ts`, `isNativeUsageAccountingEvent`) still drops it, so a remote ACPX/OpenCode turn restored mid-turn still waits for the closure timeout. That is a follow-up. - No schema or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 04:46:29 -06:00
DottaandClaude	8b3348c6ba	fix(runner): classify Claude's rejected selected model as an unavailable model (#20513) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner agents on the Claude (ACPX) provider run with the model the user saves on the agent > - When the provider cannot run that model, Paperclip should classify the run as `native_provider_model_rejected`, ask the user to choose a model, and let Retry finish the task > - Claude Code accepts an unknown model at session start and reports it only on the first turn, as an AIR `request` failure with a fixed title. The runner reported it as an unknown provider failure > - So the run failed as `native_provider_terminal_failed`, and the task stopped with `provider_effect_inventory_unavailable` ("Automatic recovery stopped. Recorded work is preserved…") without naming the model > - This pull request classifies that Claude title as an unavailable model, the category the native runtime already turns into `native_provider_model_rejected` > - The benefit is that an unknown Claude model asks for a model choice and names it, as Codex does after #20488 ## Linked Issues or Issue Description **What happened?** Config wave `w-20261110T094430-s207-cf` on `master` (82abb3e), `--adapters claude`: CF3 `model_id` failed both times (t01 K1, t04 K2) as `CF3/model_id:actionable_message`. The task showed `Stopped`, cause `provider_effect_inventory_unavailable`, "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated." The run was `native_provider_terminal_failed`, "ACP agent reported a terminal request failure." Root cause, from a direct ACP probe of the bundled `@agentclientprotocol/claude-agent-acp` 0.73.0 with the AIR `sessionFailure` capability and the model in `_meta.claudeCode.options.model` (as ACPX passes it): `session/new` lists `chaos-missing-model-abc12345` among the model options, so admission's model check passes. `session/prompt` then returns `stopReason: "end_turn"` with `_meta.jetbrains.air.sessionFailure` = `{"category":"request","severity":"error","title":"There's an issue with the selected model (chaos-missing-model-abc12345). It may not exist or you may not have access to it. Run --model to pick a different model.","actions":[]}`. ACPX turns that into "ACP agent reported a terminal request failure.", and `acpxProviderFailure` classified only AIR `limit` titles, so the provider failure stayed `unknown`. (Selecting the same model later through `session/set_config_option` instead returns JSON-RPC `-32603` "Internal error", `data.details` "Invalid value for config option model: <id>". The runner does not take that path for Claude, so this PR does not change it.) **Expected behavior** The run is `native_provider_model_rejected`, the task shows `choose_model` with the rejected model, no recovery incident opens, and Retry finishes the task after the model is repaired. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF3 --adapters claude --references model_id --tasks 2 --concurrency 2 --restarts K1,K2 --seed 207` on `master`. **Paperclip version or commit** `master` at 82abb3e. ## What Changed - `packages/paperclip-runner/src/drivers/acpx/provider-failure.ts`: for the `claude` agent, an AIR `request` session failure whose title is Claude Code's fixed "There's an issue with the selected model (…). It may not exist or you may not have access to it." is classified `model_unavailable` (`requires_action`). The native runtime already marks such a failed turn `native_provider_model_rejected`, and the server already projects that as `choose_model` with `rejectedModel`. - Other AIR categories, other agents and other request titles stay `unknown`. - `server/src/services/native-runtime/native-session-executor.ts`: a `native_provider_model_rejected` failure after the provider session opened now gets a choose-model next action ("The provider rejected the selected model. Choose a supported model for this agent, then retry the task. Automatic retries are stopped."). Before, only a rejection before the session opened got that action; the Claude case fell through to "Inspect the original provider failure and explicitly resolve the ambiguous session state; do not open a replacement provider session." Recovery mode, blocking and retry behavior are unchanged; only the action text changes. ## Verification - Red before, green after: `src/drivers/provider-failure.test.ts` — "classifies Claude's rejection of the selected model as an unavailable model", with the exact title captured from `claude-agent-acp` 0.73.0, through both `acpxTerminalResult` and the thrown form, plus negative cases. - Red before, green after: `native-startup-retry.integration.test.ts` — "asks for a model choice when the provider rejects the model on an opened session's turn (Claude ACPX)": a durable `turn.started` and the `NativeProviderTerminalFailure` the runtime throws; the coordinator and the board recovery action name the model choice, not ambiguous state. - Live: on `master` the wave failed (see above). On this branch: `w-20261110T103809-s208-cf` on head 3881650 (CF3 `model_id`, claude, K1 and K2): **2/2 pass**. Each task rests as `waiting_for_user` / `choose_model` with `rejectedModel: "chaos-missing-model-…"`, cause `native_provider_model_rejected`, next action "The provider rejected the selected model. Choose a supported model for this agent, then retry the task. Automatic retries are stopped.", rests again after the restart, and is done after the model is restored and Retry. An earlier wave on e552b24 (`w-20261110T100526-s207-cf`, 2/2 pass) still showed the ambiguous-state text that the second commit replaces. ## Risks - Low risk. The match is limited to the Claude agent, the AIR `request` category and Claude Code's fixed title. If Claude Code changes the text, the run falls back to today's generic failure. - No profile, digest or pack contract change (`rules/acpx-profile-identity.mdc`): ordinary Runner source. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, a direct ACP probe of `claude-agent-acp`). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 04:41:02 -06:00
DottaandClaude	82abb3ea04	fix(native): classify a Codex API-key unknown model as a model rejection (#20488) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner agents run on a provider (here Codex) with a model the user saves on the agent > - When the provider rejects that model, Paperclip should classify the run as `native_provider_model_rejected`, ask the user to choose a model, and let Retry finish the task > - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON envelope). With API-key auth, Codex returns a different terminal error for an unknown model > - So the run failed as a generic `adapter_failed` ("Native session failed"), and recovery opened a reconciliation incident ("Recovery needed") for a turn that never did any work > - This pull request recognizes the real API-key shape, bound to the configured model, and names the rejected model on the task > - The benefit is that the task asks for a model choice instead of a reconciliation, and Retry completes the task after the user picks a valid model ## Linked Issues or Issue Description **What happened?** The restart-chaos config fault CF3 `model_id` saves a Codex agent with the model `chaos-missing-model-<nonce>`. The run stays "running" for about 60 seconds, then fails with `errorCode: adapter_failed`, "Native session failed". The task shows "Recovery needed", cause `native_continuation_requires_reconciliation`. The grader fails as `CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01 and `w-20261110T053831-s205-cf` t02). Root cause: the wave runs Codex with an OpenAI API key. I drove the bundled `codex app-server` (0.160.0) directly with an unknown model: - With API-key auth, Codex retries the Responses API 404 ten times (about 59 s), then fails the turn with `{"message":"unexpected status 404 Not Found: The model \`<id>\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: …, request id: req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`. - With ChatGPT auth, the turn fails in about 3 s with `message` = `{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The '<id>' model is not supported when using Codex with a ChatGPT account.","code":"model_not_found"}}`. Paperclip already classified this one. `createNativeProviderFailureObservation` parsed only the JSON envelope, and the runner's provider-failure category for an HTTP 404 is `unknown`. The terminal therefore stayed unclassified. **Expected behavior** The run is `native_provider_model_rejected`, the task shows `userAction: "choose_model"` and names the model, no reconciliation incident opens, and Retry completes the task after the user picks a valid model. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex --references model_id --tasks 1 --concurrency 1 --restarts K2 --seed 206` on `master` (with `OPENAI_API_KEY` set). **Paperclip version or commit** `master` at 5ab98b0. ## What Changed - `server/src/services/native-runtime/native-provider-failure.ts`: the Codex observation recognizes the API-key terminal shape (`codexErrorInfo` `httpConnectionFailed` or `responseStreamDisconnected` with `httpStatusCode: 404`, and the message prefix for the configured model only). It classifies it as `native_provider_model_rejected` with a new closed diagnostic `NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC` (`codex`, `model_not_found`, `404`, `api_key`). No provider text or model name is stored in the diagnostic. A recoverable turn, another model's 404, or another status is not classified. - `readNativeModelRejectionDiagnostic` accepts both closed diagnostics, so run failure diagnostics show it. - `TaskInterruptionPresentation.rejectedModel` (shared type): with `userAction: "choose_model"`, the task query carries the model id from the run's native execution input. Only a `native_provider_model_rejected` run discloses it. - `TaskChatThread` names the model in the "Model unavailable" marker: `The provider rejected the model "<id>".` - The restart-chaos config wave reads `interruption.rejectedModel` as task-level text. For `choose_model`, its repair step uses the Retry that the "Model unavailable" marker offers (`POST /agents/{id}/wakeup` with `reason: "retry_failed_run"`, the same request as `agentsApi.retryFailedRun`), because a model rejection does not offer the task-level Retry. ## Verification - Red before, green after (the classification line disabled makes both fail): - `native-provider-failure.test.ts` — "classifies the API-key 404 for the configured model as a model rejection (codex 0.160.0 shape)", with the exact terminal error captured from `codex app-server` 0.160.0, plus negative cases. - `native-provider-failure.integration.test.ts` — new `api-key` scenario through `executePaperclipNativeSession`, the persisted evidence reader and `finalizeNativeRun`: the run ends `native_provider_model_rejected`, the issue is `blocked` with that decision, and no issue recovery action is created. - `task-interruption-query.test.ts` — the real task query carries `rejectedModel` only for a model rejection. `TaskChatThread.test.tsx` — the marker names the model. - `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck` pass. - Live proof: config wave `w-20261110T090801-s206-cf` (CF3 `model_id`, codex with an OpenAI API key, K2, seed 206) on commit 05e8938: **1/1 pass**. The run ends `native_provider_model_rejected`; the task rests as `waiting_for_user` / `choose_model` with `rejectedModel: "chaos-missing-model-472e1851"` and execution cause `native_provider_model_rejected` (no reconciliation incident), rests again after the K2 restart, and finishes 27 s after the model is restored and the marker's Retry is used. On `master` the same wave failed as `actionable_message` with "Recovery needed" / `native_continuation_requires_reconciliation` (`w-20261110T061914-s206-cf`). ## Risks - Low risk. The match is bound to the configured model, a 404 status and a non-recoverable turn, so other provider failures keep their current behavior. - The match depends on the Codex terminal text. If a later Codex release changes it, the run falls back to the old generic failure, not to a wrong classification. - Claude (ACPX) unknown models are not changed here. Claude Code reports an unknown model as a synthetic assistant message ("There's an issue with the selected model …") rather than a provider error; this PR does not change or live-verify that path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, a direct `codex app-server` probe). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 03:11:10 -06:00