PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-08 12:33:05 +01:00

Author	SHA1	Message	Date
Nicky Leach	8c062fa481	fix(native): deliver plugin VM Runner usage to Core (#19162 )	2026-11-04 21:02:22 -08:00
Nicky Leach	737db73275	fix(native): measure resumed run usage from the prior run's total (#19161 )	2026-11-04 18:55:14 -08:00
DottaandClaude Opus 5.5	11f2a5293d	fix(native): name the proof that holds a recovered per-turn owner (#19061 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - A follow-up must prove that the previous Runner owner is finished before it starts a new Runner. #18846 added a retirement path for an owner that finished without a saved Runner checkpoint. > - On a test deployment, follow-ups still fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The retirement path has about twenty separate fences, and each refusal gives the same error. > - The read-only run data cannot tell which fence held these runs. A code fix without that evidence would be a guess. > - This pull request names the first failed retirement proof in the error. Admission behavior does not change. > - The benefit is that the next failure on a deployment identifies the exact fence, so the follow-up fix can target it. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #18992 (settlement from live state). What happened? On a test deployment, 16 runs in one day failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. All were per-turn `acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In the runs that I inspected, the previous run committed its result. In one of them, `harness_state.backup.persist` and `session.checkpoint.persist` ran for about 346 seconds and failed, so the Runner left no suspension receipt or failover backup. That is the case that #18846 addresses, but the follow-ups still failed. The run log does not show the retirement message, and the error is the same for each fence. Thus the cause cannot be found from the run data. Expected behavior When admission holds a follow-up, the error names the proof that failed. Steps to reproduce 1. Seed a per-turn sandbox owner that committed its result without a Runner suspension receipt. 2. Make one retirement proof fail. For example, leave the owner phase `observed`, or leave a `runner.suspend` command pending. 3. Admit a follow-up in a new sandbox. 4. Before this change, every case fails with the same error text. Paperclip version or commit Observed on a deployment before `e107a75755`. The same error text is on `master`. ## What Changed - `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead of `false`. `readSettledRunnerdProjection` returns the journal refusal reason instead of `null`. - When admission tries the retirement and it fails, the error is `prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`. A warm session gets no suffix, because retirement does not apply to it. - Reasons use only fixed lists of known names: owner phase, recovery state, controller fence reason, unsettled command type and status, sandbox outcome, and quarantine copy result. Any other stored value appears as `other`. Reasons do not include payloads, file names, or paths. - The same proofs apply in the same order. Admission accepts and refuses the same states as before. - `doc/execution-semantics.md` describes the suffix. ## Verification - Failure before the change: I ran the updated `native-recovered-owner-followup.test.ts` against the unchanged executor. 18 tests failed. Example: expected `...indeterminate; retirement_refused=owner_observed`, received `...indeterminate`. - After the change: `cd server && pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 passed. Each held case now asserts its reason: `owner_retryable_failure`, `controller_remote_controller_lease_live`, `owner_recovery_resuming_session`, `journal_commands_unsettled(runner.suspend.pending)`, `journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`, `runner_lifecycle_not_suspended`, and `quarantine_runner_lifecycle_not_suspended`. - The success cases still retire the owner and admit the follow-up. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227 passed, 20 skipped. - `pnpm exec vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/__tests__/task-workspace-binding.test.ts`: passed. - `tsc --noEmit` reports no errors in the changed lines. This worktree did not build every workspace package, so the full-package typecheck shows unrelated missing-module errors. CI runs the full typecheck. - Not verified live: no deployment has run this change. ## Risks - Low risk. Admission logic is unchanged. Only the error text gets a suffix. - Consumers that match the error by prefix still match. `failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch: ")`. No code compares the full string outside tests. - This change does not fix the held runs. It finds the fence. A second change is needed after a deployment reports the reason. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution, and read-only browser access to a test deployment. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 18:50:01 -06:00
DottaandClaude Opus 5.5	3a72efebe5	fix(native): confirm no runner process when remote staging fails before launch (#19059 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native sandbox runs supervise a remote `paperclip-runnerd` process. When that process exits during a turn, the transport restarts it inside a reconnect grace window. > - The server launcher stages the runtime assets first. Then it sends one launch RPC that starts runnerd. > - When staging failed, the launcher reported the exit as unconfirmed. This happened although no launch RPC was sent and no process could exist. > - Reconnect recovery treats an unconfirmed exit as a possible second owner. So it stopped at once with `runner_process_termination_unconfirmed` and did not retry. > - This pull request confirms the exit for failures before the launch RPC. Failures after dispatch stay unconfirmed. > - The benefit is that a transient staging failure, such as a full sandbox disk, uses the normal bounded retry. A real failure ends with `transport_reconnect_grace_exceeded` and keeps the staging error as the detail. ## Linked Issues or Issue Description What happened? On a hosted staging deployment, 7 native ACPX runs failed in 24 hours with `Provider notification transport failed closed.` (`native_provider_terminal_failed`). In every run, the runner process exited during the turn. The replacement launch then failed while it uploaded the runtime context into the sandbox: `runner_process_termination_unconfirmed: replacement runner health could not be verified: ... .paperclip-upload-<uuid>.tar.gz: create: ... no space left on device`. The sandbox disk was full. No replacement process was started. But the launcher said that it could not confirm the exit. So recovery did not retry inside its grace window. Expected behavior A staging failure before the launch RPC proves that no replacement process exists. Recovery must retry within the configured reconnect grace. If the failure continues, the run must end with the reconnect-grace failure and the staging error. Steps to reproduce 1. Create a remote runner launcher whose `ensureArtifact` step throws, for example with an ENOSPC upload error. 2. Wait for the handle completion to reject. 3. Before this change, `isExitConfirmed()` returns `false`, and no launch RPC was sent. The transport then fails closed with `runner_process_termination_unconfirmed`. Paperclip version or commit Observed on staging revision `89703c3`. Reproduced on `master` at `be20d4ad97`. Related: #19035 and #19014 change other runner paths (replayed terminal turns and redacted external tool results). They do not change remote launcher exit evidence. #18999 adds a free-space reserve for host-volume seed copies. It does not change sandbox disk use or this classification. ## What Changed - `createRemoteRunnerProcessLauncher` records when it dispatches the launch RPC. If the completion fails before that point, the handle reports `isExitConfirmed() === true`. The original error is kept unchanged. - A failed or timed-out launch RPC still reports an unconfirmed exit, because the RPC may have started runnerd. - Added a launcher regression for a staging failure before dispatch. - Added safety cases for a failed and a timed-out launch RPC. - Added a transport regression with the real `paperclip-runnerd`. A replacement whose staging fails is retried, and the next launch restores the durable session without a notification failure. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote runner process supervision"` failed `confirms no replacement process exists when staging fails before the launch RPC` with `expected false to be true`. The two launch-RPC safety cases passed. - Green after the fix: `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1131 of 1131 tests. - `npx vitest run src/services/native-runtime/setup-readiness.test.ts` passes 98 of 98 tests. - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts` passes 246 of 246 tests with the real debug `paperclip-runnerd`. This includes the existing case where a replacement with a dispatched but unverifiable process still fails closed. - `npx tsc --noEmit -p server` and `pnpm --filter @paperclipai/paperclip-runner typecheck` pass. - Not run: a live hosted sandbox with a full disk. The staging data showed the exact error, but this change was not deployed there. ## Risks - Low risk. The change affects only failures that occur before the launch RPC. In that window, the launcher has not run any sandbox command that can start runnerd. - Retries now run for the reconnect grace window, which is bounded and uses the existing backoff. Each failed staging attempt cleans its own upload scratch. - This change does not free sandbox disk space. If the disk stays full, the run still fails, now with `transport_reconnect_grace_exceeded` and the ENOSPC detail. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use, shell execution, and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 18:26:02 -06:00
DottaandClaude Opus 5.5	5723fd82ca	fix(native): release a fresh claim when task preparation fails (#19065 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run claims its finalization coordinator before task preparation. Then it starts the provider turn. > - Task preparation can fail after the claim and before any provider turn. > - In that case the executor kept its claim. Failure settlement read the claim as a live owner and stopped. Nothing failed the run. > - The 60-second execution-control watchdog then failed the run as `execution_finalization_deadline_exceeded`. The original error was lost. > - This pull request releases only that exact fresh, unstarted claim after renewal stops. > - The benefit is that ordinary settlement fails the run at once with its real error and keeps the workspace files. The watchdog stays a backstop. ## Linked Issues or Issue Description Refs #19028. That PR keeps the earlier failure code when the watchdog fails a run. In this case no earlier code is saved, so it cannot help. This PR removes the wait for the watchdog. I searched open PRs for duplicates. #18820, #19034 and #19038 touch near code, but none release a preparation claim. What happened? A native task with a prepared sandbox workspace failed during preparation, after the controller claim committed. The coordinator stayed `observed` and kept this controller's lease owner. The heartbeat failure path calls workspace preservation. Preservation refuses any lease owner without a matching restart claim, so it raised `NativeWorkspaceFinalizationBusyError`. The heartbeat then treated ownership as held and returned without a terminal write. About 60 seconds later the watchdog failed the run with a generic "verify the provider stopped" action. The real error (for example `native_harness_storage_owner_missing`) stayed only in the workspace operation record. Expected behavior When no provider turn started under a fresh claim, settle the failure like a failure before the claim: retain the source, fail the run with its own error, and do not wait for the watchdog. Steps to reproduce 1. Start a native run with a prepared remote workspace. 2. Make task preparation fail after `native.coordinator.claim` commits (for example, a preparation callback throws). 3. See that the run stays `running` until the control deadline fails it as `execution_finalization_deadline_exceeded`. ## What Changed - `native-execution-lease.ts`: add `releaseUnstartedNativeExecutionClaim`. It clears the lease owner and expiry only for the exact run, company, issue, owner and attempt, while the phase is `observed` and no result exists. - `native-session-executor.ts`: when task preparation fails after a fresh claim, mark that claim. The startup owner releases it after renewal stops. A restart-recovery claim, a startup-cleanup context, Stop, restart detach, or a different owner keeps the old behavior. - Tests: an executor regression for the fresh and restart cases, and a real-database test for release and workspace retention against the exact owner, a foreign owner, a different attempt and an accepted result. - `doc/execution-semantics.md`: state that the deadline is a backstop and describe the release. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "releases only a fresh claim"` fails with "releaseUnstartedNativeExecutionClaim ... Number of calls: 0". - Green after the fix, with the same command. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-execution-lease.test.ts src/services/execution-control-reconciliation.test.ts` passes (embedded Postgres for the retention test). - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live run against a real sandbox at this revision. The staging evidence is read-only. ## Risks - The release is fenced by the exact owner, attempt, `observed` phase and no result. Renewal stops before the release, so a renewal cannot see a missing owner and detach or cancel the run. - The fix covers failures inside task preparation only. Failures later in startup, or after a restart-recovery claim expires, still reach the watchdog. Those keep fail-closed behavior and need separate evidence. - If the release write fails, the run keeps the old watchdog path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 18:25:36 -06:00
DottaandPaperclip	818d2bb661	fix(native): preserve closed post-result failure diagnostics (#19126 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution persists a result before it completes cleanup and workspace export. > - A later execution error enters the failure-write path. > - That write must reject an already selected result. The existing code reports lease loss for every rejected write. > - This can hide the category of the first error even when the same controller still owns an unexpired lease. > - This change adds closed diagnostic labels after exact ownership and accepted-result checks. It keeps the existing control error and recovery policy. ## Linked Issues or Issue Description What happened? A local real-PostgreSQL regression accepts a result through the production control plane, then injects a harmless filesystem failure. The production executor reports `native_session_lease_lost`. It loses the original failure category because the failure transition requires a null result ID. Expected behavior Keep the result guard and the existing lifecycle classification. Record a closed diagnostic category when the selected result is accepted and the controller still has exact, unexpired authority. Keep ordinary lease-loss fencing when that proof is absent or its diagnostic lock is busy. Steps to reproduce Run the new post-result cases in `server/src/services/heartbeat/runtime-execution.test.ts`. They use the real executor, control-plane result commit, heartbeat recovery path and PostgreSQL. The provider boundary is controlled. No live provider is used. Paperclip version or commit The regression fails on base `7d1e9b3477`. Related open PR #16917 changes legacy lease recovery. Open #19063 adds terminal finalizer reasons, and #15310 preserves database finalization errors. They do not change this executor diagnostic branch. This change only adds native diagnostic evidence and does not reclaim a lease. ## What Changed - Add a typed diagnostic with a closed stage, category and accepted-result flag. Do not retain the source message, path, arbitrary code or lease token. - After a rejected failure write, check the selected accepted result and the exact company, task, run, owner, attempt, generation, process identity and expiry. Use a coordinator-only `NOWAIT` lock. Fall back to the original control error if the diagnostic read fails. - Report the closed event after rollback. Keep `native_session_lease_lost` as the error message and leave every failure-write predicate unchanged. - Add real-PostgreSQL safety cases for stale and expired authority, owner replacement, Stop state, result selection, pending accounting, one provider invocation, closed labels and diagnostic sink failure. - Document the closed event, unchanged authority, and possible late diagnostic sink completion. - Add a competing-row-lock barrier. It verifies actual PostgreSQL lock refusal and control settlement before the competing lock is released. ## Verification - Permanent regression on unchanged base: RED because the closed diagnostic is absent. - NOWAIT mutation: RED when only NOWAIT is removed. PostgreSQL reports lock timeout instead of immediate lock refusal. The test uses barriers and the actual database error, with no elapsed-time assertion. - Final related suites: 56 tests passed across runtime execution and provider failure integration tests. - Full server source and test typecheck: passed. - Full `pnpm -r typecheck`: passed in an independent locked-dependency checkout. - Full `pnpm build`: passed. - Full `pnpm test:run`: running in the independent immutable checkout. All six chat shards passed; the remaining suites are still pending. - The first current-head CI attempt failed in two cases: workspace runtime initial fixture HTTP readiness and instructions-bundle error-message rendering. Both exact cases passed in isolation on unchanged PR head and base. The original causes remain unknown after bounded test-only observation. One controlled same-head rerun of each failed job passed. Current required CI and all other completed checks passed. Original failures remain recorded; no infrastructure or flaky-test claim is made. - Module boundary check and `git diff --check`: passed. - Exact-head Greptile: 5/5 with no actionable findings or open review threads. Automatic native review remains queued. That advisory check is not required by current master rules; no current-head native assessment is claimed. - Earlier fixture setup failures remain recorded. These included missing fixture fields and observing the Drizzle wrapper instead of its PostgreSQL cause. They are not the regression RED. An earlier broad typecheck was interrupted for the required NOWAIT revision; the final full typecheck passed. - No hosted export recovery or original hosted failure cause is claimed. ## Risks The extra read is diagnostic only. It uses NOWAIT and cannot wait for a competing coordinator lock. Its failure preserves the existing control classification. The event sink uses the existing 60-second execution-control bound. A slow sink can finish later and emit a late event. Result selection, Stop, accounting, retry eligibility and provider work do not depend on event success. No schema or lifecycle contract changes are required. ## Model Used OpenAI GPT-6.1, model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 17:45:53 -06:00
DottaandClaude Opus 5.5	86afc2048b	fix(native): admit follow-ups after an owner finished in this process (#19071 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - Before a follow-up starts a new Runner, it must prove that the previous owner's controller stopped. #18846 added a retirement path for this proof. > - The native controller runs inside the webserver process. Thus the owner's recorded controller PID is the PID of the webserver. > - When the same server process serves the follow-up, that PID is alive and has the same start time. The proof reads this as "controller still alive" and refuses. The follow-up then fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. This continues until a deploy replaces the process. > - This pull request uses the controller's per-process boot identity to recognize that this process recorded the owner. The process then uses its own session registry to prove that the owner's controller stopped. > - The benefit is that follow-ups on a long-lived server no longer fail before provider work. All other ownership fences stay as they are. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #19061 (names the refused retirement proof). The two changes are compatible. Whichever merges second needs a small rebase in `retireFinishedPerTurnRunnerdOwner`. What happened? On a test deployment, 17 runs over 14 tasks failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Each run had a per-turn lifecycle in a Daytona sandbox and got a new sandbox. In each inspected case, the previous run on the task succeeded and committed its result. Its run profile had no `nativeControllerRecovery` pointer. Without that pointer, `evaluateNativeControllerTakeover` is not lease-fenced, so it uses process liveness. The recorded `controllerPid` is `process.pid` of the webserver. When the follow-up is served in the same process, `process.kill(pid, 0)` succeeds and the start time matches. The result is `controller_still_alive`, and retirement refuses. Expected behavior A committed owner that this server process finished, released, and no longer runs is proven stopped. The follow-up then starts a new Runner, as #18846 intended. Steps to reproduce 1. Seed a per-turn sandbox owner that committed its result and released its lease. Record this process's controller identity (`currentNativeControllerIdentity()`) as the owner. 2. Keep the owner's file-backed journal in the local session root, and remove the database recovery pointer. 3. Admit a follow-up in a new sandbox. 4. Before this change, admission fails with `prior_owner_terminal_state_indeterminate`. Paperclip version or commit Observed on `e107a75755`. The same code is on `master`. ## What Changed - `native-restart-recovery.ts`: add `isCurrentNativeControllerProcess(owner)`. It is true only when the owner's `controllerBootId` is this process's per-start boot UUID and `controllerPid === process.pid`. A recycled PID or another container never matches. - `retireFinishedPerTurnRunnerdOwner`: when the owner was finished in this process and no `activeNativeSessions` or `nativeSessionStartups` entry exists for its run, treat its controller as stopped. Otherwise, use `evaluateNativeControllerTakeover` as before. The same in-process check runs again just before the archive. - All other retirement proofs are unchanged: committed result, released lease, no recovery state, settled journal, answered tool inputs, quarantine copies, and new or replacement sandbox. - `native-recovered-owner-followup.test.ts`: the seed helper accepts the owner's controller identity. New cases cover same-process admission (with and without a recovery pointer), an exited legacy controller, another live controller, and the same PID with a different boot identity. - `doc/execution-semantics.md`: describe the same-process stopped-controller proof. ## Verification - Failure before the change: with only the new tests applied, `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts` fails 1 case: "admits a follow-up in the process that finished the owner: local journal only". It expected admission and received `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`, the exact error from the deployment. The safety cases already passed on the old code. - After the change: the same file passes 27 of 27. - Adjacent suites: `native-recovered-owner-followup`, `remote-warm-session-recovery`, `native-runner-restart-recovery.integration`, `native-restart-recovery`, `native-session-resume`: 210 passed, 20 skipped. - `native-session-executor.test.ts`: 1128 passed. - `tsc --noEmit -p server/tsconfig.json`: no errors. - Not verified live: this change has not run on a deployment. Run data came from read-only API calls only. No recovery was done on the deployment. ## Risks - Second executor. The new proof applies only when this exact process recorded the owner, and this process has no session or startup for that run. The database proofs still require a committed result and a released lease. Another live controller, or the same PID with a different boot identity, still holds the owner. Tests cover both cases. - Scope. The change applies only to the per-turn retirement path. The warm remote recovery path (`inspectRemoteWarmSessionRecovery`) is not changed. Warm handoff in the same process uses the in-memory idle owner instead. - Hypothesis limit. The deployment API does not expose finalization rows. Thus I inferred that the same process served each follow-up from the evidence: no recovery pointer, a local journal present, and no other failing proof. The local reproduction gives the exact error. #19061 can confirm the refused fence on the deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding in Claude Code with tool use (shell, browser for read-only staging API reads, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 16:29:25 -06:00
Nicky LeachandClaude Opus 5.5	e107a75755	fix(native): decide session settlement from live state (#18992 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so a later run attaches to a session that an earlier run settled. > - Three gates decide if a session is settled: Core's journal check, the Runner's attach check, and the Runner's drain of provider messages after a turn. > - Each gate kept a list of harmless events and refused anything else. Every new harmless event became an outage. > - An example: Codex replays a settled turn's usage on `thread/resume`, so `run.attach` refused every session that restarted after its idle timeout. > - Events after a terminal describe the past. Refusing the session cannot undo them, and it hides them. > - This pull request asks a different question: is anything live or waiting right now? Past events are recorded and no longer block. > - The benefit is that harmless events stop breaking session continuity, with no event list for Core and the Runner to keep in step. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for `run.attach requires a settled Codex provider session` and `prior_owner_terminal_state_indeterminate`. Related work: #18890 added one more accepted event to Core's list. This change replaces that list. #18991 handles late usage and is independent of this change. What happened? On a test deployment, 11 runs failed in 21 days when `run.attach` refused with `requires a settled Codex provider session with no pending events`. In all 11, the four pending events were two startup audit events, `session.resumed`, and a `codex_resume_usage_snapshot` diagnostic. The diagnostic is what the attach check refused. The retry then failed with `prior_owner_terminal_state_indeterminate` in 6 of them, which hid the first failure. Expected behavior A session with nothing live or waiting accepts the next run. If attachment fails, the run reports the real cause. Steps to reproduce 1. Finish a Codex turn on a native Runner session. 2. Wait for the idle timeout so that the provider process stops. 3. Start the next run. Codex replays the usage for the settled turn on `thread/resume`. 4. The attach fails. `durable_backend_attaches_after_a_settled_resume_usage_snapshot` reproduces this on the unfixed code. Paperclip version or commit Observed on `8fd917aedf`. The same check is on `master` before this change. Deployment mode Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - Runner attach: `attach_run` refuses only for live or waiting state: no thread, a closed session, a session that needs reconciliation, an active or ambiguous turn, an active goal, a pending tool call, or an unanswered provider request. The error now names the reason. - Pending events: they no longer block attachment. They stay in the retained queue, and the Runner commits them under the prior authority before it rotates. `session.resumed` is still discarded because it is superseded. - Runner drain: `drain_completed_turn_tail_for_warm_attachment` no longer needs a list of known messages. It still fails on a message that names another turn, on `turn/started`, on a provider request, and on a process exit. - Core: `isSettledRemoteWarmControlState` accepts any authenticated event after the terminal unless it starts work (`turn.started`, `turn.accepted`, `run.attached`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created`), or a goal snapshot or update shows an active goal or work in progress. All identity, source and protocol checks are unchanged. - Visibility: the next run's log names the event types that were recorded after the previous run finished. - Real cause: a same-run retry after a failed attachment now fails with `prior_attach_failed: <original failure>`, instead of `prior_owner_terminal_state_indeterminate`. - Docs: added "Session settlement after a run" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 102 passed, 2 ignored. `--lib`: 478 passed, 2 ignored. `--test durable_recovery`: 3 passed. - Failure before the fix: `durable_backend_attaches_after_a_settled_resume_usage_snapshot` failed on the unfixed code with the same error as production. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence\|recovers a blocked\|automatically recovers only a proven settled local session\|retained\|warm"`: 476 passed, 680 skipped by the filter. - `pnpm exec vitest run src/services/native-runtime/native-session-resume.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - Real data, read-only: I decrypted the 11 failed runs' stored events on a test deployment and printed only event types, field names, and thread and turn IDs. All 11 show the same four events. - Audit of live state: I read the Runner's provider state and request tracking to build the attach check. It is a code audit. I have not proved that every kind of live work sets a flag. The Runner holds active turns, ambiguous starts, tool calls and provider requests in state today. - Not covered: the wiring that throws `prior_attach_failed` has unit coverage for its message only, not an end-to-end test. - Review: Greptile Apex scored 4/5, then 5/5 at the current head. - Not run locally: the full test suite. CI provides that coverage. - Not verified live: no live run has used this change. ## Risks - Fail-open for unknown live work: an event type that starts work, but is not in the short list and sets no state flag, would now pass. This is the main tradeoff. The previous list failed closed, but it caused real outages. - More events reach the journal after a terminal. The usage replay diagnostic is now delivered rather than refused. These events are informational, and they are named in the next run's log. - A failed attachment is not retried on the same run. Some refusals could clear on their own, such as a turn that finishes, and some attach errors are not about live work, such as a failed provider restore. The run now reports the cause, and it needs another run to continue. On a test deployment, the next successful run after each failure used a new provider thread and a new execution workspace. A later change could add same-run retry. - Rollout order: deploy Core before the Runner. An older Core would refuse the new events that a new Runner commits after a terminal. - Migrations: this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 12:04:29 -08:00
DottaandPaperclip	3a2249dd57	fix(runner): retry proven ingress bootstrap failures (#18981 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. What happened? A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. Expected behavior Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. Steps to reproduce 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. Paperclip version or commit Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 12:16:51 -06:00
DottaandPaperclip	396b7fc62a	fix(runner): scope session diagnostics to the active run recorder (#18944 ) ## Thinking Path > - Paperclip manages agents and their work. > - The native executor can retain a transport between runs. > - Each run recorder has a shorter lifetime than that transport. > - The transport kept its first recorder after warm reuse and completion. > - A later close could write to the wrong run or a sealed object log. > - This change binds diagnostics to the active run epoch and retires that binding at execution exit. > - The existing recorder still owns accepted-byte draining and honest incomplete finalization. ## Linked Issues or Issue Description What happened? A warm native transport retained the first run's diagnostic callback. Reusing the session for another run did not change that callback. A later idle close could append to a retired writer and emit a persistence warning. Expected behavior Active diagnostics belong to the current company, session and run. A retired transport callback must not use a completed recorder. Diagnostic metadata must not delay essential execution cleanup. Steps to reproduce Run the native warm-session regression with two sequential run owners. Close the retained transport after both runs finish. The original code calls the first recorder. The real completion regression also observes the resulting closed-writer warning after normal lease cleanup. This is a local source defect with behavioral red/green proof. Repeated hosted generic persistence warnings motivated the investigation, but their original exceptions were not retained. This PR does not claim to identify every such warning or repair an export failure. Related #18379 queues sanitized performance spans in heartbeat. It does not own transport diagnostic callback lifetime. ## What Changed - Bind transport diagnostics to an exact active company/session/run epoch, including warm reuse. - Activate a retained channel in the existing session-admission hook after identity validation and before attachment starts, including an attachment failure before `onSession`. - Forward active diagnostics synchronously. Detach their recorder at execution exit. Keep the original asynchronous metadata semantics. - Keep idle transport evidence inspectable. Log only a fixed category and trusted company/run identifiers to the process logger. - Preserve recorder-owned byte draining, redaction, source coverage and genuine persistence-failure warnings. - Cover actual native executor, completion and lease-close boundaries, including stalled local appends and exact object-log source/hash preservation. - Place the retained-workspace test fixture under the existing published-path contract after merging newer recovery tests from master. Production validation and native/catalog assertions remain unchanged. ## Verification - Current head: `c500e4933a85a4c453f66c872fa335108c661b15`. Normal master merges preserve upstream behavior. The one textual conflict was in the executor imports. - The first GitHub run tested merge tree `a4cb59448a3b5cc2047ac41c691720e6acec6e8d`, which included newer recovery tests from master. Five cases failed with `remote_workspace_published_path_invalid`. The same five failures reproduced on an identical local tree. The fixture now puts retained files under `.paperclip-workspaces/<workspaceId>` and keeps the target, lease and authority bound to that path. Production validation and the native-success/catalog-denial assertions are unchanged. - On unchanged `c500e493`, the full recovery file passed 244/244 tests. Existing cold-restore and occupied-root controls passed 4/4. Server typecheck, server production build, module boundaries and whitespace checks passed. An intermediate missing-import setup failure was corrected and is separate from the five-case behavioral red. - Before that fixture correction, local `f2fdaddb` passed 1,221 tests across the five diagnostic, executor, output, completion and object-store files. Recursive typecheck and full repository build also passed. Those results are tied to that earlier revision; [fresh current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41842043248) must qualify the final merge. - Behavioral regressions cover the original closed-writer warning, warm A-to-B ownership, and an actual Runner attachment exception before `onSession`. The admission hook gives the attachment diagnostic to B and leaves A untouched. The attachment fixture uses synthetic event persistence and outer failure projection; it does not certify a real database terminal projection. - The tests preserve synchronous diagnostic admission and asynchronous metadata recording. A stalled append reaches the existing bounded finalization result with unknown size/hash. The normal object-log control preserves canonical bytes, SHA256 and complete source coverage after lease cleanup. - An initial local run had 37 PostgreSQL startup failures because the Darwin package lacked aliases to its included versioned libraries. Only isolated `node_modules` aliases were restored. No source or lockfile changed for this setup issue. A passing intermediate run crossed a source merge and is excluded from immutable-head proof. - An earlier combined test run passed 1,216 assertions but exited with a completion `afterAll` timeout. Its cause remains unproved. Later isolated 60/60 passes used the original timeout and do not erase that failure. An earlier repository-wide `pnpm test:run` was interrupted after its source became obsolete; it is not full-suite or current-head proof. Greptile reviewed the exact `c500e493` head at 5/5 with no open findings. Fresh CI must pass before merge. The native review is queued behind its separate saved-file recovery; no current-head native assessment is claimed. The provider execution loop is a fixture. These tests do not start a model or prove hosted recovery. Missing accounting receipts remain pending, not zero cost. Checkout-release invocation is observed; this is not a separate task-arbiter journey. The original native review published a valid finding before a separate workspace-export failure. The finding is fixed; this PR does not claim to fix that export failure or every hosted persistence warning. ## Risks Idle process logs intentionally omit raw provider text. The existing bounded transport evidence retains its own diagnostics. Accepted fs appends that do not settle retain the existing unknown-size/hash warning and cannot be certified complete. No new timeout, storage policy, schema or provider execution is added. ## Model Used OpenAI `gpt-6.1-sol` through Codex assisted with diagnosis, implementation, tests and review. The exact context-window size is not exposed. Tool use and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local tests and recorded passing checks, pending checks and unresolved failures above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will require all Paperclip CI gates to be green before merge - [x] I will require Greptile 5/5 with no unresolved findings before merge - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:04:52 -06:00
DottaandPaperclip	b891e69bd2	fix(native): correct unread wake comments before completion (#18937 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents report completion through a server-owned tool boundary. > - A truncated external-chat request requires a complete run-bound comment read. > - The terminal guard checks that receipt after the provider ends its turn. > - Early completion feedback omitted the same check and could accept an unread report. > - The provider then lost its chance to correct the report in the same turn. > - This pull request returns the existing read requirement through the completion tool before accepting the report. > - The final guard still protects the exact source and snapshot after provider settlement. ## Linked Issues or Issue Description Related: #16995 keeps run-scoped readers available to retained provider threads. #18913 preserves queued comment input. This change fixes a separate completion-feedback boundary. What happened? A native external-chat run could submit an accepted completion without reading all server-bound comments. The terminal guard then failed the run with `native_current_wake_comments_unread`. Early feedback also accepted partial, changed and foreign read receipts. The agent did not receive the required correction while it could still act in the same turn. Expected behavior The completion tool must reject the report with a `read_current_wake_comments` instruction and the cursor requirements. A complete current read must permit the corrected report in the same turn. The final snapshot check must remain effective. Steps to reproduce 1. Create a running native task with a truncated external-chat wake and exact accepted comment IDs. 2. Call completion feedback before reading the comments, or after only a partial or stale read. 3. Observe that the old code returns an accepted report instead of the read correction. ## What Changed - Check the existing run, company, task and agent binding before every completion-feedback acceptance return. - Reuse the authoritative current-wake reader and terminal receipt validator. - Share the two unchanged read-error codes from their owner across the validator, feedback and executor classification. - Convert only the two owned unread or changed-snapshot errors into a bounded tool correction. Other errors still propagate. - Keep the final validator, receipt parser, result persistence and admission policy unchanged. - Add real PostgreSQL regressions and an actual OpenCode MCP tool sequence with a controlled local provider process. - Document the early correction and final snapshot fence. ## Verification The source checkout uses Node 24.22.2 on macOS arm64 with a fresh owned embedded PostgreSQL database. Final published tip: `37bfb269432f86f98cab1a58c09b5519ed9477fd`. Its production code is commit `8c7297318c3a841fc1c72c846acfaabf58adabe9`; the last commit adapts three existing module mocks. The expanded affected group passes 1,284 cases with 35 existing opt-in skips. The skips require captured private-runner transition fixtures and a configured fresh database, which were not supplied. All nine real PostgreSQL/MCP cases execute and pass. Recursive typecheck and the full build pass for production commit `8c7297318c`; the final changed test layer also passes. - Untouched master `1a39dea9aa2bb9bea416439b0a574c788e88b9a0`: four causal failures. Unread, partial, changed and foreign complete receipts incorrectly receive accepted completion feedback. The ordinary inline-complete control passes. - Current source: all nine new cases pass. The actual authorized reader returns the full accepted comment. The same running task and run remain intact, with no replacement or accepted-result replay. - The OpenCode native backend sends a real MCP finish rejection to the controlled provider process. The process follows the returned reader instruction, reads every page, and resubmits finish. One provider turn produces one semantic result proposal. The terminal snapshot fence still rejects a later comment edit. - All 98 affected OpenCode driver and MCP bridge cases pass with the fixture-mode addition. - The first tool-sequence attempt failed on a fixture-only relative path before starting the provider. The test now uses its module URL. This setup failure is separate from the four causal failures. - The first recursive typecheck found missing local workspace links in the clone's excluded sandbox-provider directories. Existing dependency links were copied into the isolated clone. No lockfile or source change was made for this setup repair. - At `c57e532a87`, all 1255 cases in the six-file affected server family, recursive Node 24 typecheck and the full build pass. At production commit `1ae9f54107e1762bc8222572b365a013d4572d1a`, all 1,257 affected cases, recursive Node 24 typecheck and the full build pass. - Greptile found that the initial correction could restart a partial read after page two. On `c57e532a87`, following that correction fails with the actual reader cursor-order error. The follow-up continues the caller's most recent cursor and retains the changed-snapshot restart path. All nine new cases pass, including this regression, a stale-partial control, the foreign receipt refusal and the actual same-turn MCP sequence. - [Linux server3/12 job](https://github.com/paperclipai/paperclip/actions/runs/41788671370/job/126437378334) found five failures in the older format-feedback SQL fixture. Its flat heartbeat row did not match the real joined projection. The same five failures reproduced locally. The test-only follow-up returns a coherent run/agent/task projection and keeps the real resolver and formatter assertions. All 14 format and real-PG/MCP cases pass, and the generated server test layer typechecks. Production code and the earlier 1,257-case/typecheck/build proof are unchanged. - Qualification of the shared-code change found 49 failures because whole-module mocks omitted the new exports. The three mocks now pass through actual module exports and keep their controlled reader methods. Literal error assertions remain independent. The final 1,284-case group passes with the exact 35 opt-in skips described above. No production fallback or substitute code was added. - [One Linux Runner package attempt](https://github.com/paperclipai/paperclip/actions/runs/41792315273/job/126448500037) passed 4,913 cases and skipped 10, but its synthetic post-rename credential durability case failed with ENOENT. The complete Runner package tree and lockfile are identical to the [earlier Linux control](https://github.com/paperclipai/paperclip/actions/runs/41790199252/job/126441739681), where all 42 credential cases passed on Node 24.22.2. The exact case also passes once on both base and head on macOS with Node 24.22.2. The original failure cause remains unverified; OS/load and shared-test timing uncertainty remain. Credential code and assertions are unchanged. The single affected-job rerun passes, including all 42 credential tests. The original failure remains unexplained. - An ordinary full local test attempt was stopped before the review repair. It has no completion summary and is INCOMPLETE. Current-head full CI completed; no local full-suite pass is claimed. Greptile reviewed the final published tip at 5/5 with zero open threads. All current-tip checks pass: 54 successful check runs, one successful legacy commit status, 3 skipped and 2 neutral, with no pending or failing checks. Paperclip also reviewed final tip `37bfb26943` at 5/5 with no remaining findings. Its prior 4/5 code-ownership recommendation is fixed and resolved. No review nudge or hosted task retry was issued. The controlled OpenCode process uses the supported v1 factory envelope to exercise the common native feedback and MCP boundary. It does not call a live model or qualify a provider pack. PostgreSQL, authorization, pagination and receipt checks are real. No hosted task, model, permission or saved result was changed. Recovery of an already accepted historical run remains unverified. ## Risks - Each finish call adds an exact binding query. A mandatory read also validates the current accepted-comment snapshot through the existing transaction. Cost grows with those exact comment IDs, bodies and attachment metadata; it does not scan unrelated task history or add idle polling. - A valid read can become stale before the provider settles. The unchanged terminal guard still rejects that race. - Missing ownership or unexpected validation errors remain refusals. This change does not grant replay authority or turn an accepted historical result into fresh work. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window. The coordinator performed independent read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:19:12 -06:00
DottaandPaperclip	d8c1b9d4b5	fix(runtime): reclaim failed remote checkpoint retries (#18877 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must preserve the original execution and provider state. > - A failed remote run can retain its provider PID after heartbeat releases the task execution lock. > - Normal resumption excludes that PID, while restart recovery rejects the released lock. > - Recovery must reclaim that exact execution only after current admission and durable ownership checks pass. > - Failed retries and admission holds need separate finite budgets. Neither can prove a provider stopped. ## Linked Issues or Issue Description What happened? A failed remote native run remains in retryable failure without a task execution owner. Normal resumption and restart recovery both reject it before inspecting the preserved remote provider. The current remote workspace reference format also misses the due-retry routing path. Expected behavior A due exact-checkpoint retry reclaims its original task execution only when current ownership and safety gates permit it. Newer task input prevents an old claim. Lost or ambiguous provider state still prevents replay or replacement. Ineligible recovery eventually becomes a quiet manual reconciliation obligation. Steps to reproduce 1. Start a remote native run and persist its provider checkpoint and encrypted Runner authority. 2. Fail with exact-checkpoint recovery and a retained remote PID. 3. Release the task execution lock at heartbeat completion. 4. Run ordinary resumption and restart recovery after the retry deadline. 5. Observe that both reject the same eligible run before remote inspection. Paperclip version or commit Reproduced from source at `d59791828482`; integrated with upstream `db9be4c5a6`. Deployment mode Remote workspaces with PostgreSQL. Local regressions use disposable PostgreSQL and encrypted checkpoint persistence. Related public repairs #18846, #18854, #18860 and #18841 cover finished per-turn owners, pre-provider failures, unstarted follow-ups and storage migration. They do not reclaim this failed run's released execution lock. Merged #18879 also adds scheduled-resume lock retention and reclaim. This PR keeps that compatibility and requires encrypted Runner authority, canonical source ordering and a charged failed retry before reclaim. Its scheduled-resume regression now uses a real encrypted authority and retains a snapshot-only refusal control. ## What Changed - Share normal admission checks and one guarded execution-binding restore/event helper between normal and remote resumption. - Keep agent-name normalization in a module with no imports. Preserve heartbeat's existing export. Native admission no longer loads heartbeat orchestration for a string key. - Recheck company, assignment, agent eligibility, Stop, pause, approval, dependencies, execution and checkout ownership before reclaim. - Reject newer task input using PostgreSQL creation-time and ID ordering, preserving timestamp precision. - Require the recorded remote target, encrypted Runner authority, exact checkpoint identity and existing controller fence. - Commit task binding, controller lease, run status and recovery event atomically. Recognize both remote workspace reference formats. - Charge failed reattachments against the existing three-provider-attempt budget, including retained-lock crash windows. Active controller adoption preserves its charged attempt. - Use the existing five-retry/30-minute system-recovery policy for admission holds. Ended tasks and newer sources exhaust it immediately. Preserve PID, checkpoint and task bindings, suppress resource settlement, and assign manual reconciliation with executor stop/state explicitly unverified. - Register recovery intent before deadline writes; preserve live/future deadlines quietly and roll back zero-row writes. Record changed hold reasons once and terminal dispositions separately. - Keep historical provider-budget exhaustion behind current admission/source/controller/authority checks. Make an existing orphan-cleanup fixture explicitly due. - Include exact plugin task leases that have no workspace reference. Reuse every reclaim and retry-budget guard, with a bounded company/task/run lease projection. - Share strict manual ownership proof across stale-lock cleanup, adoption, queue release and wake admission. Require termination receipts for every exact remote lease before normal cleanup can release either binding. - Preserve governing Stop through the existing audited cancellation path. Its acknowledgement does not release an unverified provider. Ordinary Retry can wait for exact Stop and file preservation, then queue one continuation. ## Verification - Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its [current CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858) is terminal and clean: 54 successful, three skipped and two neutral contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on this exact head with no open findings. No fresh Paperclip 5/5 assessment is claimed. - The [prior complete CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388) passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and Paperclip both 5/5 and no open threads. - All 233 tests across the six affected heartbeat, restart-isolation, remote-admission, scheduled-resume and session-binding families pass on production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and plugin task leases, competing controllers, exact rollback, Stop, assignment/company boundaries, attempts, work-intent ordering, finite backoff and quiet held state. - The merge preserves #18931's per-run audit-failure isolation and this PR's unchanged-transition hold. Two initial composition cases failed because their setup or expected result described the older contract. The rollback test now expects the exact per-run deferral while retaining full coordinator equality and no work intent. The routine fixture writes a real encrypted checkpoint through the normal owner before reaching the actual routine unique constraint. No production admission guard or rollback assertion was weakened. A separate agent reviewed this exact merge and fixture delta without finding a blocker. - Full recursive `pnpm -r typecheck` and `pnpm build` pass on production merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits change only three test files. Both the two-file `a5104fa` and final `5d7baf` server test-layer typechecks pass. - Four production-identical `a5104fa` probes pass with the rebuilt actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL checkpoints under Node 24.22.2. Workspace-reference and plugin-lease admission each cover surviving and lost providers. Replacement preserves one provider launch, one original side effect, the charged attempt and restored task execution binding. Lost providers reject snapshot recovery without replacement. This is protocol-boundary evidence, not a live Claude or hosted provisioning claim. - Initial `a5104fa` CI found one additional routine integration fixture missing the persisted `exact_checkpoint_resume` mode. Its encrypted checkpoint was already present. The exact case fails locally before repair and passes after recording that mode and expecting the charged provider attempt. Every exact routine conflict, rollback, checkpoint, task binding, successor-count and sibling-startup assertion remains. Final head changes only that fixture. The correct native-server group passes all nine files: 175 executed cases and 14 container-only skips, matching the now-successful native-server CI gate. - The earlier canonical HTTP Retry acceptance passed on the prior reviewed head with real PostgreSQL, the recovery worker and real files. Pending Stop protects both bindings and starts no successor. Exact termination and copyback queue one preserved-workspace continuation. No browser or live-model outcome is inferred from that test. - The predecessor's full local `pnpm test:run` failed in an unchanged Discord Gateway fixture with a 200 ms lease and a 422 initial ownership error. Focused predecessor and untouched-base controls pass; that full-run cause remains unproved. The required command was also started on production-identical `a5104fa`. All six local chat shards passed, covering 1,850 cases. The duplicate local invocation was interrupted with SIGINT while the general-server group was still running after this head's complete CI and the combined repair qualification had passed. The wrapper exited 1 on cancellation, no process retained its test log, and no full local pass is claimed. An earlier attempt to select the native lane through the CI wrapper instead selected local package sharding: 2,631 cases passed and 13 failed. Those separate failures are retained without causal attribution or unrelated source changes; that run is not counted as native-server qualification. The direct native group and current complete CI matrix pass. - Live Claude Opus 5.5 continuation, hosted plugin provisioning, deployment and recovery of historical tasks remain unverified. No original task was retried or mutated for this verification. ## Risks - Restoring a task binding grants remote inspection; it does not prove that the provider survived. - Unknown effects, lost providers and missing semantic results retain their existing reconciliation requirements. - Exhausted admission holds preserve execution bindings and resource evidence for manual reconciliation. They do not authorize a fresh executor or provider/resource release. - The source guard is conservative: newer task input prevents recovery of an older failed source. - No database migration or provider profile change is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and shell execution. The runtime does not expose the exact model snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:07 -06:00
DottaandPaperclip	84aa989bf9	Preserve native sessions through legacy workspace enrollment (#18943 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 04:43:56 -06:00
DottaandPaperclip	612d9931a6	Support per-turn Runner sessions on ephemeral web servers (#18932 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. What existing behavior does this improve? Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. Current behavior The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. Proposed behavior Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. Reason and benefit Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. Breaking changes The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The `e9aae9cf` revision passed all 57 successful checks, with 5 skips and 2 neutral checks, and exact-head Greptile 5/5. The latest `9cbec30a4f` follow-up changes only E2E catalog/admission controls; its 291 affected tests pass, all 57 check contexts succeeded (5 skips, 2 neutral), and exact-head Greptile is 5/5 with no open review threads. - Migration numbering and safety checks pass. - A local real PostgreSQL/S3 roundtrip preserved 2,870,101,591 payload bytes across 89,100 archive entries after deleting the source directory. Restored SHA-256 matched; upload/readback took 56.5 seconds, restore 74.8 seconds, process peak RSS was 1.40 GB, and all fixture resources were removed. - Archives preserve contained relative links, reject link chains/traversal, and enforce independent byte/entry/object limits. Offline verification streams hashes. Orphaned deleted-run backups stay on the original volume. Obsolete object retention preserves the selected backup, live lease references, and pending uploads with owners. - Isolated API-key Codex deployment: original active runner/session transferred to the replacement and committed. All 567 reads, 57 writes, and 57 readbacks succeeded. A second replacement completed a same-session continuation with all 287 reads and 29 writes succeeding. The strict cell failed two agent-authored shell commands; this result is retained as failed. - Isolated API-key Claude deployment: original active runner/session transferred and committed. All 584 reads, 59 writes, and 59 readbacks succeeded. A second replacement completed a same-provider-session continuation, with 211 reads and 22 writes succeeding. The strict cell failed an initial completion call that omitted a registered attachment citation; the agent corrected it. This result is retained as failed. - Isolated API-key OpenCode deployment: the same active runner/session transferred and committed. All 481 reads, 49 writes, and 49 readbacks succeeded. A second replacement retained the provider session and completed useful work, with 156 reads and 16 writes succeeding. The strict cell failed agent tool calls: a denied external-directory write, malformed artifact hashes, and a second completion after the first result was committed. There was one submitted/completed turn and one committed result; reconnect notification replay was identical. This result is retained as failed. - Grok's real first turn now completes with settled usage and provider cost ($0.41397); the next turn is admitted normally. That attempt then exposed a shared workspace-path identity regression before deployment. Three cold-restore cases reproduced it before the fix; 106 workspace/transport tests and four focused cold-restore cases pass afterward, including destroyed allocation and occupied-directory preservation. Server typecheck and build pass. At `b5b0dbb03f` both controller switches passed the read/write probe without failure, and the active runner retained its process while the controller generation advanced. The third turn exposed an invalid reasoning-token subset check; `ad07e681ba` fixes it with the actual receipt regression. Two later live attempts failed with an upstream Grok rate limit before deployment. A tiny preflight prompt succeeded before the second attempt, but the full workload was still rate-limited. The final live gate remains incomplete. No accounting or identity guard was bypassed. Earlier failed attempts and their exact resource-cleanup receipts remain recorded. - Earlier Codex/Claude/OpenCode receipts predate #18922; they establish their tested revisions. Current-head live qualification remains pending; #18943 separately preserves pre-enrollment logical/provider workspace addresses and adds actual legacy migration proof. Its first Codex migration attempt stopped safely at skill-version verification. - Earlier attempts remain recorded: missing local provider-pack setup blocked one Claude launch, and the observer rejected identical reconnect replay in another. Setup and the observer were corrected before the latest run. - Named staging enrollment has not started. Its serving revision and post-deployment journeys remain separate delivery gates. - Broader E2E harness validation exposed deployment-catalog and installed-lane coverage drift; the affected 291 controls now pass. Twelve other failures reproduce on unchanged base `1a39dea9aa`. They concern cleanup call expectations, dated operating-overhead fixtures, a Pi profile-version expectation and workflow-security expectations. No baseline failure is reported as passing. ## Risks - Provider archives can contain credentials. Objects use authenticated encryption with scoped associated data and a wrapped data key. The original stable master key remains required. - Storage failure prevents publication and sandbox cleanup. Operators must keep shared storage available and preserve the previous volume during enrollment. - Legacy state without verified identity cannot become executable recovery authority. Archival preserves evidence and existing failures; it does not recover already-lost provider files or clear blocked tasks. - Local overlap probes establish the tested application behavior. They do not establish Railway routing behavior or live managed-subscription qualification. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and live integration tests. The exact runtime variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 02:12:51 -06:00
DottaandPaperclip	e2dc2cc0be	fix(native): retire idle shared-sandbox ingress owners before dispatch (#18893 ) ## Thinking Path > - Paperclip manages agents and their tasks. > - Native tasks can share one reusable sandbox workspace. > - A completed child can keep its provider process warm. > - A parent has its own logical session, but the sandbox has one ingress listener. > - Retirement by task identity leaves the child's listener in place. > - A new parent Runner then fails to bind that listener. > - This pull request retires a proven idle physical owner before the parent's distinct session starts. ## Linked Issues or Issue Description Refs #15353 and #17097 for listener configuration and warm transport rotation. Refs #18682 for owned process retirement. #18860 repairs a different resume-source failure. Searches found no existing repair for retained warm sessions on different tasks that share the same physical ingress allocation. What happened? A completed child and a later parent can use the same reusable sandbox. Warm-session retirement compares logical task ownership. It does not find the child's distinct task scope. The child keeps the fixed ingress listener, and the parent's new process fails with `runner_ingress_bind_conflict` / address already in use. Expected behavior A parent must start in its own logical session after the proven idle prior listener closes. Active, foreign or unverified owners must remain protected. Steps to reproduce Run the `retires the completed child listener` cases in `native-session-executor.test.ts`. The actual executor registers a warm child. PostgreSQL holds its committed, accounted result and clean expired lease. A distinct parent receives the same allocation and current workspace writer. The provider boundary uses real TCP listeners. The original executor reaches the second bind with the child still listening and fails with EADDRINUSE. The changed executor waits for the owned close, then starts the distinct parent session. Paperclip version or commit Reproduced on `ab2d2098a2`. The branch includes current master `44a4ca1c61` before this PR was opened. Deployment mode Native Runner with provider ingress on a reusable sandbox workspace. Local regression uses real PostgreSQL and local TCP with a controlled provider-process boundary. No hosted provider call was made. ## What Changed - Keep physical ingress identity separate from logical task/session ownership. Never attach the parent to the child's session. - Reserve a physical allocation during local admission. Retire a distinct warm owner only after canonical company, agent, workspace, provider, allocation, lease, task and coordinator checks pass. - Require the current remote workspace writer and location to match the admitted run. Require the prior run to be committed and accounted, with its original controller generation and a clean released or expired lease. - Use the existing close, checkpoint and stopped-file collection path. Wait for close before parent dispatch, then recheck canonical authority before staging files or calling the parent backend. Retain the entry when close is uncertain. - Add actual executor/PostgreSQL/TCP safety cases and document the contract. Keep maps unchanged for this bug fix. ## Verification - Untouched production at `ab2d2098`: actual executor + PostgreSQL + TCP fails EADDRINUSE. Same positive case passes after the change. - Final focused suite: 20 passed. It covers distinct parent session identity; foreign company, allocation and workspace; active or unaccounted owner; changed controller; reassignment; competing lease/writer; stale allocation metadata; Stop before and after claim; workspace-lock race; failed close that leaves the listener and accepted result intact; and controller transfer, Stop or writer change during awaited close. These last cases reject the stale parent before backend invocation after the old listener closes. - Untouched head3425acbe: transferring the incoming coordinator during close still reaches the parent backend. The same actual executor/PostgreSQL/TCP case passes after the post-close authority check. - Broader `native warm session supervision` suite: 160 passed. This run preceded the final two-line logical-session collision guard; the final 20-case suite includes that guard. - `pnpm --filter @paperclipai/server typecheck`: all source and test layers passed, including `src/__tests__`. - `pnpm --filter @paperclipai/server build`: production server and Runner build passed at current head572acfef79. The exact-head build stamp is recorded. - The provider/close boundary is controlled. The fixture seeds canonical committed/accounted finalization data in PostgreSQL. These tests do not claim a live model result, operator ACK, hosted PID-to-listener identity, or recovery of a historical task. - Full repository pnpm checks were not run locally for this server-only slice. Required current-head CI and reviews are pending. ## Risks The new retirement path can close a process from another task. It requires the same canonical company, agent, allocation and workspace, a current exclusive writer, and a completed idle owner. Missing or changed evidence leaves admission blocked. Active owners, changed generations, reassignment and cancellation remain fenced. A failed close retains the owner and saved result. This covers current-controller warm entries with recorded physical identity. Historical processes outside the registry keep the existing exact-owner recovery requirements. No port shuffle, port kill, task capability expansion, new polling loop or invented receipt is added. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex, with source editing, terminal execution and tool use. The host did not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 19:44:12 -06:00
Nicky LeachandClaude	8fd917aedf	fix(native): accept late provider notices in recovery (#18890 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep a provider conversation across task turns. > - When a later run gets a new task lease, recovery must prove that the prior run settled. > - Recovery required `run.terminal` to be the last committed event. > - A provider can send a late notification for a settled turn. The Runner then records a `provider.notice.recorded` warning after `run.terminal`. > - This false denial stranded a clean session. The notice stays in the journal, so each later run on that session meets the same denial. > - This pull request lets recovery accept a recoverable notice after the terminal event. All other new events still block recovery. > - The benefit is that a late warning no longer ends session continuity. A denial now also names the check that failed. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for the error code, the event name and the function name. I found no duplicate. What happened? A run failed in about nine seconds with `runner_state_identity_mismatch: plugin_checkpoint_recovery_unproven`. No provider work started. The prior run had finished and was committed as succeeded. Its journal had one `provider.notice.recorded` warning, "ignored a notification for a settled Codex turn", 1 ms after `run.terminal`. A second session had the same warning about 61 seconds after a normally completed turn. Expected behavior Recovery accepts the session and continues the provider conversation. A recoverable notice does not start work, so it must not block recovery. Steps to reproduce 1. Start a warm native session in a plugin sandbox environment. 2. Finish a run so that a provider notice follows `run.terminal` in the journal. 3. Wait until the idle timeout stops the Runner task and releases the lease. 4. Start the next run on that session. It gets a new task lease. 5. The run fails with the error above. The notice stays in the journal, so a retry on the same session meets the same denial. The `local-receipt-late-notice` test mode in `native-session-executor.test.ts` reproduces this failure on the unfixed code. Paperclip version or commit Observed on `d8a9b3eb0c`. The same check is on `master` before this change. Deployment mode Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - `isSettledRemoteWarmControlState` accepts a `provider.notice.recorded` event after the terminal event. The notice must be recoverable and have `info` or `warning` severity. It must come from the same runner, run, session and lease. - Error notices, unrecoverable notices, notices from another source and new work still block recovery. - Checkpoint recovery and local quiescent recovery now use this function. They no longer have separate checks that require `run.terminal` to be the last event. - Each checkpoint recovery denial adds a specific reason after the existing code, for example `plugin_checkpoint_recovery_unproven: journal_not_settled`. The `runner_state_identity_mismatch: ` prefix does not change. - I reworded the existing comment on `isSettledRemoteWarmControlState` so that it stays correct. I added no new code comments. ## Verification - Failure before the fix: I added the tests first and ran them on the unfixed code. 38 tests failed. In the 4 `local-receipt-late-notice` cases, checkpoint recovery rejected with `runner_state_identity_mismatch`. In the `late provider notice` case, quiescent recovery rejected with `quarantine_owner_unverified`. The new unit test for notices also failed. The other 32 failures were assertions for the new denial reasons. - Targeted tests after the fix: 134 tests passed, with 987 skipped by the filter. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence\|recovers a blocked\|automatically recovers only a proven settled local session"` - Type error found in review: Vitest does not check types, and my first version passed these tests. Review of the diff showed an incompatible `identity` parameter type. I changed that type to `Record<string, unknown>`. - After the type change: `remote-warm-session-recovery.test.ts` passed (23 tests) and `pnpm run typecheck` in `server` passed. I did not run the 134 tests again, because this change affected only types. - Real data, read-only: I ran the new function on 285 session journals from a test deployment. The result changed for 2 journals. These are the 2 stranded sessions, and each has only provider notices after `run.terminal`. - Not run locally: I did not run the full test suite. I stopped a run of the wider native-runtime test files before it reported a result. CI provides the full regression coverage. - Not verified: no live run has reached `read_state` on a stranded session after this fix. The tests use a fixture for that call. ## Risks - Stricter identity check: checkpoint and quiescent recovery now also require the journal's saved identity to match. Recovery reads this identity from the same file, so only damaged state should fail this check. - Wider goal snapshot rule: these two paths now accept idle goal snapshots after the terminal event. Warm handoff already used this rule. - Wider notice rule: warm handoff and the settled projection check also accept recoverable notices now. A notice does not start work, and the checks for pending commands and unanswered tool input stay in place. - Longer error text: code that compares the full `plugin_checkpoint_recovery_unproven` message for equality would stop matching. I found no other reference to this code in the repository. - Stranded sessions: these sessions need no data repair. They can still fail with `checkpoint_not_settled` if the sandbox checkpoint is not clean. - Migrations: this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the diagnosis and approved the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-03 16:37:39 -08:00
Devin FoleyandPaperclip	4e29f00139	fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. What happened? A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. Expected behavior A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. Steps to reproduce 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. Paperclip version or commit `master` at `b0c4e345267`. Deployment mode Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - Upgrade failure. `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - Silent stall. The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - Monitor. `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - Candidate admission carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - The agent that loads the conversation stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - In-flight provider work is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - Launch shape. A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - Artifacts are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press Retry once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 16:30:58 -08:00
Devin FoleyandPaperclip	909d86b59a	fix(native): retry runs that fail before provider work (#18854 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start the provider through Runner. `session.open` starts the provider session before any prompt. > - When `session.open` fails, the server records the generic `native_session_interrupted`. Recovery evidence then reports `ambiguous_state`, so the first failure blocks the task with no automatic retry. > - The failure is often temporary. On one instance an agent's provider account hit its usage limit. For about 14 hours every later run failed at `session.open`, and 47 tasks stayed blocked. Outside limit windows, the same error also occurs once and a later retry succeeds. > - The server already has a bounded same-run bootstrap retry for failures that are proven to come before provider work. It did not recognize this failure, and it did not consider a known provider limit. > - This pull request classifies a definite `session.open` failure as a pre-session failure. Bootstrap retries back off and wait out a known provider limit. The task is blocked only after the shared budget is used, with a clear reason. > - The benefit is that a temporary startup failure or a provider limit no longer strands tasks on the first failure, and every existing replay guard stays in force. ## Linked Issues or Issue Description Refs #18847. That pull request keeps the provider cause on `session.open` rejections and names typed categories (for example `classification=provider_rate_limit`). This change does not depend on it. With #18847, a typed limit at `session.open` uses the same cooldown path, because the server reads the cause with `providerFailureFromError`. Refs #18622, which added the `ProviderFailure` cause. Refs #13993, an open change about legacy adapter startup backoff; it does not touch the native recovery path. What happened? A native run (`paperclip_runner`) failed at `session.open` before any prompt: `ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)`. The server recorded `native_session_interrupted`. Recovery evidence had no checkpoint and no provider event, but the cause was not in the pre-session list, so the mode was `ambiguous_state`. The coordinator went to `terminal_failure`, a board-owned `active_run_watchdog` action was created, and the task was set to `blocked` on the first failure. During a provider usage limit, every later run of the agent failed the same way, so each task was blocked. Expected behavior A run that provably failed before provider work retries automatically, with backoff and a bounded budget. If the agent has a known provider limit, the retry waits for the cooldown. The task is blocked only when the budget is used, and the recovery action explains why. A run where provider work may have started is never replayed. Steps to reproduce 1. Make Runner reject `session.open` for a native run (for example a provider at its usage limit). 2. Observe the coordinator: `terminal_failure`, `recoveryMode: "ambiguous_state"`, `providerSessionEstablished: false`. The task is `blocked`. 3. The new `native-startup-retry.integration.test.ts` reproduces this with a real database and the executor. Paperclip version or commit `master` at `5877d751c0e`. Deployment mode Native Runner with an ACPX Claude provider. The fix is in the server and is not specific to one deployment mode. ## What Changed - `nativeSessionFailureSourceCode` returns a new cause, `provider_session_open_rejected`, when Runner reports that `session.open` failed or was rejected. Those outcomes are definite, and the command runs before any prompt. An `indeterminate` outcome stays `native_session_interrupted`. Earlier, more specific causes (for example a missing runtime package) keep their classification. - `nativeProviderRecoveryEvidence` treats the new cause as pre-session. The durable evidence still decides. A bootstrap retry happens only with no provider session, no provider event and no checkpoint. Otherwise the mode stays `ambiguous_state` and the task is blocked as before. - New module `native-runtime/native-startup-retry.ts` plans each bootstrap retry. It is called only for `bootstrap_retry` mode. - Backoff: the existing `infrastructureRetryDelayMs` (about 30 s, then about 60 s, plus or minus 20 %). The jitter derives from the run and attempt, and it is persisted with `nextAttemptAt`. - Limit cooldown: evidence is a `rate_limit` or `quota_exhausted` provider failure on the current failure (the #18847 shape), or on an agent run that started in the last 24 hours. A `native_provider_usage_limit` or `provider_quota` error code also counts. The cooldown is the provider's `retryAt` (or `retryNotBefore`), or one hour after the failure when no reset time is known. A later successful run of the agent clears older evidence. The cooldown is capped at 24 hours. The retry runs after the cooldown plus the backoff delay. - A failed evidence read is logged and falls back to plain backoff, so it cannot stop the failure from settling. - The budget does not change: three provider attempts, shared with checkpoint resumes and replacements. When it is used, the coordinator records `native_session_retry_exhausted`. The task is blocked, and the board-owned action says: "Provider startup failed on 3 attempts before any provider work began. Automatic retries are stopped." When a limit is known, the message includes the expected reset time. - While a retry waits, the coordinator detail, the recovery action evidence and the scheduled wake record `providerCooldownAt`. The next action names the reset time, and the task projection shows "Retry scheduled". - `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour) moves to `system-recovery-state.ts`. The legacy provider-quota recovery default now refers to it. The value does not change. - A chat retry of an exhausted `provider_session_open_rejected` run is accepted like the other pre-provider causes. The existing proofs still apply. - `doc/execution-semantics.md` describes the backoff, the cooldown and the exhausted state. ## Verification - Red: before the fix, the first new integration case failed. The coordinator was `terminal_failure` with `recoveryMode: "ambiguous_state"`, `originalFailureCode: "native_session_interrupted"` and `providerSessionEstablished: false`, and the task was `blocked`. Four more cases failed the same way. The two never-retry cases passed before and after the fix. - Green: `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` in `server` (embedded PostgreSQL, the real executor, and the real resume claim). 8 cases pass: - a rejected `session.open` is retried with backoff (attempt 1 in 24 to 36 s, attempt 2 in 48 to 72 s), and the task stays `in_progress` with an agent-owned action and a `resume_native_run` wake; - after three attempts the task is blocked, the action is board-owned with the clear reason, and the resume claim refuses another attempt; - a recent limit failure holds the retry until one hour after that failure, and the claim refuses it before the cooldown; - a provider reset time from an earlier run, or from the startup failure itself, sets the cooldown; - a distant reset time holds the retry for at most one day; - a later success, another agent's limit, and an expired limit do not delay the retry; - a durable provider event, or an indeterminate `session.open` outcome, still blocks the task with `ambiguous_state` and no retry. - `native-session-executor.test.ts`: all 1,087 tests pass, including a new classification case for `failed`, `rejected`, `indeterminate` and a different command. - `chat-channels.integration.test.ts -t "requires terminal-safe native ownership for an exact chat retry"`: 23 pass, including the new `session_open` case. That case fails without the list change. - Related suites pass: native session resumption, native provider failure, capacity and overload integration tests, provider failure classification, native status arbiter corpus, native restart recovery. - `npx tsc --noEmit -p server` passes. `git diff --check` passes. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41713988964) is green on head `32561aa335d`. This includes `ci / verify`, `ci / e2e`, typecheck, build, all server, chat and workspace groups, all serialized server suites, and all Runner checks, which run the native-runtime server tests. In the first attempt, eleven jobs stopped because their runners received a shutdown signal; no test reported a failure. A rerun of those jobs passed. - A local run of the whole `server/src/services/native-runtime` directory was stopped before completion because it runs serially for a long time. CI ran those files. - Runtime dependency: the new work is one indexed query (`company_id`, `agent_id`, `started_at`) with `LIMIT 20`. It runs only when a bootstrap retry is planned or exhausted, not on the normal path. No new timer or polling: the wait uses the existing persisted `nextAttemptAt` deadline and the existing resume dispatch. - Not verified: a live provider account at its usage limit, and a live remote sandbox that waits through a full cooldown. The full repository suite was not run locally; CI ran it. ## Risks - A task that waits for a cooldown stays `in_progress` with its execution lock, for up to one day. This is the existing same-run retry state. Stop, reassignment, pause and an explicit Retry keep their current behavior through the existing claim checks. - During the wait the failed run keeps its environment lease in the same way as any same-run retry. A remote lease with an unexported prepared workspace is retained by the orphaned-lease sweep until the retry runs. Other active leases of the failed run are released after the usual five minutes, and the retry acquires a new one. - A `rate_limit` failure without a reset time (for example a short HTTP 429) delays only startup-failure retries of that agent, by up to one hour. Normal runs are not delayed, and a later success clears the evidence. - A startup failure that needs user action (for example a sign-in failure) now uses two automatic retries, about 90 seconds in total, before the task is blocked. Before, it was blocked at once with an ambiguous reason. - The agent's other queued runs are not held during a cooldown. Each one fails at `session.open` without provider work and then waits for the same cooldown. An agent-wide admission hold has no clean extension point in the current queue, so it is not part of this change. - Re-arming consumed scheduled monitors is out of scope. This change makes it matter less: a run that consumed a monitor wake now retries itself on the same run. A monitor that a run consumed before its retry budget was used stays consumed. - No schema or API change. The new error code appears on runs and in run error codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with extended reasoning and tool use (shell, file editing, test execution). The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:24:59 -08:00
Devin FoleyandPaperclip	80c308f872	fix(runner): admit follow-ups after recovered per-turn owners (#18846 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one conversation across task turns. > - After a per-turn sandbox turn, the controller saves a Runner suspension receipt or a failover backup. > - A full sandbox disk can stop that save after the turn is complete. > - Recovery then commits the result, but the controller has no proof that the Runner suspended. > - Each follow-up rebuilds the old controller journal from the database and rejects it again. > - This pull request lets admission retire that old authority when the prior owner is provably finished. > - The benefit is that the task continues with its saved workspace and does not stay blocked. ## Linked Issues or Issue Description Refs #18705 (checkpoint save on a full sandbox disk). Refs #18829, #18718 and #18343 (related admission and controller recovery). Refs #18811 (a related warm continuation wait). What happened? A per-turn native run in a remote sandbox finished its turn. The post-turn `harness_state.backup.persist` and `session.checkpoint.persist` steps failed because the sandbox disk was full. The controller got no Runner suspension receipt and no failover backup. Workspace recovery later saved the files. The run committed its result and ended as `succeeded`. A board comment then started a follow-up in a new sandbox. Admission failed after about two minutes with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Execution recovery then blocked the task. The cause is in durable-root admission. `restorePerTurnRunnerdController` rebuilds the prior controller journal from the database checkpoint. `verifyPriorRunnerdStateForSessionScope` finds a terminal owner in the same scope with no suspended lifecycle and no retained warm owner. It returns `terminal_state_indeterminate`. Admission quarantines the root and fails. The database checkpoint does not change, so every later admission rebuilds the same journal and fails again. A board Retry after that failed follow-up does not repeat the same error. Runtime selection picks the failed follow-up as the resume source, because it carries an inherited provider checkpoint. Controller recovery then fails with `native_controller_recovery_unsupported:metadata_format`. The task stays blocked. This pull request prevents that failed follow-up. It does not change source selection (see Risks). Expected behavior When the prior owner is provably finished, the follow-up must start. It must keep the saved workspace. It must not start while the prior owner can still be alive, while terminal delivery or provider cleanup is unsettled, or while quarantine holds evidence from a different owner. Steps to reproduce 1. Complete a per-turn remote sandbox run. Commit its result through recovery and release its controller lease. 2. Keep its database controller checkpoint and recovery pointer. Do not save a suspension receipt or a failover backup. 3. Admit a follow-up in a new sandbox with no local controller projection. 4. Admission rebuilds the old journal, quarantines it, and fails with the error above. 5. Admit the same run again. It rebuilds the journal and fails the same way. The new regression test runs these steps against PostgreSQL. Paperclip version or commit Reproduced on `e0276e0166`. Deployment mode Native Runner with a per-turn lifecycle, the built-in remote sandbox transport, and the ACPX provider. The admission path is the same for other per-turn sandbox providers. ## What Changed - Add `retireFinishedPerTurnRunnerdOwner` to durable-root admission in `native-session-executor.ts`. It runs only when the existing verifier returns `terminal_state_indeterminate`, and only without a restart claim or a retained warm owner. - The function requires all of these. Otherwise the existing quarantine and rejection apply. - Per-turn lifecycle on the built-in remote sandbox transport, with a `created` or `replacement` sandbox. A resumed sandbox, a warm session, and a plugin task keep their existing paths. - No local Runner lifecycle evidence for the old owner. The session root contains only known authority entries. - A journal for that owner with its terminal delivered and only idle goal snapshots after it. All commands are completed. Each tool input has a completed result. No warm transition or staged attachment exists. - A terminal prior run in the same task. Its coordinator is `committed` with a result, has no lease owner, and has no recovery state. - A stopped prior controller, from `evaluateNativeControllerTakeover` with `requireStoppedController`. A durable controller is fenced by its released database lease, as in remote warm recovery. - Each quarantined root for the scope passes the same journal and local Runner evidence checks for the same owner. A retry rebuilds only the database journal, so quarantined Runner evidence must stay decisive. - When the proof holds, the function reads the journal again, moves the old authority into `continuity-breaks/<time>-<uuid>` in the session root, and logs one line. The follow-up then starts a new Runner authority. The provider session follows the existing resume-or-replace policy. - Share the list of continuity archive entries with provider replacement (`archiveContinuityState`). Provider replacement behavior does not change. - Add `native-recovered-owner-followup.test.ts`. It uses real PostgreSQL, a real encrypted controller checkpoint, and the production source selector and checkpoint rebinder. - Document the rule in `doc/execution-semantics.md` and the test in `doc/testing-native-remote-continuity.md`. ### Design choice I considered two approaches: 1. Start a new Runner authority for the scope at admission (this pull request). 2. Make the recovery path that committed the prior run also settle its Runner and controller projection. I chose the first approach for these reasons: - The provider history stayed in the unreachable sandbox. A settled projection cannot make it resumable. Admission would pass, but a replacement sandbox would then fail on the missing backup, and a new sandbox would only try to resume a session that is gone. - A settled receipt written by recovery would claim a suspension that the Runner never wrote. Other checks rely on that receipt as Runner evidence. - Several paths can commit a result. Only runs that commit after a settle-at-commit fix would benefit. The admission fix also unblocks tasks that were committed before it. - Admission already owns this decision. It already has the scope claim, the journal, and the database rows. ### Safety argument - No live prior owner. The coordinator has no lease owner, and the controller must be proven stopped. The follow-up runs in a different sandbox than the old Runner. A resumed sandbox keeps the existing path, which reads the old Runner's own lifecycle there. - No unsettled terminal delivery or effects. The controller acknowledged the terminal and completed every command, including the suspend command. Every tool input has an answer, and the coordinator committed the result. The unreadable Runner outbox can then hold only post-terminal telemetry. - No unsettled cleanup. Any local Runner receipt, including an unsuspended one, keeps the hold. A coordinator recovery state keeps the hold. - No ambiguous quarantine. A quarantined root from a different owner keeps the hold. A quarantined root with Runner lifecycle evidence, unknown entries, or an unsettled journal also keeps the hold. - No lost evidence. The move keeps the old authority in the session root. Existing retention expires it like other continuity breaks. A journal change during the database reads cancels the move. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 12 tests pass. - Before the change, the four admission tests failed with the exact error above. This includes a re-admission that rebuilt the journal from the database and failed again. - After the change, `created` and `replacement` sandboxes are admitted. A re-admission is admitted. A task with an earlier quarantined copy of the same owner is admitted. - Eight controls keep the hold: uncommitted result, live controller lease, unexpired controller lease, recovery state, pending command, unanswered tool input, a different owner in quarantine, and an unsuspended Runner receipt in quarantine after a retry rebuilds the root. - I disabled each new guard in turn. Each change made its matching control fail. - An exploratory run of the same fixture before the change answered the Retry question above. The second follow-up selected the failed follow-up as its source and failed with `native_controller_recovery_unsupported:metadata_format`. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: all 1,087 tests pass. - Related suites pass: `task-workspace-binding`, `native-controller-recovery` (unit and integration), `native-harness-backup-expiration`, `native-session-resume`, and the restart and diskless integration suites. Their container and Runner binary cases skip locally. - `node scripts/typecheck-low-memory.mjs`, `pnpm check:module-boundaries`, and `git diff --check` pass. - CI passes at `879bb6a5c7`. Two attempts lost jobs when CI runners received a shutdown signal; the third attempt passed every job. - I did not run the full `pnpm test:run`, `pnpm build`, or the opt-in Linux Runner container cases locally. CI runs them. - I did not test a live sandbox provider or a live model. I could not inspect the original checkpoint contents. If a stranded run's journal does not show a delivered terminal and completed commands, its follow-up stays held. ## Risks - A follow-up after this path cannot use the lost provider history. The provider session then follows the existing resume-or-replace policy, and the task context comes from Paperclip. A checkpoint that requires the same session, such as an unfinished goal, still fails at provider recovery. - Post-terminal Runner telemetry in the lost outbox, such as a late usage report, stays lost. It cannot carry a workspace effect. - A task that already has a failed follow-up from this error can still be blocked on a board Retry. Source selection picks that failed run and fails with `native_controller_recovery_unsupported:metadata_format`. This pull request does not change source selection. A same-run retry of the failed follow-up is admitted. - The change is limited to per-turn runs on the built-in sandbox transport. Warm sessions, plugin tasks, resumed sandboxes, and restart recovery do not change. - No migration or configuration change is required. ## Model Used - Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`) in Claude Code, with extended thinking, repository tools, shell and code execution, and local test runs. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:24:16 -08:00
DottaandPaperclip	e491b23eb5	fix(runner): retire sandbox suspension receipts after admission (#18829 ) ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Native Runner sessions preserve a conversation across task turns. > - Remote suspension records a local receipt for the stopped Runner identity. > - The built-in sandbox path now writes the same receipt as plugin tasks. > - Admission cleared that receipt only for plugin tasks. > - A later warm turn could reject a clean journal because the old receipt named an earlier run. > - This pull request clears the receipt after successful admission for both sandbox paths. ## Linked Issues or Issue Description Refs #18343 and #18554 for the shared suspension receipt producer. Refs #18718 for the related controller projection refresh. What happened? After a suspended built-in sandbox session resumed, its next warm turn could fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Its journal named the completed resumed run. Its local suspension receipt still named the older run. Expected behavior Successful admission must retire the prior suspension receipt. The next compatible warm turn must reuse the admitted session and provider history once. A failed admission must retain recovery evidence. Steps to reproduce 1. Complete and suspend remote turn A. 2. Resume turn B through the built-in sandbox transport. 3. Keep the controller warm, with full Runner state owned by the sandbox. 4. Admit turn C. The old local receipt conflicts with B's journal identity. Paperclip version or commit Reproduced in the executor regression on base `367f29478d`. Deployment mode Native Runner with the built-in remote sandbox transport and a warm lifecycle. ## What Changed - Clear the local suspension receipt after a remote sandbox session passes provider recovery and durable admission. Apply this to built-in runners and plugin tasks. - Extend the existing three-turn warm admission regression to cover a suspended receipt and remote-only Runner state. - Keep an active-owner control that rejects admission and retains its receipt. Keep the existing foreign-owner, retirement, configuration, plugin, and unchanged-session controls. - Add a real Linux Runner continuity case that suspends A, resumes B, and keeps B warm for C. Register the test fixture's owned local API listener. - Document receipt ownership and the new continuity case. ## Verification - The new behavioral regression failed before the production change. Turn C raised the exact identity mismatch above. The active-owner control passed and retained its receipt. - All 1,087 native executor tests pass after the fix. - Both opt-in Linux process cases pass at the final commit: ordinary warm reuse and suspension followed by warm reuse. They use PostgreSQL, TLS, current-source Rust binaries, and a deterministic Codex fixture. Each preserves one conversation, starts exactly three turns, and reuses the warm Runner and provider on turn C. - Full repository typecheck passes with `node scripts/typecheck-low-memory.mjs`. Full production build passes with `pnpm build`. - All current-head CI checks pass at `a95a6fdfe8`, including the full test shards, browser shards, Rust checks, build, and typecheck. The policy job needed one retry after repository checkout timed out before checks ran. - Greptile and Ponytail each give 5/5 at the final head. There are no open review threads or findings. - The local full `pnpm test:run` attempt passed 351 cases in its first chat shard. The session interruption stopped it during the second shard, so it has no complete local verdict. The complete CI test run passes. - Module boundary checks and `git diff --check` pass. ## Risks - A receipt removed before admission could discard recovery evidence. Removal stays in the successful `onSession` callback. The negative admission control checks retention. - A stale owner must not gain authority. Journal, run, lease, and scope checks remain in place before the callback. - Historical contradictory or quarantined state still requires the existing verified recovery path. This change does not delete those records. - No migration or provider configuration change is required. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose a more specific model ID or context limit. Used reasoning, repository tools, code execution, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:42:21 -06:00
DottaandPaperclip	27d35314be	feat: connect experimental Cursor Cloud to native Runner (#18069 ) ## Thinking Path > - Paperclip helps people manage AI agents at work. > - Native Runner owns execution and task authority. > - Operators need an explicit native Cursor Cloud choice. > - Tools/files need company, task, session and run scope. > - This connects setup and lifecycle to Runner Core. > - Legacy agents retain their selected runtime/history. ## Linked Issues or Issue Description Depends on #18068. Managed-only API placement and API-owned Stop containment are in #18310. Refs #5664, #11803, #17299, and #17471. No duplicate native integration was found. ## What Changed - Add explicit native setup, Test, saved settings and dispatch with no-prompt account/catalog/callback checks. - Scope MCP to company, agent, task, session and run through authenticated HTTPS. Register verified artifacts/work products; preserve unpriced accounting, completion and budgets. - Connect Question/Plan continuation, cold handoff and admitted fresh sessions; recheck waiting admission under the task lock. - Present audited Stop intent while preserving physical containment proof; bound typed Stop metadata. Preserve title edits through preference/navigation readiness. - Select one live/history representation per run to prevent duplicate settled cards. Integrate upstream task/read-state/retry/schema contracts and repair two test fixtures. ## Verification Core #18068 merged normally as `3910352c923a`. Product head `5a60f541c982` has passing repository CI/build/browser checks, including required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile 5/5 review with zero unresolved threads. Two non-required external app checks remain pending: Paperclip Review is queued waiting for its assigned agent; Superagent Security Scan is reviewing the diff. These are not claimed completed. The normal merge endpoint enforces the repository rules; no policy or status override is used. The dependent Top PR is addressing its fresh deferred-Stop review finding. Product `5a60f541c982` passed 270 UI and 20 selected Cloud/native/disk-full tests, UI/server typechecks, tokens and maps. Top’s clean normal merge passed tokens/maps/diff checks; its matrix is pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native retirement and 19 settlement controls, normal Runner build, and runtime 9af build/recursive typecheck. Reproduced loading/shutdown CI fixture failures passed 7/7 browser and 121/121 unit tests, including a broken-unsubscribe negative control. No timeout/oracle was relaxed. The actual-parent restack adopts 75 mainline paths, including schema/permission and generic runtime changes, beyond the prior two fixture repairs and client projection fix. The full difference from hosted `9af1a5de42a2` is therefore larger than four files; backend source also differs. The merged config-read checks match Core391, and the Cursor placement/runtime/relay/selection modules match frozen9af; selected affected regressions pass. Paid proof keeps its actual 9af source/image identity. The separately tested live/history card fix is not claimed deployed on that host. On 9af, original saved settings survived reopening and fresh no-prompt Test authenticated catalog/model/binding; no new Save is claimed. Fresh Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded exact 27 bytes and matched 121 canonical identities in complete S3-served history. First Stop’s strict quiet FAILED on a 10.373-second gap despite independent cancellation/no late artifact. One targeted unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet / maximum gap 4.584 seconds, 67 ordered canonical identities, and separate CANCELLED/artifact-absent status after the original 180-second deadline. The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for the three new provider agents is awaiting authorization, so full grades and the fresh Artifact’s post-cleanup download remain blocked. A historical deleted-provider artifact was newly downloaded on 9af; original F226 Question/Plan/reconnect and cleanup proof keep their identities. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier title/startup/custody/Stop/deletion failures with unknown causes/bounded recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun passed 128. Retry alone establishes no cause or full local-suite pass. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and checked through behavior; unpriced provider dollar cost stays UNKNOWN. Callbacks need authenticated reachable HTTPS. Scope, revocation, budgets, completion and cleanup remain authoritative. Operator Stop intent is separate from physical containment. Saved legacy agents retain their selected runtime. Prior shared title-save/deletion failures remain disclosed. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 09:01:39 -06:00
DottaandPaperclip	3910352c92	feat(runner): add experimental Cursor Cloud provider (#18068 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner gives each provider durable execution, tool, and completion contracts. > - Cursor Cloud currently has a direct adapter outside that native contract. > - Cloud work needs exact ownership across HTTP, SSE, Stop, and recovery. > - This PR adds a qualified Cloud API driver and verified proxy to Runner. > - Product setup and hosted placement remain separate dependent changes. ## Linked Issues or Issue Description Refs #5664, #11803, #17299, and #17471. Product integration is #18069. Managed-only API placement is #18310. The related direct-adapter model/transcript work is not replaced. No duplicate native Cursor Cloud integration was found. ## What Changed - Add native `cursor_cloud` / `cursor_cloud_api` with explicit model/repository validation, bounded HTTP/SSE, durable per-turn checkpoints and a qualified Node proxy launched through the Rust facade. - Commit terminal custody before publication/release; coalesce consumers, preserve ambiguous create/Stop/recovery fences, reject late tools, and distinguish definitely-unsent oversized requests from ambiguous responses. - Support scoped cold attachment, current authority, reconnect and verified artifact staging. Keep unsupported warm/external and Windows paths closed. - Preserve early source cursors and verify immutable legacy prefixes before suffix replay, including supported compaction/masking. Optional proof publication failure falls back to complete cold verification. - Preserve canonical records at the durable native producer with JSON-safe privacy and the existing 8MiB bound; drain admitted successor flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and retain S3 and legacy CLI checks. ## Verification Core head `37c442ec8dd4`: all 66 check contexts are terminal successful/neutral/skipped, including required `ci / verify` and `ci / e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads. Dependent Product `81e828d20a77` and Top `dd2959689627` include the later client handoff correction and are running their own fresh gates. Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and normal Runner build passed. Runtime Top `9af1a5de42a2`: build and recursive typecheck passed. Earlier source-scoped evidence includes 398 focused tests, five relay tests and token/type gates. The 251-assertion, 7/8-suite run failed physical PostgreSQL teardown; the unchanged stable WSP suite passed 128 through the documented wrapper. The downstream fixture corrections passed 7/7 browser and 121/121 startup tests without timeout/oracle relaxation; a broken-unsubscribe control fails. Hosted runtime `9af1a5de42a2` has a passing normal build and recursive typecheck. Latest dependent heads additionally contain a pure client fix for stale live/history overlap, with 48 focused tests, UI typecheck and token gates passing. Its new client is not deployed in the frozen hosted image; server/native/API/provider source is unchanged by these four downstream files. Fresh hosted Linux admission on `9af1a5de42a2` retired its owned runner/proxy/socket with zero prompts. The saved configuration survived reopening, and Test authenticated the bound key/catalog/selected model without a prompt. Artifact with `claude-sonnet-4-5` completed, downloaded the exact 27 bytes, matched 121 canonical identities against complete logs and independently reached FINISHED. The active Stop retry was clicked within its immutable boundary, produced one canonical interrupted turn and acknowledged run-only cancellation, retained a valid 35.076s quiet window (maximum gap 4.584s), and independently remained CANCELLED/artifact-absent beyond the original 180-second command deadline. All provider-observer credential callbacks, DB clients and loaders closed; accounting is settled and explicitly unpriced. The first fresh Stop retains its failed 10.373s sampling gap and every sample. On the upgraded9af server, the historical F226 artifact was reopened and downloaded with its exact original hash after its provider agent had been deleted/GET404 earlier today. Historical F226 full Artifact/Stop cleanup, Question, corrected Plan and reconnect retain their original identities. Deleting the three newly created disposable provider agents and the new-case post-delete grade remain blocked on explicit user approval after automatic review rejected expanding the two original approved IDs. No new-case cleanup or complete fresh matrix pass is claimed; this is separate from the directly observed task/Stop behavior and retained exact cleanup proof. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier startup-title, title-save, custody/Stop and deletion failures with unknown causes and bounded recoveries. `dea023` Artifact passed 17 checks but Stop failed its false warning and 50 duplicate log identities. Pre-repair ACK failure and original Core CI failure remain recorded; retry alone establishes no cause or full local-suite pass. Provider dollar cost stays UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. The caller’s explicit values are submitted and behavior checked live. Unpriced provider dollar cost stays UNKNOWN. Only qualified per-turn controllers are admitted. Uncertain work and incomplete cleanup stay fenced. Existing damaged/duplicate archives remain failed evidence. Prefix proof adds bounded object-store I/O; the 8MiB producer bound does not guarantee future providers’ output fits. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 08:38:57 -06:00
Devin Foley Paperclip Dotta	17e8bd7735	fix(runner): save native checkpoints when the sandbox disk is full (#18705 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox save two things after each turn: the provider checkpoint and the workspace export > - An agent can fill the sandbox disk during its own work, for example with a package install > - The turn still finishes, but both saves need free sandbox disk, so they fail and the task becomes blocked with no clear reason > - This pull request makes the checkpoint work on a full disk and makes a full disk visible in the export failure > - The benefit is fewer stranded checkpoints, and an operator sees at once why an export failed ## Linked Issues or Issue Description Related: #18715 added the shared-principal response-recovery control synchronized below. #18644 reduced the scratch disk that the workspace capture uses. #18172 added the disk-free streaming fallback to the Daytona directory sync-out. Both were deployed when this failure occurred. Neither changes the checkpoint copy, and neither can help when the disk has no free space. What happened? A native agent ran a package install for a large monorepo in a 10 GB Daytona sandbox. `df` in the run showed `10G 10G 4.6M 100%`. The agent committed its work and reported `done`. Then: 1. The provider checkpoint made a filtered copy of the provider directory inside the sandbox (`tar -cf archive; tar -xf archive -C snapshot`) to apply its exclusions. On the full disk, tar failed. The run recorded `runner_remote_checkpoint_snapshot_failed` without the reason, and the partial copy stayed on the disk. 2. Workspace finalization failed three times, and file preservation failed six times. The capture builds its archive inside the sandbox, so each attempt failed. Every failure was recorded as `remote_workspace_operation_failed` / `export_interrupted`, so nothing named the full disk. 3. The task went to `blocked`. The only copy of the commit was in the retained sandbox. Expected behavior The checkpoint does not need free sandbox disk when the provider can stream. A workspace export that cannot fit says that the sandbox disk is full. Steps to reproduce 1. Run a native agent with a provider whose checkpoint has exclusions (for example ACPX) in a small remote sandbox. 2. In the turn, fill the sandbox disk (for example `pnpm install` of a large repository, or `fallocate` of the free space) and finish the task. 3. The checkpoint fails with `runner_remote_checkpoint_snapshot_failed`, and every workspace export attempt fails with a generic code. Paperclip version or commit Observed on `410b0e27015`. This change is based on `5955dd4eb4c`. ## What Changed - Synchronize the shared-principal response-recovery regression with its saved comments and empty final deadline. Cover an in-flight discovery invalidation. Keep every authority lock, exact publication, error, and no-retry assertion. This changes test synchronization only. - `syncRemoteRunnerDirectoryOut`: when the in-sandbox checkpoint copy fails with a full-disk tar diagnostic, remove the partial copy and pass the exclusions to the provider sync-out (`exclude` on the directory mapping). The Daytona sync-out streams the directory without sandbox disk when it cannot write its archive. Other copy failures and timeouts still throw `runner_remote_checkpoint_snapshot_failed`. They now also remove the partial copy. - `remote-workspace-transport.ts`: the helper install and the capture scripts report `remote_workspace_sandbox_disk_full: ...` when Node reports `ENOSPC` or `EDQUOT`, or a Git or tar child reports "No space left on device". `command()` passes only that exact message through, like the existing unfinished-Git-operation message. - `native-workspace-finalizer.ts`: a remote save that fails with that message adds it to the failed operation's stderr. The failure code, the retry budget, and the recovery flow do not change. ## Verification - Current-head infrastructure interruption: the [Build job](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125923826158) completed the actual build and production bundle boot, then the runner received a shutdown signal during the clean public Grok npm-install check. GitHub marked that step cancelled. The exact host-stop cause was not retained. One bounded rerun of only the failed Build and dependent `ci / verify` jobs passed on the unchanged head. [Build attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125931400745) completed successfully, including the clean Grok npm-install check. No product code, assertion or workflow changed for this interruption. - CI failure retained: commit `6c5937fcae` failed [chat shard 1/3](https://github.com/paperclipai/paperclip/actions/runs/41634638570/job/125913957217) because the shared-principal fixture saw zero saved comments after `worker.ready` and zero active work. The other chat child passed. The exact case passed once locally and then 30 times on the same Node 24.22.2 version as CI; those passes alone do not explain the CI interleaving. - Controlled RED/GREEN: invalidate the real database discovery once while its read is in flight. The worker correctly retains a future reconstruction deadline, but the old initial-pass/active-count barrier reproduces the same zero-versus-one comment assertion. Waiting for both saved comments and final `active=0`/`nextWakeAt=null` passes. Six real-PostgreSQL controls pass on final commit `a60e5629e8`, including endpoint independence, source revocation, and external lock failure/recovery. No sleep, timeout increase, weakened assertion, or product scheduling/authorization change was added. - A local server typecheck first lacked the checkout's built Runner declarations. After the declared Runner TypeScript build, the server typecheck passes on `a60e5629e8`. Final-head remote verification is complete: 54 successful checks, 3 expected skips, no failures or pending checks, and successful `ci / verify` and `ci / e2e` aggregates. Greptile is 5/5 on this exact commit with no unresolved review threads. - Merge preparation: rebased onto master `b22ef22d56` and kept both test-import additions. Added the checkpoint fallback and capacity-failure contract to `doc/execution-semantics.md`. On rebased commit `6c5937fcae`, 47 focused checkpoint, transport, finalization and retained warm-projection tests passed across three files. The name filter skipped 1,040 unrelated cases. Map and whitespace checks passed. The final-head CI and Greptile results are recorded above. - `npx vitest run server/src/services/native-runtime/native-session-executor.test.ts server/src/services/native-runtime/native-workspace-finalizer.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts` passes (1,152 tests). - `npx vitest run server/src/services/remote-workspace-transport.test.ts server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts` passes (30 tests). - New tests: - Checkpoint, with real files: a full-disk snapshot leaves a partial copy. The provider then gets the exclusions, the session files are kept, the package caches are left out, and the partial copy is removed. - Checkpoint: a permission failure and a timeout still fail, remove the partial copy, and do not call the sync-out. - Transport: the real capture scripts run with an `ENOSPC` preload, for the helper install and for the archive write. Both report the full-disk message and leave no partial attempt. A message with extra text still collapses to `remote_workspace_operation_failed`. - Finalizer: the operation stderr names the full disk and keeps `workspace_sync_out_failed`. Other failures keep the bare code. - With the source change removed, the 3 checkpoint cases and the 2 transport ENOSPC cases fail. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Suites that refer to the changed error codes (`explicit-native-continuation`, `native-recovery-notice-work`, `native-run-finalizer-telemetry`, `native-workspace-export-retry`, `remote-workspaces.integration`) pass. In `chat-channels.integration`, 2 cases fail locally on unmodified `master` too (a bundled skill path outside the checkout, and a chat run timing assertion). They are not related to this change. - Manual recovery of the observed task: the retained sandbox was resumed with free space, file preservation was requested again, and the saved workspace contained the agent's commit. ## Risks - The fallback runs only after the snapshot fails with a full-disk diagnostic. The normal path does not change. - Provider exclusions are not anchored to the directory root. In the fallback, a nested entry that has an excluded name (for example a nested `tmp` or `config.toml` in a Codex home) is also left out. The fallback never copies more than the snapshot, so credentials stay out of the checkpoint. - The fallback depends on the provider honoring `exclude`, which the sync protocol requires. The Daytona and Kubernetes providers honor it. - A workspace export on a full disk still fails. This change only names the cause. A full fix needs one of these: free-space reserve for the export, removal of re-creatable caches before the export, or a capture that streams its archive. Each of these changes behavior, so this pull request does not include one. - No migration. ## Model Used - OpenAI Codex, GPT-6 family, assisted with merge preparation, conflict resolution, documentation, test synchronization and verification. The harness does not expose the exact serving model ID or context-window size. - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). It did the root-cause analysis from run events, tool output, and sandbox inspection, wrote the code and tests, and ran local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <bippadotta@protonmail.com>	2026-11-03 07:09:52 -06:00
DottaandPaperclip	b22ef22d56	fix(runner): show provider failure details on tasks and runs (#18622 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts providers and records their run outcomes. > - Provider failures can contain useful details below a generic protocol error. > - Those details were lost at the sidecar, command receipt, and run projection boundaries. > - A blocked task therefore showed a generic failure even when the provider gave a clear reason. > - This pull request carries a bounded, sanitized provider failure through those boundaries. > - Operators can read the actual cause on the task and run pages. Existing recovery policy still controls retries. ## Linked Issues or Issue Description Refs #17543 for the related closed JSON-RPC classification work. Refs #17272 for provider recovery in the legacy OpenCode adapter. This change adds display evidence for native Runner providers and keeps recovery policy separate. What happened? A native Claude session returned an HTTP 429 message below an ACP `Internal error`. The task and run showed only a generic startup failure. Operators had to inspect the provider response to learn that the account was rate limited. Expected behavior The task and run show the sanitized provider message, provider name, startup or turn phase, and available HTTP status. Unknown categories keep the message. Display evidence does not grant a retry or replace a recovery decision. Steps to reproduce 1. Start a native Claude run whose account is rate limited during model selection, or return a structured provider failure from a Codex or OpenCode turn. 2. Let the run fail and open its task page. 3. Before this change, the page shows a generic failure. With this change, the page shows the provider explanation. The run page shows the same explanation after reload. Paperclip version or commit Source checkout based on `f8a888a072`. ## What Changed - Add one JSON Schema for `ProviderFailure` and generate matching Runner and shared TypeScript readers. Keep the Runner package independent of the app workspace at runtime. - Normalize known ACPX, Codex, and OpenCode error fields. Preserve unknown messages. Redact known credential values and credential patterns before applying the 4,000-character limit. Do not copy response bodies, headers, stacks, or prompts. - Carry startup failures through the sidecar, Rust Runner, durable command receipts, and private transport error details. Keep provider text out of generic transport error strings and Sentry failure messages. - Keep turn failures in authenticated run events. Recover the same evidence from persisted events during server finalization. Store the cause in the company-scoped run result. - Show one shared notice on failed tasks and run details. Keep existing retry permissions, scheduled retry state, and resolved-task behavior. - Reapply registered run-secret and current-user redaction to task display data. Batch secret lookups for only the visible provider causes. - Document the contract and update the capability maps. Add regressions for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode errors, redaction, durable receipt replay, restart finalization, and UI rendering. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions build, test, package, and validation jobs passed on `d49c79ab4a`. The security checks passed. Greptile gave this head 5/5 with no open review threads. Paperclip Review remains pending. Its review worker exhausted six workspace recovery attempts with export_interrupted. A fresh review request was accepted and is waiting for that saved workspace. The ordinary Retry action rejected reuse of the stopped request. This is an external review-worker blocker; no check was overridden. The duplicate local `pnpm test:run` was stopped after the CI test shards passed. - Focused server settlement, provider evidence, and restart tests: 93 passed. Runner admission and normalization tests: 103 passed. Focused Runner wire and terminal tests: 29 passed. The real Rust/sidecar command receipt replay test passed. - The new projection test failed before implementation because the task had no provider cause. The startup producer regression also failed before extraction moved ahead of the generic error-code replacement. Both pass with this change. - Live check: a rebuilt local sidecar received an actual Claude HTTP 429 during startup. The new cause kept the full provider message. A synthetic task in an isolated local app then reached the same provider error through the native Runner path. The dedicated run result, task API, task banner, and run detail all showed the same message after reload. The later reporting fix keeps run.error generic; a regression proves the display cause still survives restart and stays out of Sentry capture. - The first isolated app attempt failed earlier in `admission_credential`. An ordinary manual Retry reached the provider and produced the 429 above. This PR does not claim to fix that separate admission failure. An initial parallel local database test setup also failed; the serial restart suite passed. - Security review found that copying display text into run.error could forward it to Sentry. A regression failed with that behavior. The fix keeps the cause in its display field. All 38 settlement/restart tests and the server typecheck pass. - All 29 protocol contract tests pass, including bounds and extra-field rejection. An earlier full Rust core run passed. The later full run hit one 5-second receipt-limit timing assertion; its isolated rerun passed. The full Linux and Windows Rust CI jobs passed on this head. - Review regressions pass for explicit Codex driver identity, native Rust startup and resume receipt propagation, oversized OpenCode/ACPX errors, task read-time registered-secret redaction, and the standalone source closure. The focused Runner run passed 103 tests; projection/restart tests passed 16. - Read-time redaction edge cases passed: a missing decryption key omits only the unsafe optional explanation; expanded redacted messages remain visible within the Unicode character limit. All 14 task projection tests and server typecheck pass. - The package Build check found an old Pi assertion against Error.message. The probe now checks the exact closed classification in the authenticated receipt and requires the new provider cause. Both polling transport cases pass. A local credential-free Pi probe stopped at admission_installation because this checkout lacks the Pi distribution; the installed Linux package probe passed in CI on this head. - UI coverage includes startup and turn text, unknown categories, literal rendering of provider text, read-only retry behavior, and resolved-task suppression. Token gates and map checks passed. - The live check covers Claude startup on macOS. Codex, OpenCode, terminal failures, and restart paths have deterministic regression coverage. The change is not deployed. ## Risks - Provider messages are untrusted text. The boundary selects specific fields, redacts credential patterns and known environment secret values, bounds the message, and renders plain text. New provider-specific secret formats may need added redaction patterns. - Optional fields preserve compatibility with older producers and stored runs. Old runs without the field keep their current fallback text. No database migration is required. - The cause stays in company-scoped run data. This change does not send provider text to first-party Telemetry or add it to OpenTelemetry span attributes. - Normalized categories and retryability are display evidence. Existing recovery classifiers, retry limits, ownership checks, and approval gates keep their authority. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser verification. The host does not expose the exact serving variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:13:26 -06:00
DottaandPaperclip	718336eb04	fix(native): refresh prior warm controller projections (#18718 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native remote sessions keep their controller state in a database checkpoint. > - An admission can also leave a full local copy of a prior run's state. > - The prior owner can commit more state without changing its run ID. > - Warm admission skipped a refresh when the local and saved run IDs matched. > - This pull request refreshes that prior-run copy when no retained warm owner exists. > - The next turn can use the committed state while existing ownership checks remain effective. ## Linked Issues or Issue Description What happened? A prior warm run can complete after an earlier admission reads its controller checkpoint. The local copy can still contain a pending command or lack the final terminal event. A later admission sees the same run ID and skips restoration. It can then hold work against that old copy even though the database checkpoint has settled. Expected behavior Read the canonical prior-run checkpoint before reuse when no retained in-memory owner exists. Keep current-run state with its controller. Keep unresolved canonical work and invalid checkpoint bindings blocked. Steps to reproduce 1. Create a full prior-run local projection while a command is pending. 2. Complete that command and record the terminal in the canonical checkpoint under the same run ID. 3. Start an ordinary remote warm continuation without a retained in-memory owner. 4. Before this change, the local command remains pending. With this change, the completed command and terminal replace the stale copy. Paperclip version or commit Reproduced on master `36df8933f0` before changing production code. Related work: #18018 restored per-turn checkpoints; #18343 added controller replacement recovery; #18554 restored missing remote suspension receipts. This change handles a full prior-run projection whose run ID already matches the saved supervisor. It does not change the settlement guard or MCP compatibility policy. ## What Changed - Refresh a prior-run warm controller projection even when its run ID has not changed. - Keep current-run and retained warm owner authority on the existing path. - Add five executor cases for a settled source, a pending canonical command, a current run, legacy file-backed state, and an invalid checkpoint binding. - Add an opt-in Linux continuity case with a stale full local projection and two follow-up turns. - Update execution documentation and the paired session verification recipes. ## Verification - RED before the production edit: the executor fixture retained `pending` in its local command record when the canonical record was `completed`. - GREEN on the final commit: 48 focused tests passed across executor restoration, retained warm owner rotation, per-turn restoration, and remote settlement guards. - `pnpm --filter @paperclipai/server typecheck` passed. The server TypeScript check also passed after the integration fixture was added. - `node scripts/check-maps.mjs` and `git diff --check` passed. - The unit fixture uses the real durable controller serializer and the real executor entry. It stops at current tool authority before provider/backend creation. It proves restoration and guard behavior, not full task acceptance. - Actual protocol RED/GREEN: the same opt-in continuity fixture uses real PostgreSQL, TLS, the current Linux Rust Runner, and a deterministic fake Codex backend. With only the executor restored to its parent revision, continuation fails with `prior_owner_terminal_state_indeterminate` in the settlement guard after one provider turn (20.16 seconds). The PR source completes all three turns with one provider thread and final suspension (34.77 seconds). Hard recovery permits one provider-process replacement; the healthy final follow-up starts no additional process. Both runs use the same Runner binary and owned container. - The exact PR executor source was restored after the baseline comparison, its Git blob was verified, and the worktree is clean. The disposable local container was removed. This fixture proves the controlled stale-projection recovery path; it does not establish a live model or Daytona API result. - Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. Greptile is 5/5 on the current commit, with no unresolved review threads. No original task was retried and no staging sandbox lifecycle was changed. ## Risks - Admission without a retained owner performs one additional scoped checkpoint read when a full prior-run projection already exists. - The canonical checkpoint must still pass its company, task, session and record validation. A pending canonical command stays held. Current-run and retained owner state are not replaced. - The regression proves a stale local projection defect. It does not establish that a historical incident had those same local bytes. ## Model Used OpenAI Codex, GPT-6 family. The harness does not expose the exact serving model ID or context-window size. Capabilities: repository inspection, code generation, tools, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:12:22 -06:00
DottaandPaperclip	16643a6e03	fix(runner): recover pending Stop cleanup after controller restart (#18682 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A Stop can remain pending when the controller restarts. > - Recovery must reach the existing session without starting more paid work. > - Startup and budget fences currently block that cleanup, and interrupted runs leave the restart selection. > - This pull request gives exact audited Stop cleanup a shared authority check and records its real acknowledgement. > - The benefit is safe cleanup while usage debt remains honest. ## Linked Issues or Issue Description What happened? A pending native run-only Stop can retain an unfinished execution owner after controller loss. Its startup cancellation fence prevents recovery admission. If teardown marks the run interrupted and clears its task lock, restart selection also omits it. A recovered session can receive cancellation without the durable acknowledgement that the normal Stop request would save. Expected behavior Recover only the existing remote session for the audited pending Stop. Keep fresh provider work blocked, including during budget pause. Keep incomplete usage pending until a real complete receipt arrives. Steps to reproduce 1. Start a native run with a held reservation and an authenticated remote checkpoint. 2. Persist a run-only Stop and its startup and execution cancellation fences before acknowledgement. 3. Lose the controller. Leave the coordinator unfinished, mark the run interrupted, and clear its task lock. 4. Attempt restart recovery and observe the missing claim or cancellation admission rejection. Related work: #14733 binds caller Stop requests. #18658 handles plugin startup cleanup obligations. #18035 proposes broader accounting changes. This change repairs existing-session Stop recovery and preserves current accounting guards. ## What Changed - Add one shared check for the exact run, company, issue, pending Stop, intent audit, live recovery lease, generation, and remote reattachment disposition. - Include eligible interrupted Stop owners in restart selection. Keep generic interruptions and another task execution owner fenced. - Reuse only the original held reservation for cleanup. Keep budget pause effective for fresh, bootstrap, dead-runner, and unproven recovery work. - Apply the same cleanup authority at heartbeat runtime selection and native entry. Skip attachment staging and warm-owner retirement only for that exact cleanup. Revalidate at session publication and fail closed if the proof changed. - Send the existing Stop through the audited cancellation producer before turn submission. Close the coordinator only after acknowledgement, including system-owned run-only Stops. - Give verified cleanup a transport purpose that retains pending semantic inputs without replay, dispatch, or a semantic reply. Add an explicit Core retain-only intake mode; preserve authenticated identity, schema, digest, sequence and replay checks. Ordinary execution still requires its dispatcher. - Add PostgreSQL, native protocol, and startup-fence regressions. Update the execution contract and recovery maps. ## Verification - The combined Stop regression fails on unchanged production at the cancellation reservation guard. Interrupted restart selection also fails before the repair while its running sibling passes. - An assembled `heartbeatService.recoverNativeRunsAfterRestart` regression exposes the earlier runtime-selection fence with the rest of the repair present. The fixed path reaches selector/claim, runtime selection, original held-budget reuse during pause, dispatch, real native protocol, audited cancellation, and heartbeat settlement. It starts no provider turn, acquires no new sandbox, and does not reclaim the task lock. - Fail-if-called attachment staging and turn boundaries remain untouched. A retained warm-owner regression prevents retirement. Invalidation at session publication fails closed. Actual backend/driver constructor tests verify cleanup-purpose forwarding, absent default options, and denial under foreign ownership. - Missing usage retains accounting debt and the hold. A later complete receipt records one 25-cent charge and settles that same hold. Concurrent duplicate cleanup is denied. Both user-owned and system-owned Stops pass. - Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary saved-tool and completion recovery, prove the authority-denial race before cancellation, and qualify cleanup with pending ordinary/completion inputs and either handler ordering. - Real authenticated Core intake: 11 pass. Callback-present and callback-absent retain-only cases persist and acknowledge unanswered input without dispatch, recovery or result. Default missing-dispatcher rejection and digest-integrity/suspension guards remain effective. Omitting the callback alone first failed by disconnecting; the explicit retain-only mode fixes that defect. - Repository `pnpm -r typecheck`, the server/runner-vendor build, map checks, and module boundary checks passed locally before integrating the latest retained-startup change. That integration passes 11 native startup/cleanup compatibility cases and 3 assembled recovery/accounting cases. The final broad affected-suite attempt was stopped without a verdict; it is incomplete. Full repository tests and build are delegated to current-head GitHub CI. - Review exposed an additional real-process close failure with unanswered input. The corrected path requires a fresh exact provider-stop receipt confirming exit, followed by durable event drain and exact runner suspension. All 11 focused transport cases pass, including missing/false exit-proof negatives and ordinary recovery siblings. Runner TypeScript and map validation pass after this correction. - The assembled test controls provider, workspace-transfer and GitHub bootstrap I/O. It does not qualify hosted credentials, physical provider termination, workspace preservation, or historical receipt recovery. Local socket-permission setup failures are separate from semantic regression results. No hosted replay has run on this repair. - CI exposed an ordinary factory-call compatibility regression and an obsolete authority-read test fixture. The repair preserves single-argument ordinary calls and updates the race fixture without changing its Stop assertions. The complete native executor suite now passes all 1,049 tests; the ordinary factory boundary and all five Stop-accounting cases pass. A separate local environment-default case twice exceeded its five-second wait while other work ran; its prior-head CI case passed. The same unchanged case passed an isolated diagnostic without changing its assertion or timeout; the precise contention source remains unproven. - The final fixture edit initially used the wrong overloaded promise type. CI rejected it; the annotation-only correction passes the exact `pnpm run typecheck:build-gaps` entry point locally, including server service tests and remaining workspaces. No runtime assertion changed. - Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required `ci / verify` and `ci / e2e`, build, typecheck and test groups pass. Greptile is 5/5 with both findings resolved. Three expected checks are skipped; the optional native check is neutral, while its final current-head comment gives 5/5 from source inspection without a checkout or test execution. The first server-shard attempt lost its self-hosted runner connection; one scoped rerun passed. Its original annotation and unavailable-log evidence are retained. ## Risks - This changes execution authority. Cleanup requires an audited, exactly bound pending Stop, an unfinished coordinator, a live remote reattachment claim, and the original held reservation. It never authorizes a replacement provider turn. - Missing checkpoint, original held reservation, termination, or usage evidence can keep cleanup or accounting pending. The repair does not manufacture acknowledgement, replies, receipts, or zero cost. - Retain-only intake is an internal trusted constructor option. It changes no agent tool permission. Ordinary callback requirements and semantic dispatch remain unchanged. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. The context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run affected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 18:22:40 -06:00
Nicky Leach	3a7bb4dac6	perf(runner): activate shared retained startup context (#18405 )	2026-11-02 15:12:57 -08:00
DottaandPaperclip	3f7811eef0	fix(runner): keep requests responsive during retired backup cleanup (#18619 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native runs keep durable harness backups for continuation and recovery. > - Publishing a new backup retires earlier backup directories in the server process. > - The synchronous permission walk and recursive removal block unrelated requests. > - This pull request makes that cleanup asynchronous and keeps publication waiting for completion. > - The benefit is continued request progress while large retired backups are removed. ## Linked Issues or Issue Description What happened? Publishing a backup with many small files blocks the Node event loop during directory permission repair and removal. A local comparison with 20,000 retired files reproduced 1.37–1.95 second event-loop stalls. This is a pause candidate. A short hosted profile has not established it as the dominant source of sustained CPU use. Expected behavior Unrelated requests can progress during cleanup. Publication still waits for complete removal, preserves rollback on a failed lease stamp, and leaves live files and symlink targets unchanged. Steps to reproduce Run the benchmark in `doc/performance/native-backup-publication.md`. It compiles the actual publication service from a pinned baseline and the working tree, prepares immutable retired backups, and sends foreground HTTP probes from a separate process. Paperclip version or commit Measured baseline: `774ddc72713d6aa7cb516691c71f1fc23bcfa5ff`. The measured treatment source is committed in this branch. Current head: `0a69cf2d43c2f9fcf032d96c1dceb56163bf21ba`. The runtime patch is unchanged from the measured source; later commits only strengthen the test and benchmark evidence paths. Deployment mode Local compiled service on macOS arm64 with Node v24.22.2. No hosted rollout has occurred. Related work: #18380 changed retention scheduling. #13676 included the existing immutable-backup cleanup behavior. Neither change makes publication cleanup asynchronous. The GitHub duplicate search also found #11929, which concerns database dumps rather than native harness backups. No matching open event-loop cleanup issue was found. ## What Changed - Use asynchronous filesystem operations for retired-backup cleanup. Visit directories serially and skip unnecessary metadata calls for file entries. - Await cleanup before publication proceeds, during rollback, after successful rotation, and in executor finalization. - Exercise immutable current and previous backups in the publication test. Preserve the existing symlink-isolation and rollback tests. Add a real separate-process HTTP probe that must complete while publication is still active. - Add a repeatable service benchmark and document its fixture, metrics, and acceptance limits. ## Verification Draft status: focused correctness tests, the local comparison, and repository typecheck pass. The complete local test gate is incomplete. The build passed. Complete CI, hosted acceptance, and final-head review remain pending. - Final focused Vitest run: 2 files, 4 tests passed in 649 ms with the normal server configuration. The regression warms a separate Node client, waits for its HTTP request during the stamp callback, queues the response for the next event-loop turn, and then allows final retired-tree removal to start. The async treatment serves that response before publication completes. The original synchronous cleanup fails the same final regression in a separate source fixture. That expected control failure is retained. - Syntax and `git diff --check` passed. - `pnpm -r typecheck` passed on head `332c57b88c` in 152.18 seconds, including the native Runner build. Validation uses Node v24.22.2 and the repository-pinned pnpm v9.15.4 with isolated dependencies. - `pnpm test:run` was intentionally stopped after 1,390.24 seconds (exit 143). All six chat shards passed, covering all 1,817 parameterized cases across their exact partitions. The general server phase was advancing but incomplete. The repository timing manifest estimates another roughly two hours for the serial remainder, so validation moved to final-head parallel CI. This is not a full local test pass. The command began on `332c57b88c`; later changes only refine the regression and benchmark, and the final regression has its own focused pass. - Benchmark failure/metadata checks pass in a tiny isolated Git fixture: an injected worker exit 17 is reported in 0.44 seconds without hanging, and the result includes a staged source change when the unstaged diff is empty. - `pnpm build` passed in 41.80 seconds on `9f7e4e669b`. The final commit only releases a pending test probe on failure; runtime and build sources are unchanged. An injected client exit verifies that final test failure path exits promptly in 0.97 seconds instead of hanging. - Three alternating baseline/treatment pairs used fresh fixtures with 100 directories × 100 files in each of two retired backups. Each publication removed 20,000 files. A separate process sent HTTP probes at fixed 5 ms intervals, including overdue arrivals. - Every run verified the replacement manifest, complete cleanup, and unchanged symlink-target contents and mode. All 2,395 HTTP requests completed without errors. The largest measured generator delay was 8.8 ms. \| Metric \| Baseline median (range), ms \| Treatment median (range), ms \| \| --- \| ---: \| ---: \| \| Foreground request p95 \| 1,357.7 (1,287.4–2,014.0) \| 1.8 (1.7–25.9) \| \| Maximum event-loop delay \| 1,439.7 (1,371.5–1,954.5) \| 19.9 (17.4–40.7) \| \| Publication completion \| 1,429.1 (1,360.2–1,952.9) \| 1,439.5 (897.9–1,649.6) \| \| Process CPU during publication \| 1,194.1 (1,041.8–1,225.5) \| 2,423.1 (1,995.7–2,514.7) \| The treatment served 180–329 requests while publication was active. The baseline served zero during that interval. Process CPU includes foreground work that the baseline cannot serve until cleanup completes. The result supports responsiveness, with a CPU tradeoff. It does not establish lower total CPU or a hosted latency gain. Reproduce with Node v24.22.2: ```sh pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-harness-backup-cleanup.test.ts src/services/native-runtime/native-harness-backup-publication.test.ts node server/scripts/benchmark-native-backup-publication.mjs --baseline-ref `774ddc7271` --output /tmp/native-backup-publication.json ``` The benchmark records the baseline SHA, treatment HEAD and source diff. Raw local evidence remains outside the repository. A first small smoke run hit a sandbox restriction on loopback listeners; the authorized retry resolved it. The smoke also exposed load-generator interval drift. Fixed-clock arrivals replaced that generator before the reported measurements. Pending gates: complete current-head CI, final-head Greptile review, and the real run/checkpoint/recovery journey. The full local test run remains explicitly incomplete. The four concrete review findings were fixed. The prior head received 5/5; final-head review remains pending. Local evidence remains outside Git: `/private/tmp/backup-publication-measurement-20261102.json`, `/private/tmp/native-backup-focused-final.log`, `/private/tmp/native-backup-typecheck.log`, `/private/tmp/native-backup-test.log`, `/private/tmp/native-backup-test-interruption.json`, `/private/tmp/native-backup-build.log`, and `/private/tmp/native-backup-harness-check.json`. The harness-check receipt points to its isolated metadata and worker-failure logs. The final client-failure and baseline-control logs are `/private/tmp/backup-benchmark-check-gprcfiiz/failed-client-cleanup.log` and `/private/tmp/backup-benchmark-check-gprcfiiz/final-probe-baseline-control.log`. These local paths are retained for the author; the commands above let reviewers reproduce the synthetic proof. ## Risks - Asynchronous removal adds filesystem-pool work and used more process CPU in the fixture. Check resource pressure as well as responsiveness before rollout. - Filesystem operations can now interleave with unrelated work. The existing session owner still controls publication. Cleanup remains awaited; no deletion is detached and no scheduler is added. - Errors must still fail publication or restore the prior backup as before. Corruption checks and destructive lease authorization are unchanged. - This synthetic macOS fixture does not represent every Linux volume or backup shape. It does not qualify a deployed fix. - Rollout acceptance should confirm the serving SHA, checkpoint completion, continuation/recovery, error rate, and loop delay under matched backup load. Revert if cleanup or recovery fails. ## Model Used OpenAI Codex, GPT-6 family, with tool use and code execution. The exact served model ID and context-window size were not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 17:00:42 +00:00
DottaandPaperclip	baa4a2e64e	fix(runner): restore Codex auth before source recovery (#18612 ) Restore disposable Codex authentication and safe provider configuration before cold source recovery. Preserve source receipts and conversation state until reconciliation, and rebuild assigned tools from the admitted launch binding. Validate remote launch destinations, clear stale authentication when no selected credential exists, and atomically publish SSH file transfers without following retained links. Red/green recovery regressions, a real two-turn Codex/MCP probe, and all current PR checks pass. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-02 10:14:24 -06:00
Dotta devinfoley Paperclip	06bc8db75c	fix(runner): install verified API artifacts in execution workspaces (#18494 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents use authorized API tools to read saved results. > - Copying a large response as model-generated text can lose bytes. > - The execution workspace needs an exact copy of the saved artifact. > - This change copies the authorized bytes without shell API credentials. > - The runtime verifies size and SHA-256 before it reports success. > - A live evaluation checks capture origin, installation and delivered bytes. ## Linked Issues or Issue Description What happened? A native agent lost one byte while copying a 24576-byte API text page. Existing capture and paging controls passed. The runtime had no verified saved-artifact-to-workspace copy tool. A separate historical screening refusal does not establish capture corruption. Expected behavior An authorized native run can install the complete saved artifact. The stored and installed bytes must match in size and SHA-256. A conflicting file must remain unchanged. A lost publication receipt must not cause an unsafe repeated write. Steps to reproduce Run `server/src/services/native-runtime/artifact-transfer.integration.test.ts` on baseline `c46b8f35add9255fc71be294d27f5a7865bcb47f`. Its capability assertion fails because `download_artifact` is absent. Apply this PR and run the same regression with the filesystem and API controls. It verifies 388131 synthetic bytes through the authority and installation boundary. Refs #18429. Related #17393, #17086 and #17285 cover paging, transfer foundations and remote output reads. This PR adds the inbound installation contract. ## What Changed - Add `download_artifact` for supported standard-mode native runs, with a 10 MiB limit. - Reuse current company, task, run, asset-content and responsible-user access checks. Check authority again before publication. - Bind installation to the current execution lease. Stage through confined Linux descriptors and publish with an atomic create-only link. Keep API credentials out of execution commands. - Verify bytes at storage and execution boundaries. Reject unsafe paths, symlinks, different existing files, truncation and corruption. Reconcile an unknown publication through verification only; an identical explicit retry remains safe. - Add API documentation, access-map entries and content-free audit records. - Preserve only the pinned Daytona SDK numeric HTTP status when failed creation cleanup is unconfirmed. Keep provider exception messages and headers private. - Keep only the finished runner binary and portable provider pack in their builder layers. Remove Rust targets/downloads and the Node install workspace/store before export. Report tree sizes. - Add one explicit Codex/Daytona Product E2E cell. Its independent oracle checks the server capture audit, redacted tool receipts, stored source, delivered attachment and browser link after reload. Uploaded substitutes and one-byte loss fail. ## Verification - Merge update at `6766b46553cab5b9787ade87a002eccaf50e68fc`: merge current master and preserve both explicit suites in the catalog. Focused catalog, image and oracle checks: 122 pass. Filesystem and authority integration: 20 pass. Maps, module boundaries and whitespace checks pass. New-head CI and Greptile are running. Full local build and typecheck remain unavailable because cargo is absent. - Behavioral red/green: the absent-tool regression failed before implementation and passed afterward. The initial missing-package attempt is recorded as setup failure. - Linux filesystem and real PostgreSQL/authority integration: 19 filesystem tests and 1 authority test pass. Existing API authority, capture and paging controls: 70 pass, 2 skipped. Provider transport in these integration controls is simulated. - Qualification calibration: 118 focused tests pass. The real persisted-event redaction case and a byte-identical uploaded substitute each failed before their oracle repair. The unchanged controls then passed. Harness typecheck passes. - Full recursive typecheck and build pass locally on the integrated production source at `086f7a2901`. Later commits change only qualification tests and documentation. - Structural map/module checks and whitespace checks pass. - Full local `pnpm test:run` was attempted by the implementation worker and stopped under its 8 GiB memory limit, with no full-suite verdict. Current-head CI must supply the full test result. - The complete local Product E2E support suite has 13 failures. The unchanged pre-qualification baseline reproduces all 13: host PostgreSQL semaphore capacity, stale timeout/profile/workflow inventory expectations, and expired fixed-date allowance fixtures. This is not a full local suite pass. - Earlier CI at `086f7a2901` failed a browser test during cleanup cancellation with a database lock timeout. It remains recorded. All browser shards passed on both later fully checked heads. - Two qualification campaigns were canceled during setup before provider execution while correcting the oracle. They provide no live product verdict. - Previous-head CI at `28fa001457f59df1c300c016dc569049e72d0061`: 53 checks pass, 3 intentional selection skips; no failed or pending checks. - Real image/export regression: campaign 41272351780 compiled the Linux runner and passed root/unprivileged CLI and transport checks, then failed BuildKit export with `no space left on device` (exit 102, filesystem 99.99%). No provider cell ran. Removing the Rust build trees alone was insufficient: campaign 41273604815 also failed during provider-pack export with exit 102. Commit `5bbb549c54` also consolidates dependency installation and pack construction so the disposable workspace/store can be removed in the same layer. All 122 image/oracle/catalog checks pass. Image and remote-pack publication then passed in campaign 41275196741 after cleanup changes plus a scoped 60-to-120 GiB expansion of only its delete-on-termination builder disk. The VM terminated and the expanded volume was deleted. This does not establish that cleanup alone makes the default 60 GiB builder sufficient. - The Daytona admission probe then failed before agent creation, with unconfirmed provider cleanup. The queued continuity case was canceled because it shares this prerequisite. No artifact-transfer or continuity success is claimed. - Status-diagnostic regression: the original generic wrapper failed the HTTP-status assertion. After the repair, the full plugin suite passes 501 tests, with 7 skips; its standalone TypeScript check passes. Controls reject invalid status values, avoid accessors, preserve the original create status and exclude credential canaries. - Previous-head Greptile: 5/5; its summary names `28fa001457f59df1c300c016dc569049e72d0061`. All review threads are resolved. - Live artifact transfer and three-turn warm continuity: NOT VERIFIED. The diagnostic case in https://github.com/paperclipai/paperclip/actions/runs/41277598289 confirmed Daytona HTTP 401 before agent creation. The configured `DAYTONA_API_KEY` in the existing `runner-e2e-paid` environment needs verification or refresh. The queued continuity control was not run after its shared admission prerequisite failed. No further provider attempt is planned until the credential changes. Unconfirmed provider cleanup remains recorded. ## Risks - Source-provenance changes can repeat the provider dependency install. The combined build layer trades install-cache granularity for lower retained disk use. - Linux descriptor semantics and Node are required on the execution host. Unsupported providers and restricted runtime profiles fail closed. - Storage reads buffer at most 10 MiB. A provider outage can leave a reserved staging file. Partial bytes never become the destination. - A lost publication receipt can leave a complete file with an unknown result. Recovery verifies the existing file without installation. Conflicts are never overwritten. - Artifact contents remain untrusted. The copy does not bypass screening or authorize downstream execution. - Live qualification uses the governed Codex API profile. It does not establish every model, subscription profile, platform or historical failure cause. ## Model Used OpenAI GPT-6 in Codex assisted with diagnosis, code, tests and qualification. The desktop runtime does not expose a more specific model deployment ID or context-window size. The model used reasoning, source inspection, tools and code execution. The native implementation agent is configured with OpenAI `gpt-6.1-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 08:27:09 -06:00
DottaandPaperclip	1a6017d646	fix(server): resume suspended sessions after controller replacement (#18554 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve a completed conversation as well as active execution. > - A warm remote Runner can suspend before the replacement admits another turn. > - The replacement restores its controller checkpoint from PostgreSQL, but lacks the Runner suspension receipt. > - It therefore tries to adopt a live process and rejects the safely suspended source. > - This change restores the verified receipt and preserves retained launch files while the old authority is reconciled. > - Later turns can keep the same provider conversation after the old web server filesystem is deleted. ## Linked Issues or Issue Description Refs #18552, #18543, #18366. What happened? The real rolling-deployment journey passed both HTTP handoffs, with 765 reads and 78 verified saves. The first three agent turns kept their native and provider sessions. After the second replacement, the fourth continuation failed with `remote_warm_session_not_idle`. The remote Runner was suspended with an empty outbox and no pending terminal. A focused regression reproduced that exact failure. After restoring its suspension receipt, the same test exposed a stale launch-file read during source recovery. Expected behavior A replacement can resume the same provider conversation from a settled suspended Runner. It must validate the physical sandbox and all persisted identity fields. A live source must still pass the existing adoption checks. Steps to reproduce Enable the Linux Runner fixture for `native-runner-restart-recovery.integration.test.ts`. Run the `diskless-warm-suspended` cases. The fixture completes and gracefully suspends each source, deletes the controller projections, and admits a later turn from another process. It does not write a synthetic suspension receipt. ## What Changed - Reuse the existing per-turn suspended-receipt reader before warm admission chooses live adoption or suspended recovery. - Keep the receipt read retryable when a failed remote read leaves only the controller projection. - Leave every non-suspended warm state to the existing live-adoption checks. A preliminary backpressure observation cannot reject admission before that owner inspects the current Runner state. - Preserve current sandbox binding, settled-outbox, exact-identity, local-evidence and admission checks. - Pass the structured launch specification to remote asset preparation. Source reconciliation keeps the old authority's retained launch files; a current-authority launch still stages its current files. - Add two Linux Runner/PostgreSQL regressions for two controller-cache deletions and three turns on one provider thread, including an interrupted receipt read. - Update the deployment qualification record, including the failed live evidence and the successful staging database-idle observation. ## Verification - Before the fix, the focused regression fails with the exact live `remote_warm_session_not_idle` error. - Both new regressions pass with real PostgreSQL and Linux Runner. The provider is deterministic; this is not a live model qualification. - The first receipt-only fix failed on the deleted controller skills cache. The completed fix passes that same scenario without recreating fixture launch assets. - All 1,067 surrounding launch supervision, warm recovery, launch-asset and controller recovery tests pass. The exact `cc23019691` server build passes. All ten Linux continuity cases pass, including live adoption, graceful suspension, per-turn resume, changed configuration and failed-read retry. Each case checks provider-thread identity and exact turn counts. The live Codex/Daytona journey passes on the same clean source and exact build: 518 reads, 53 saves and 53 verified saves with zero probe failures; both predecessor homes deleted; the active execution completes once; all four turns preserve the provider conversation and native receipts; saved files and the final command output are independently checked. No automatic retries. Child credential and object-storage bucket cleanup pass. Browser reconnection remains a separate hosted acceptance check. - The review refinement has a failing-before regression: only the preliminary read reports transient backpressure; the subsequent adoption reads the real Runner state. It and both suspension cases pass on the final source (three cases), with server typecheck. The existing idle check remains unchanged. An initial retry hit a fixture port left by the intentional pre-fix failure; a fresh container resolved that infrastructure error. - Independent cleanup verification found two archived setup sandboxes left by the live harness. Both were deleted and confirmed absent. The failed automatic-cleanup check remains recorded. - No claim of the ten-deploy Railway campaign or production canary yet. ## Risks - A cold warm-session admission can now make one bounded remote state read when local lifecycle evidence is absent. Its failure stops admission and remains retryable. - This path cannot replace contradictory local evidence, transfer authority, or treat a suspended process as a live one. - Source recovery must keep the previous authority's launch files until reconciliation completes. The launch identity selects that behavior; ordinary launches still prepare current assets. - No schema migration or public API change. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge CI qualification note: first final-head run `41322376054` passed the native/runtime and remaining suites but failed the unrelated `automatic_message_on_reactivation` browser fixture at its pre-reactivation deferred-queue count (0 rather than 1). The unchanged test passed locally against isolated PostgreSQL (1 test, 43.6 s); inspection found an asynchronous company-pause/fixture-seeding timing window. Original CI logs and local evidence are retained. The single failed-job retry passed; all final-head CI gates are green. Both reviews are 5/5 and no review threads remain unresolved. The first local embedded-Postgres launch failed before tests during cluster initialization; the isolated external-Postgres test passed, and its disposable database was removed after correcting the private launcher cleanup call. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 10:08:51 -06:00
DottaandPaperclip	7751fc3107	fix(server): complete volume-free controller admission and recovery (#18543 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its volume-free web server must admit remote warm execution without depending on a local disk. > - Fresh run rows retain the initial legacy label during workspace preparation. > - The native runtime selector commits the execution mode later. > - The earlier guard rejected valid remote runs before that selection. > - This PR checks storage during preparation and the resolved lifecycle during runtime selection. > - Agent-file recovery also has to reconstruct disposable cache paths from the retained remote directory. > - The deployment test removes the complete predecessor home, including immutable caches, to enforce that contract. ## Linked Issues or Issue Description Refs #18518, #18529, #18533. What happened? A real Codex/Daytona deployment test failed during setup because the volume-free workspace guard checked the initial run label. After fixing admission, controller retirement first failed on an immutable skill cache. Removing that cache exposed a production recovery failure: B rejected the saved agent-directory path from A, even though the remote execution identity was unchanged. The next live attempt recovered that directory and transferred the same runner, but adoption then tried to recreate provider launch assets from the deleted controller home. Expected behavior Allow remote workspace preparation before native selection. Reject actual legacy and per-turn execution at the runtime boundary. Delete the entire stopped predecessor home, including immutable caches, without following links to shared dependencies. B must reconstruct its local agent-file projection from the exact registered remote lease while preserving the save baseline and unsaved remote edits. Reattaching an already-running provider must not prepare a new launch or require the old launch files. The full backend must recover the admitted instructions from the matching durable run checkpoint. Steps to reproduce Run the explicit `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and an owned Daytona fixture. The new boundary regressions also reproduce the premature guard and missing runtime rejection without a model call. Paperclip version or commit The failures were observed on `247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on `e122478368`. ## What Changed - Validate Runner and remote workspace authority before runtime selection without trusting the initial run label. - Reject resolved legacy execution and local-source fallback, preserving the native warm-policy check before admission. - Rebind remote agent-file caches on ephemeral controllers and retrieve their validated contents without uploading over the retained execution. Preserve pending saves when the remote collector is unavailable. - Restore over-quota remote directories without applying save quotas to run admission; ordinary saves still enforce quotas. - Skip provider launch asset materialization when adopting an existing Runner. A real-process regression deletes the old skill/instruction sources before reattachment. - Recover admitted system instructions through Runner-owned checkpoint validation for the same run, session and context digest. Fresh launches still compose current instructions. - Make the attention-retention test advance past its database-assigned deadline, fixing a clean-master clock mismatch. - Serialize remote collection and cleanup across controllers with an operation-scoped Postgres advisory lock; fence stale cache-path updates. - Keep synthetic deployment probe tasks in a separate disposable company so they cannot block the business completion oracle. - Share the launcher's directory cleanup with rolling retirement. Reject a symlink root and skip linked descendants. - Test admission with real PostgreSQL transactions and cleanup with real read-only directories and external symlinks. - Update developer documentation and the canonical qualification record. Broader deployment acceptance remains in progress. ## Verification - `pnpm -r typecheck` and `pnpm build` passed. - All 156 workspace preparation, runtime selection and ephemeral contract tests passed. Two new regressions failed before the source fix. - Seven focused Product E2E cleanup/replacement tests and `pnpm test:e2e:runner:typecheck` passed. - `pnpm test:run` stopped at an unrelated 15-second chat integration timeout after 338 passes. That exact test passed in isolation without increasing its timeout. The later full-suite shards did not run. - `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures. Clean master reproduced 11 failures in unchanged onboarding expectations, operating receipts, Pi catalog assertions, workflow bootstrap counts and embedded PostgreSQL startup. The other two workflow assertions were fixed by the intervening master commit. These failures are retained, not counted as passes. - All 122 workspace-preparation tests pass after the reviewed host-fallback fix. The first broader attempt exposed a test environment leak; restoring the ephemeral flag between cases resolved it. - Agent-directory and instruction regressions passed 179 tests. Five focused cache-loss cases pass: different home, same path on an empty filesystem, foreign lease, concurrent controller, and missing collector. - The live fixture at `a18f6b85a3` gracefully stopped A and deleted its entire home, then failed the 420-second remote reattachment deadline on the saved local agent-directory path. That failure is retained; exact owned sandbox cleanup was independently confirmed. The next live attempt at `e14ca5f3e9` restored the cache and transferred the same runner, but failed while rematerializing launch assets from A. Both failures are retained and their owned sandboxes confirmed absent. No active-overlap, ten-deploy, or production-canary acceptance is claimed. - All 71 related Runner adoption/recovery cases pass, including the new removed-context regression (failed before the source fix). - The revised full agent-directory suite passed 110 cases and failed three cleanup timing cases. All 20 cleanup/cache-loss cases then passed in isolation with unchanged timeouts, including over-quota recovery. - Server and Runner TypeScript checks pass after the latest corrections; the live retry remains pending. - The live test at `67714af9ff` adopted the same remote execution after A was deleted, then failed while the full backend read the retired instruction bundle. All owned sandboxes were independently confirmed absent. Its regression now passes with actual backend recovery and no new thread or turn. All 59 backend factory tests and focused checkpoint identity tests pass. Runner and server TypeScript checks pass; the exact committed server build passes. A fresh paid-model retry followed. - The attention-retention CI timeout reproduced on clean master. The deadline correction passes all 53 tests in that file with unchanged timeouts. - The real `cd690de4d3` run transferred the same runner/session after deleting A, completed, and completed the next revision. During handoff, 372 reads and 38 saves had zero failures; all 38 saves were verified. The full attempt was interrupted because the probe's open task contaminated the business company oracle. This remains incomplete evidence. The scope regression fails before the fixture fix, then passes; all nine deployment support tests and E2E typecheck pass. All three owned sandboxes were confirmed absent and temporary credentials/storage were cleaned up. A clean corrected attempt is running. ## Risks - The runtime guard must still reject legacy and per-turn execution before dispatch. Tests cover these refusals and a valid resolved warm policy. - Directory permission changes apply only during cleanup of stopped, test-owned homes. Traversal does not follow symlinks. This changes no production cache permissions. - Cache reconstruction must not change remote execution authority or rebase the original save baseline. Tests verify foreign-lease rejection, preserved receipts, and absence of local-to-remote upload. - Postgres locking is scoped to active remote file work and adds no idle polling; local/durable-server locking remains unchanged. - No schema or public API changes. External conversation exemptions remain unchanged. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest real qualification at `f5915fdc33`: both empty-home replacements completed with 523 reads, 53 saves and 53 verified saves, zero probe failures. The active remote execution survived and completed once; saved project bytes and the later useful answer passed. The full workflow remains FAILED: the later continuation reset its native/provider session and the public run row lacked the native terminal contract. These newly exposed compatibility/projection gaps are tracked in the next implementation slice; this PR is not a claim of full Railway qualification. All owned Daytona sandboxes were independently confirmed absent after cleanup. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 08:24:14 -06:00
DottaandPaperclip	16787e4167	Qualify rolling web deployments and preserve Stop through handoff (#18526 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 02:53:27 -06:00
DottaandPaperclip	006d5691a6	feat(server): support volume-free web server handoff (#18518 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. Current behavior The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. Proposed behavior Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. Reason and benefit This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 01:58:54 -06:00
DottaandPaperclip	75fa37f30f	fix(server): expire native backups from retention deadlines (#18380 ) Reviewed and verified at `5b137b5cda`. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 14:31:31 -05:00
DottaandPaperclip	b7f1c3769c	feat(native): record verified provider-pack session admission (#18469 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions can use an installed pack or a retained earlier pack. > - The current app revision does not identify the pack that a session admitted. > - Core already verifies the selected pack and binds its launch artifacts. > - Normal run records lack a closed receipt for that admitted selection. > - This pull request records that receipt after session admission and attachment. > - Operators can compare the actual Core admission identity with their expected pack. ## Linked Issues or Issue Description Refs #17749 for protected provider-pack selection and #18343 for retained controller recovery. The related OpenCode producer repair is #18456. Those changes do not publish the closed run receipt added here. A search found no existing `nativeProviderPackAdmission` PR. What happened? Core can admit a compatible installed provider pack with a source revision that differs from the current app. A retained session can also keep its original pack. The normal run record does not show the admitted pack digest, proxy digest, and source revision. An expected launch path or current app revision cannot fill that evidence gap. Expected behavior The run has a closed, server-owned receipt after actual session admission and attachment. The first receipt remains immutable history. A separate current-proof state verifies only the actual current admission epoch. Retained recovery needs positive original provenance, and missing legacy evidence stays unknown. Steps to reproduce 1. Select and verify an installed pack whose digest differs from the current app pack. 2. Open the native session successfully. 3. Inspect the normal run profile for a bound admission receipt. Paperclip version or commit The original selected base was `8bcc5a01edd5d87540401521e6f30d3df43e1848`. The branch now includes exact actual prerequisite parent `3ee4c5b6138da82d0b1f51aa010540561bf43c22`; the PR diff against it is the five receipt, executor, test and documentation files. Deployment mode Core remote native execution with provider-pack selection and retained session attachment. ## What Changed - Retain the verified selected pack identity only after a backend returns the matching admitted session. - Publish `runnerProfileJson.nativeProviderPackAdmission` from the session callback after attachment succeeds. - Mark the receipt with `stage: session_admitted` and `authority: core_verified_selection`. - Store bound company, agent, run, session, runner, logical `runnerEnvironmentLeaseId`, pack digest, OpenCode proxy digest, and source revision fields. Retained attachments identify `inheritedFromRunId`. The logical Runner scope is distinct from the physical environment lease. - Lock the current coordinator claim before the profile write. Check owner, attempt, generation, controller boot, lease expiry, phase, and absence of an accepted result. Merge JSON fields without replacing other profile data. - Restore retained metadata only from the exact prior closed Core receipt, positively verified prior state and matching retained binding. Capture same-run prior provenance from the locked claim row before invalidation; cross-run recovery reads only the exact source. Legacy absence remains unknown while recovery continues. - Preserve the first receipt as history. Atomically mark a new opaque admission epoch pending in the existing mandatory claim transaction. Publish current verification only for the matching claim and epoch after actual attachment; changed or unsupported admission becomes unknown without blocking authorized execution. - Preserve synchronous session publication and quarantine ordering before the best-effort receipt write. Fresh replacement selects its newly verified pack; a captured retained-process flag cannot relabel the replacement with an older pack. Preserve the existing private pack authority even when a local dirty revision is ineligible for the closed receipt. - Add focused admission, recovery, identity, PostgreSQL ownership, lock, and JSON-preservation tests. Document the receipt contract. ## Verification - RED: the actual managed-backend path verified and admitted an installed pack but had no closed metadata. Failed open published no evidence. Additional controls fail when retained recovery invents missing history or dirty-revision eligibility gates the existing private authority. - RED calibration: removing mandatory pending invalidation fails the actual pre-admission control; reusing the retained flag for fresh replacement falsely restores the old pack. Removing the epoch fence fails the real PostgreSQL stale-callback case; reintroducing a receipt-specific execution error fails both authorized Runner/logical-lease replacement controls. - GREEN: 49 focused Core tests, all 33 admission identity/transaction tests (20 pure and 13 PostgreSQL), and all 23 existing private-authority tests passed on exact integrated head `165a9096cf3e0d34d2db1e6b52960ad4ac0ad132`. No all-skipped run is counted. - Final test-only head `11ac34baf00552a5d322b5f5f1a8e491859cacec` adds the positive executor-callback SQL-writer wiring control, exact session/identity/claim/epoch assertions, profile preservation, and null-session zero-write check. Removing only the production writer call makes it fail on a missing receipt. All70 focused identity/Core tests pass with this closure; production code, the13 PostgreSQL tests and the23 private-authority tests are byte-identical to the qualified165 source (106 distinct passing controls across the combined proof). - The PostgreSQL tests used an owned loopback-only cached Postgres 17 container with the supported `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` helper and aliases to current DB/shared source. The container was removed afterward. They cover immutable history, JSON preservation, stale claims and epochs, expired leases, terminal runs, changed/unsupported replacements, and a failed write retaining pending proof. - Historical pushed `f7f12fe` CI executed all 24 earlier admission tests, including nine PostgreSQL cases, in server5. Its warm-quarantine failure exposed the new hook-ordering issue fixed here. The same run also encountered a six-character UUID fixture-prefix collision and the independently repaired cached issue-link browser selector; those historical failures are not claimed as green. - Node 24.22.2 and Vitest 5.0.3. The credential-free normal QA runtime import preflight passes with networking disabled. Map contracts, token gates and `git diff --check` pass. - Local TypeScript validation still sees stale borrowed Runner/shared declaration mismatches outside the changed ranges. No diagnostics remain in the new helper or changed test/hook ranges; clean current-head CI is required. - Repository-wide typecheck, full tests and build were not completed locally. A full local executor attempt reached this host's unsupported native PostgreSQL listener and was interrupted; its result is not counted. Canonical run `41214512239`, attempt 2, is successful on exact final head `11ac34baf00552a5d322b5f5f1a8e491859cacec`. All 59 checks are terminal: 55 success, 3 skipped and 1 neutral; none failed or pending. The actual Linux suites execute all 33 admission tests (including 13 PostgreSQL cases) and all 1,022 executor tests successfully. The [original five failures and bounded unchanged-source controls](https://github.com/paperclipai/paperclip/pull/18469#issuecomment-6545593767) remain preserved. One failed-only request was made; GitHub expanded the reusable workflow call and dependents. No second request was made. Fresh exact-head Greptile review is 5/5, all review threads are resolved, and the PR is mergeable. - These local results claim no live provider admission, rollout or recovery. ## Risks - The authority is Core's verified selection plus successful session admission. It is not an independent provider statement or continuous runtime measurement. - Receipt fields cannot grant execution or replace the existing provider, process, lease, workspace, or recovery checks. - Legacy missing receipts and unavailable database reads remain unknown. The current release cannot label an older admitted pack. - The closed receipt requires a clean, 40-character hexadecimal source revision. Verified local `-dirty` packs still execute and retain their existing private recovery authority; their closed admission proof stays `unknown`, so it cannot certify an immutable source revision. - The first receipt is historical. Consumers must require current state `verified`; pending or unknown cannot certify the current pack. Changed Runner scope or pack evidence does not add execution policy. - A receipt save failure leaves the atomically invalidated state pending. The existing artifact authority, cancellation, workspace and process checks remain responsible for execution safety. - The receipt adds no manifest, environment, credential, or filesystem path to the run record. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 10:59:24 -05:00
DottaandPaperclip	baf651c8d0	perf(heartbeat): drive retry promotion from durable deadlines (#18387 ) Reviewed and verified at `f1c48c924d`. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 09:07:52 -05:00
DottaandPaperclip	12d8c3043e	fix(native): fence retries when replaying an audited Stop (#18457 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs keep Stop intent and retry state in durable records. > - A provider error can leave a failed retry with an earlier acknowledged Stop. > - The replay path returned the existing receipt before it closed that retry. > - This pull request applies the existing retry fence to the validated replay. > - The same Stop can settle without another provider cancellation or a new receipt. ## Linked Issues or Issue Description Refs #14733 for the related caller-bound Stop contract. This change covers a later retry that already has a Stop acknowledgement. The older CI-only #14743 is not a production delivery PR. What happened? A failed native run can have a retryable coordinator and an earlier audited Stop. A repeat Stop returns the old acknowledgement without moving the coordinator to `terminal_failure`. The final failed-to-cancelled transition requires that fence, so the Stop cannot settle through that path. Expected behavior A valid replay closes the same retry and keeps the original intent and acknowledgement. It does not send a second provider cancellation. Physical stop and workspace cleanup still require their own evidence. Steps to reproduce 1. Store a failed native run with a `retryable_failure` coordinator. 2. Store a matching run-scoped Stop with a dispatched acknowledgement and audit IDs. 3. Call `cancelNativeSession` for that same run and inspect the coordinator. Paperclip version or commit Reproduced on public master `f2db909393de124caf6a5380a52d4e0e1d621661`. Deployment mode Core native execution. The regression uses the real cancellation executor with controlled persistence and provider boundaries. ## What Changed - Reuse the existing guarded coordinator fence for a validated Stop replay. - Test repeated replay, unchanged receipts, one fence, and no second provider cancellation. - Reject foreign company, run, issue, scope, effect, and missing-audit receipts before any write. - Check that an acknowledged Stop with an unconfirmed remote lease still fails the physical-stop guard. - Document the separate retry fence and physical-cleanup requirements. ## Verification - RED on the selected public base: the new executor case returned success but left the coordinator in `retryable_failure` with a future retry. One case executed and failed for that assertion. - GREEN: 62 focused tests passed across `native-session-executor.test.ts` and `native-cancellation-request.test.ts`. These include ownership, caller conflict, changed-phase, and receipt controls. - Command from `server/`: `node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-cancellation-request.test.ts -t 'native session cancellation\|caller-bound native cancellation\|failed native retry caller admission'`. - Canonical Linux CI passed all 1,018 executor cases and all 32 caller-cancellation cases. The exact file completion lines are in [server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41203989386/job/124545366271) and [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/41203989386/job/124545366219). - Node 24.22.2 and Vitest 5.0.3. Map contract checks and `git diff --check` pass. - Local dependencies are linked from an existing checkout. Full local TypeScript validation reports older Runner/shared declarations that lack current Kimi and continuity APIs. The changed ranges have no diagnostics. The canonical clean CI Typecheck + Release Registry gate passed. - A broader two-file run was stopped when an unchanged integration case tried to start embedded Postgres and hit the host listen restriction. It is not a passing full-suite result. No host settings or other services were changed. - Repository-wide `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` were not completed locally. Current-head CI passed against clean source and dependencies. - Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/18457#issuecomment-6543772487) on `dbc1c1a5b71450194d568725592969660864cb5a`: 5/5 with no actionable findings. The review-thread inventory has no threads and no pagination. Full [current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41203989386) passed: 59 terminal checks, 55 successful, 3 skipped, and 1 neutral. No failure or pending check remains. Merge status is CLEAN. No live recovery, rollout, or physical termination is claimed. ## Risks - A wrong or stale Stop must not close another retry. Existing run/coordinator locks, caller checks, receipt binding, and the final compare-and-set remain in place. - This fix changes logical retry settlement only. Provider termination, workspace preservation, lease cleanup, and continuation gates remain separate. - The regression controls persistence and provider boundaries. It does not establish the physical state of a remote provider. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 08:25:16 -05:00
DottaandPaperclip	492d537340	fix(native): recover controllers across webserver replacement (#18343 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description What existing behavior does this improve? Native remote controller recovery and retained-session continuation during a webserver upgrade. Subsystem affected server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. Current behavior Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. Proposed behavior Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. Reason and benefit A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. Breaking changes No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. Additional context Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component. Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes steady PASS and burst FAIL. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exact `fc68584`, with the matching server build stamp. The final paired native feature-cost comparison explicitly uses the same cap-three policy and common current DB factory in both variants; its baseline is controlled, not the unmodified historical cap-two configuration. Original gates and workload counts remain unchanged. The one strict Linux/Daytona case started at 04:40:56 UTC only after both new profiles passed; it passed on the first attempt with all 49 workflow checks. Independent cleanup confirms all three owned sandboxes absent, owned Postgres/controllers/observer gone, VM terminated, root volume absent and ephemeral SSH key/agent removed. Final controlled cap-three qualification: PASS. Baseline native implementation `59da15066a` and candidate `fc68584ce6` use the same current DB factory and checkpoint policy (request/liveness/checkpoint = 10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000 historical tasks. All correctness, absolute and 10% relative gates pass; original cap-two failures remain unchanged. \| Mode \| DB CPU µs/op, baseline → candidate \| WAL B/op, baseline → candidate \| DB-request p99 ms, baseline → candidate \| Maximum drain \| \| --- \| --- \| --- \| --- \| --- \| \| Steady \| 542.40 → 554.83 (+2.29%) \| 1509.01 → 1558.27 (+3.26%) \| 107.11 → 111.90 (+4.48%) \| 26.35 ms \| \| Burst \| 1036.36 → 1063.11 (+2.58%) \| 4153.12 → 4317.64 (+3.96%) \| 103.736 → 103.841 (+0.10%) \| 930.14 ms \| Every matched arm preserves operation/statement/transaction counts, including 3,900 PRP and 3,900 snapshot transactions. All three actual pools are included; there are zero lost owners, missed lease deadlines, dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses one Node process simulating 300 controllers; it does not qualify two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL contributes to measured database CPU. The original unmodified cap-two baseline is separately retained as failed; this cap-three comparison is explicitly controlled. The exact final-source 8/512/8,192-chunk test also passes: zero recovery SQL, transactions or changed rows in each text-only interval, zero idle SQL, seven lifecycle transactions and twelve changed rows per sample. Existing lease traffic remains separate. The final ordinary managed application replay passes active recovery, completion, native-cache removal and C continuation; its UNKNOWN local debug-daemon provenance and partial instruction-observer limits remain recorded. The live case uses the attested Linux producer instead. Live qualification: PASS, one attempt, no retries. Campaign `native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started 04:40:56 UTC on the exact final Linux webserver source; suite definition hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`. The existing immutable image and attested Bookworm Runner artifact are unchanged. - A was SIGKILLed and its native cache discarded. B started after 300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms and resumed at 317,682 ms within the original 420,000 ms grace. The same active run, Runner, provider PID/start fingerprint and one submitted/started/accepted turn were preserved; one final result/terminal and workspace finalization completed, with no retry/recovery action. - After active and queued completion, B was SIGKILLed and its native cache discarded again. C received exactly one ordinary task-comment API follow-up, completed a new tool and produced a useful reply while preserving native/provider/Runner identities. This closes the earlier failed post-C gate. A separate C OS-process PID field was not exported; the physical-PID proof is specifically A→B. - Four distinct results were accepted and published; all four finalizations committed at attempt 1, all four accounting-v2 projections settled, and the task-session pointer advanced to C. The coordinator independently checked these facts and all 49 canonical checks against the sanitized receipts. - Canonical cleanup passed and independent checks confirmed three exact sandbox 404s plus complete inventory/zero owned remnants, no owned PG or controller/observer processes, EC2 terminated, root volume absent, and the ephemeral SSH key/agent removed by 04:54:25 UTC. Live scope is classic remote Codex warm continuity with non-native storage dependencies available. Prepared-task completed B→C, full per-turn application disk loss, browser reconnection, two-server load and overall Railway/HTTP continuity were not live-qualified here. Earlier failed campaigns and the original cap-two burst failure remain failed historical evidence. A report-renderer cache setup failure was preserved; the normal sanitizer subsequently succeeded with an explicit cache path and a new output directory, without changing validators. The repair keeps retained events under their original six-field authority until persistence and accounting accept them. Unknown semantic effects stay unresolved. A compact staged target survives source bootstrap, cached-command delivery and authenticated handoff, including an uncertain commit response. Direct and same-guest-listener recovery require the verified original Runner and the exact endpoint. Target snapshot publication marks readiness in memory, with retirement coalesced into an existing command write. Headers contain only the staged descriptor/digest; a prequeue template is written once as a changed checkpoint part and then recovered from the existing command/receipt. No second session snapshot transaction, streaming SQL path, new pool or recurring idle scan is added. \| Current repair gate \| Evidence and scope \| \| --- \| --- \| \| Source-tail regression \| Real Runner protocol RED reproduced attachment rejection. Frozen core/transport pass five source-tail cases: source usage attribution, held persistence/no premature ACK, lost commit response plus cache loss, unknown semantic hold and accounting rejection hold. \| \| Handoff loss windows \| Direct and listener modes pass queued/prepared/activated/completed boundaries (eight cells), preserving the actual Runner/provider process. Missing adoption proof and changed endpoints reject recovery. \| \| Compact storage \| A 31 KiB attachment grant writes its separate part once, leaves it unchanged across source snapshots and removes it at queueing. Headers do not duplicate the grant. \| \| Accounting \| 21 real-Postgres usage tests and 19 source-binding tests pass. Recovery hydrates immutable source receipts without resetting their producer or billing classification; conflicting/incomplete usage holds. \| \| Admission and malformed evidence \| Driver 16, staged evidence 42, restart classifier 27 and source usage 19 focused cases pass. Four malformed-file regressions pass at d6a7, whose executor is byte-identical at 484, and server typecheck passes. Full executor file: 990 pass; 16 embedded-Postgres `initdb` setup failures before assertions, retained separately. \| \| Required checks \| Exact `fc68584` recursive typecheck and build pass locally, with matching server stamp. Forced full CI succeeds (48 jobs pass, one intended skip); all PR checks are clean. Maps/token gates pass for their unchanged source surfaces. Fresh exact-head Greptile is 5/5, all six threads resolved. \| \| Ordinary application path \| Exact `fc68584` managed component passes active recovery after natural expiry, completion, native-cache loss and ordinary C continuation. Production bytes match before/after; temporary fixtures restored and owned DB/services cleaned. UNKNOWN local debug-daemon/partial instruction-observer limits remain; the separate live case uses the attested Linux producer. \| \| Producer \| Locked Bookworm build from exact 5b7 Rust tree `d03e494217`, equal to final `fc68584` Rust source. SHA-256 65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf; 38,515,328 bytes. Actual Debian 12 execution and live admission pass. Controller/admission/DB changes do not relabel the producer source. \| \| Historical cap-two comparison \| Exact 59da baseline vs 5b7 candidate, 300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%); WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms (+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass relative gates, but absolute drains above fail. Identical operation/SQL counts, zero lost leases/drops/errors. One Node process simulates 300 controllers; this does not qualify two-webserver overlap or HTTP capacity. \| \| Live acceptance \| Strict final-source Linux classic Codex case passes 1/1 on attempt 1, all 49 checks. Active recovery within grace and useful retained C continuation succeed; four accepted/published/committed/settled outcomes. Independent provider, PG/process, VM/root and SSH cleanup complete. Prepared/task and broader rollout limits are explicit above. \| Historical qualification snapshot through `9eb9ac6`. The head at this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and Runner package sources are byte-identical to `283e05ffaa`; the two later commits change the explicitly scoped qualification and its documentation only. It includes master through the storage-owner fix `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot, CI/review passed but the strict post-C native recovery gate failed; the PR stayed draft and required another native repair. Prior `39a4343` CI and review passed, but its full live workflow failed; that grade remains unchanged. The first repaired `435ccd2` live attempt failed before any injected outage. It exposed a legitimate terminal snapshot before result acceptance; no accepted result ID or publication receipt was observed. Its canonical report remains 0/1 `candidate_failure`, with cleanup passed and independent provider absence confirmed. `0862062` defers publication until result acceptance and fixes the fresh review's coordinator/issue lock-order finding. Both failures have deterministic real-Postgres regressions. The `0862062` live attempt then proved accepted-result publication but failed active recovery: a resumed-lease preparation guard demanded suspension after the exact live Runner was already verified for adoption. `64c0b746` preserves that verified active/idle-warm process, revalidates its proof/liveness, and keeps ordinary suspension checks and later PRP authentication. Its live run passed preparation but exposed a second active-warm boundary: all durable binding fields matched except opaque turn/item attachment IDs, which the server had regenerated. `7a203274` carries the complete verified six-field binding and keeps the strict Runner mismatch guard. Its cloud case then passed active adoption, completion, publication and accounting. The queued continuation exposed accepted-result retirement writes and an implicit-home fingerprint drift; both are fixed with deterministic regressions in `39a4343` before this fresh full qualification. The prior live case exposed a real publication gap. A goal-less native turn could persist its accepted result and terminal snapshot before provider cleanup, while the task-session pointer was still empty. Normal later admission then selected a fresh session. The fix publishes that pointer and a private bound receipt in the existing post-acceptance terminal checkpoint transaction, before cleanup. Earlier terminal observations persist without a handled receipt or continuation pointer. No result-acceptance or snapshot transaction was added. The late completion path recognizes the receipt. This also prevents a late callback from restoring a pointer after a user reset. Goal/control publication keeps its existing owner. The dedicated Daytona fixture now uses the existing warm/reusable environment. Its sandbox, Runner and provider storage must survive. The former generic per-turn fixture disposed the sandbox after completion and kept provider-history backup bytes under the discarded directory. Whole per-turn disk-loss acceptance therefore has a separate provider-backup storage dependency. It remains distinct from the passing native per-turn component checks with backup storage available. The 60-second lease, 300-second outage and 420-second grace are unchanged. \| Gate \| Current repair evidence \| \| --- \| --- \| \| Assembled regression \| Baseline fails with an accepted result and terminal snapshot but a NULL task pointer while real provider cleanup is held. The fix passes normal continuation and reset cases, with no pointer seeding. Exact merged `92b7447` passed 2/2 twice, including after normal package TypeScript output refresh. Production files are byte-identical at frozen `435ccd2`. \| \| Persistence and concurrency \| 19 helper/transaction cases and all 59 existing control-port cases pass on `0862062`. They cover bound result IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay, missing stubs, foreign bindings and atomic 4 KiB rollback. Existing snapshot transaction count stays one. The new tests reproduce the exact live missing-result-ID error and the inverse replay/status lock order against `435ccd2`, then pass after the correction. Replayed results still reject stale generations. \| \| Completion \| All 32 completion-boundary tests pass on merged `92b7447`; the source is unchanged at `435ccd2`. Native result-shaped session metadata is preserved. \| \| Qualification fixture \| A valid review report reproduced three rejected correct replies. Exact token-boundary matching fixes them. 140 focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens, marker prefixes/suffixes and wrong authors remain rejected; retained-identity checks are unchanged. \| \| Required checks \| Full recursive typecheck and standard build passed at `435ccd2`. Both exact commands also passed on `0862062`, with its exact server build stamp; both also pass on `7a203274`, with its exact server build stamp. Fresh `0862062` Greptile passed 5/5, with zero unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48 success, one expected skip) and Greptile passed 5/5 with zero unresolved findings. The `7a203274` normal CI also passed 48 jobs with one expected skip, with fresh Greptile 5/5 and zero unresolved findings. Current `39a4343` recursive typecheck and standard build both pass with persisted exit codes and its exact build stamp; fresh Greptile is 5/5 with zero unresolved findings. Its normal CI completed successfully: 48 jobs passed and one expected job skipped. The fresh live result remains separately failed. The `0862062` normal CI had 46 passing jobs, one expected skip, one worker communication failure and its aggregate failure; GitHub states the self-hosted runner lost communication. No failed assertion or specific OOM/spot cause was established, and no blind retry was used. \| \| PostgreSQL supplement \| One corrected all-pool pair at the preceding repair passed 300-controller/10,000-history correctness and absolute gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613 each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL 70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a repeated improvement claim or an exact `0862062` CPU measurement. Its two later changes defer premature publication and reorder existing locks; they add no transaction, stream/idle writer, pool or timer. The first collapsed-pool attempt remains invalid and retained. This is not an exact final-head CPU/WAL measurement; later fixes change ordering, identity selection and in-memory skill replacement without adding snapshot transactions, pools, stream writes or idle SQL. \| \| Live publication proof (`0862062`) \| Baseline run `06fcc806-f186-4190-9b57-087b61d75533` accepted result `58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names the same result, the task pointer has parameters and `lastRunId` matches, and the normalized native identity matches the following active run. Accounting v2 settled. \| \| Active recovery (`0862062`) \| FAIL. SIGKILL/cache discard at 23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms; natural generation-2 ownership at 312,906 ms. At about 319 s, preparation failed `runner_harness_state_mismatch: resumed_not_suspended`. No authenticated after-receipt passed within 420 s. Exact original Runner PID 668/start/fingerprint was independently still alive; provider auto-stop was 15 minutes. No retained C follow-up qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains failed; subsequent independent cleanup deleted all three owned resources with exact 404s and confirmed full intended-account inventory with zero owned resources. \| \| Verified adoption preparation (`64c0b746`) \| 91 focused cases pass, including actual preparation for four active providers and a distinct prior/current warm proof. Unproved resumed state, dead process and stale warm owner remain rejected. The earlier options-only fixture missed this gate. No spawn, backup restore or PRP-authentication bypass was added. Server typecheck passes. \| \| Active warm identity (`64c0b746`) \| FAIL. B claimed generation 2 at 312,995 ms. Preparation passed, then strict PRP recovery rejected the generated attachment IDs at about 319 s. The saved and desired Runner, lease, normalized session and run IDs matched; only `turn_attachment_…`/`item_attachment_…` differed from generated `turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains failed; independent cleanup confirmed deletion/404s and zero owned provider resources. \| \| Six-field identity repair (`7a203274`) \| Actual server-factory regression RED4→GREEN38. A real Rust Runner test passes healthy first run → random-ID warm handoff → held active second run → restored controller. It verifies a new live authenticated connection, exact identity, same Runner/provider OS identities, no duplicate launch/turn, and no redispatch or invented receipt for an uncertain tool. The synthetic wrong pair still fails the unchanged guard. This protocol case does not claim business completion. Server and Runner typechecks pass. \| \| Active recovery/completion (`7a203274`) \| PASS for the active component. Full five-minute SIGKILL/cache-loss hold. B ready at 308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated at 317,720 ms and resumed processing at 318,527 ms. Same run `1e1ccb00-0728-4935-97da-57562d81cb42`, Runner `246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start 23:48:27.470, provider lease/session and one submitted turn. Accepted result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization and accounting v2 all settled. \| \| Full workflow (`7a203274`) \| FAIL, preserved. Queued late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed `runner did not durably suspend before checkpoint` before outage 2. Canonical 0/1 candidate failure, cleanup passed, evidence valid. C retention was not exercised. Independent provider absence, controllers/PG cleanup, EC2 termination and root-volume absence all confirmed; closed receipt SHA-256 `676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. \| \| Retirement and stable configuration (`39a4343`) \| Real Runner + Postgres RED reproduced the exact suspension failure and nine owner rejections. `turn.stop` is now lifecycle checkpoint maintenance under unchanged owner/result/generation fences. GREEN25/25 proves actual persisted suspension, exact identity and completed stop/drain/suspend; unaccepted, foreign, stale and new-work negatives remain rejected. Two default-home fingerprint REDs become GREEN9/9; explicit mode changes still invalidate. No global matching exemption or historical fingerprint rewrite. \| \| Fresh full Daytona warm cell (`39a4343`) \| FAIL, preserved. Launched 00:15:29 UTC with automatic retries disabled. Active adoption passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms; natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed 330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner `4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and provider/session identities. Accepted result `08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict suspension error before outage 2. Its row selected a new native identity, but no new third provider process or invocation was observed. Recovery-time configuration still reset only `adapterConfig`; native provider, context and workspace subtrees were unchanged. Canonical 0/1 candidate failure, evidence valid, normal cleanup passed; all campaign provider/PG/process resources independently absent. Closed campaign receipt SHA-256 `5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`; EC2/SSH remain temporarily held for the concrete follow-up repair. Original 60/300/420-second limits and strict queued/second-replacement checks remain unchanged. Direct-process path; no new live prepared-task claim. \| \| Native retirement authority (`283e05ffaa`) \| Exact `39a4343` facade regression RED3→GREEN9. The successor now uses its existing production coordinator claim and prompt renewal before strict retirement. A real Runner/Postgres case starts in the actual native-observed/unclaimed phase, claims through production code, then closes and durably suspends the predecessor in 1.657 s. All 25 real-Postgres cases pass; unclaimed/foreign writers remain rejected. Shared attachment staging stays after retirement; Stop and denied claims neither close nor stage. No new claim or SQL path was added. \| \| Configuration continuity (`283e05ffaa`) \| Real shared/project-primary environment selection → configuration with four skill versions reproduces `reset: true`, changed category exactly `adapterConfig`, after a Postgres JSONB read. Pinned restoration had reordered the generated preference while actual runtime skills stayed identical. Replacing existing pinned rows in place preserves the original fresh name/key order and fingerprint. The assembled regression passes, as do three existing pinned-version/rename/alias cases and server typecheck. Genuine model/skill-selection changes still invalidate. No fingerprint-format change or broad compatibility exemption. \| \| Final production repair (`283e05ffaa`) \| Full recursive typecheck/build and [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106) PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved threads. Live baseline, recovered run and queued continuation all succeeded, each accepted/published/committed once with accounting v2 settled. Their native/provider identities and configuration fingerprint match; both recovery and continuation report no reset, no changed categories and task-session reuse. Active outage/authentication remains within original timing. \| \| Browser-dependent continuation (`283e05ffaa`, definition 6) \| Whole cell FAIL, preserved. After B was killed at 00:46:38.928 and its cache discarded, C became ready in 9,151.410 ms. `openParent()` then hit a 30-second `page.goto`/DOMContentLoaded timeout before posting the final request. No fourth native run or C continuation was exercised. Cause of navigation timeout is unknown. C task had no execution lock, queued comments, handoff or recovery blocker; all three prior runs were committed/accounted. Normal cleanup passed; independent provider absence, controller/PG cleanup, EC2 termination and exact root-volume absence confirmed. Sanitized report bundle SHA-256 `2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. \| \| Native API continuation qualification (`9eb9ac6f9e`, definition 7) \| FAIL, preserved. The real post-C API request dispatched run `c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed `native_session_interrupted` / `native_session_recovery_failed` at 01:10:00.640 UTC. It selected the same normalized native identity, Runner and checkpoint-provider identity as the three succeeded turns. No C result was accepted; pending accounting remains explicit. Active recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019 ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms, resumed 316,612 ms. Queued continuation succeeded/accepted/published. B was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803 ms. Original classifier `transient_infrastructure`, whole-case failure and cleanup passed are unchanged; that classifier does not establish the root cause. All owned provider/PG/process resources, EC2, root volume and SSH agent independently absent. Closed receipt SHA-256 `7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`; sanitized bundle `71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`. Exact backend detail is unavailable in approved structured evidence. \| \| Current required checks (`9eb9ac6f9e`) \| Recursive typecheck, standard build and exact build stamp PASS. [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417) PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited summary checked, zero unresolved threads. Exact8/512/8,192 stream probes again report zero recovery SQL/transactions/rows and settled-idle SQL; seven control transactions/twelve changed rows each, with 36 separate legitimate renewals in the longest case. CI success does not supersede the failed live C gate. \| The completion supplement observed secondary request/liveness/checkpoint transaction p99 increases of 23.0%/45.7%/15.6% in its single pair. Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm and no lost leases. The named pool caps stayed 10+2+2. An aggregate activity sample peaked at 15 non-observer entries without backend-type classification, so no claim is made that all were client connections. No sampled lock episodes does not establish zero lock waits. Twelve pure reader/selector cases also pass with the new adjacent receipt. This is not another executed process rollback. Development reader `5cf7aead` can parse the recovery state, but its old run-detail redactor can expose receipt IDs/outcome to an authorized run reader. The first production release containing recovery readers, complete attachment-identity handling and both private-field redactions is the rollback floor. The implicit-home fix prevents new recovery-time fingerprint drift; it does not rewrite previously stored fingerprints from failed candidates. The development probe is not a published supported rollback target. Latest actual `0862062` Runner tests independently passed 8/512/8,192 text chunks with zero recovery SQL, transactions and changed rows, plus zero settled-idle SQL. All three observed seven setup/control transactions and twelve changed rows. The 8,192-chunk case separately recorded 31 legitimate lease-renewal statements, with no lost lease. These are component SQL observations, not a production CPU/WAL claim. The new adoption and identity branches affect recovery preparation only. The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed all three stream sizes with the same zero-SQL/idle result; each latest 8,192-chunk case recorded 36 separate renewal statements. Exact `39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its separate native group passed 150 tests plus eight skips. The first API fixture revision omitted the mandatory final screenshot; this was caught before paid launch and allocated no provider/model work. The correction captures the actual retained Chromium page with an explicit unqualified-browser label, without changing the standard report validator. Automatic approval review rejected exporting raw private API/workflow snapshots because of sensitive payloads and insufficient destination-specific authorization. Sanitized reports and bound/hash-only receipts are retained; raw snapshots were not copied through another channel. This does not change the failed grade. Repair characterization also preserves its failed attempts. An unconditional staging relocation broke three existing same-scope scripted cases; the final implementation defers staging only for an actual superseded warm scope, and all nine order/race cases pass. One baseline transform overlapped an edit and is explicitly invalid; the actual immutable `39a4343` loader then produced the three genuine REDs. The independent task-directory hypothesis produced a RED but was excluded by the live shared-workspace evidence; no directory policy change was landed. A shared-workspace fixture authorization omission was corrected separately from the four-skill semantic RED. Setup failures are retained separately from semantic outcomes. These include an old host port after the owned Postgres fixture restarted and a sandbox-denied TSX IPC pipe. The supported external-Postgres run passed, and the final exact recursive typecheck passes with normal process permissions. Seven stale declaration errors after the master merge cleared after compiling the actual merged SDK/Runner TypeScript sources. No declarations were edited by hand. <details> <summary>Earlier qualification measurements and failed attempts (historical, before this repair)</summary> Historical blocked gate (before the repair below). Final native implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later commits change only qualification fixtures, their matcher and documentation; `server/` and `packages/` are unchanged. It includes master through `fd008979f5794fda3d5f54151d36acd31d0de140`. The live Daytona measurement proves five-minute active recovery, once-only completion and accounting, but fails retained-session identity after the second replacement. The saved checkpoint and recovery metadata exist. The task-session pointer remains empty, and ordinary admission creates new native/provider sessions. The exact skipped publication boundary remains unproven. A useful saved-file result alone does not satisfy retained-session continuation. This application-path gap blocks merge independently of CI. A separate new-master CI failure is owned by the storage workstream (fix in #18357). No storage guard, session-identity oracle, lease/grace limit or provider result was relaxed. \| Gate \| Observed result and exact scope \| \| --- \| --- \| \| Distinct source recovery and compatible rollback \| PASS at A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B `5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four controller processes, natural 60-second lease expiry, absent native projections before replacements, three succeeded/committed unique results, one provider thread, unchanged Runner identity, and strict idle cleanup. 177.782 seconds. \| \| Immutable installed native component \| PASS at installed A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B `eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each controller imports its contained compiled server and DB. Three unique committed results, stable Runner/thread, no extra provider restart on same-contract rollback, strict cleanup. 106.672 seconds. This is compiled component proof, separate from full CLI-native application acceptance. \| \| Normal public CLI startup \| PASS for B `eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64: fresh external DB, 415 migrations, health ready, root HTML 200, company POST 201 plus GET readback. The installed tree remained unchanged after execution. \| \| Five-minute component outage \| Actual controller absence 300,168 ms. Natural generation-2 claim at 407.196 seconds after crash, same provider attempt and Runner, original run succeeded/committed. The larger 540-second case timed out before its retained follow-ups, so it is an active-adoption pass, not a complete four-controller pass. The complete short source/installed cases above independently cover follow-up and rollback. \| \| Drain/resume review regression \| Fixed in `226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases pass: overdue normal/restart waits, future timer cancellation, another facade resuming the original owner, and transient retry. Against exact prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail and one passes. The later head changes documentation only. \| \| Final corrected-fixture Daytona \| PARTIAL: active recovery PASS; retained identity FAIL, exact live source `0b6afc499f`. A was SIGKILLed and native cache discarded; actual offline hold 300,000.375 ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at 312,059 ms, authenticated at 315,996 ms and resumed processing at 316,519 ms, within the original 420-second boundary. The same run, Runner PID/start, session, lease and event prefix survived; one turn submitted. Original execution succeeded and settled accounting. After completion, B was killed and its native cache discarded; C was ready in 9,077.779 ms. C executed the saved project and returned correct `queue_r`, with all four runs succeeded/accounted. However, native and true provider-session identities changed: the unchanged retained oracle fails. No whole-workflow pass is claimed. \| \| Prior matched-producer Daytona \| FAILED at exact `abed921e12`. The matching current-source Runner passed normal admission, completed the first task and settled its accounting. During the active revision, A was SIGKILLed and its native cache discarded. Actual offline hold was 300,000.727 ms; B was ready at 308,304 ms and naturally claimed generation 2 at 309,651 ms. At about 318 seconds, the remote process check reported `process_not_running` for the original admitted Runner. No authenticated after-receipt or retained follow-up passed. The current failure is preserved; no automatic retry, lease/grace relaxation, debt edit or provider replay is used. Bounded forensic read found preserved exact-bound state with no recoverable failure, a transport-reset diagnostic and no terminal delivery; this supports external process loss without proving the exact cause. The sandbox had a five-minute auto-stop policy, conflicting with the required surviving-sandbox assumption. Owned cleanup precedes the separate corrected-fixture case. \| Live retained-session boundary. Prior completed run `3806…` and C follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database session snapshots and `nativeControllerRecovery`. Their tool fingerprint, provider configuration, workspace descriptor, runtime-context digest, driver and work mode match. There is no recorded force-fresh/workspace-selection change, and this is an ordinary task rather than a conversation-generation reset. The same issue/agent task-session row has `session_params_json: null` and `last_run_id: null` from baseline time. Both native normalized UUID and actual provider session/thread hashes differ after C. Full application admission therefore did not select the retained source. No fallback to file execution or relaxed identity comparison is accepted as proof of continuity. Independent cleanup completed. Full current-account provider inventory confirms no resources owned by either final fixture, with exact-ID absence checks. Owned controllers and campaign Postgres are stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its delete-on-termination root volume returned `InvalidVolume.NotFound` at 22:14:08 UTC, and the temporary SSH key/agent were removed. The canonical failed report and immutable oracle input are retained. Closed qualification receipt SHA-256: `b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`. Original measurement and grader repair remain separate. The C reply contains the requested marker followed by a newline and the correct output. The old workflow waited for one literal space, so its normal receipt collection stalled; that original attempt was closed and its canonical `cleanup_failure`/interrupted result is preserved. Commit `af9e2d157d` repairs only this reply predicate, with 144 focused cases, E2E typecheck, discovery and diff checks passing. Ordinary string matching stays literal. The corrected predicate accepts the saved reply, but the unchanged retained-identity oracle still fails. Pure active-recovery grading of the captured measurement passes 7/7; this is not a rewritten passing campaign. The sandbox idle-window conflict was corrected only for this explicit fixture (15 minutes). Actual baseline and later sandbox metadata confirm that policy; the original active box was disposed before a post-gate policy read, so its 15-minute setting is established by the normal create/environment configuration rather than an after-read of that box. Production native lifecycle and the 60/300/420-second constants remain unchanged. The development compatibility reader A is a genuine backport commit based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required idle-goal reader fixes. It is not a previously published release. Its older executor differs from B. The rollback floor must include this reader capability. The source component cases admitted the Linux ARM64 Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the installed component case admitted the `5cf7aead` package. Their Runner bytes are identical: SHA-256 `b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each admitted artifact remained unchanged throughout its case. [Final public package build](https://github.com/paperclipai/paperclip/actions/runs/41121460635) and [reader-floor build](https://github.com/paperclipai/paperclip/actions/runs/41122170506) both passed all four daemon builds, package assembly and both Darwin clean consumers. Both Linux ARM consumers failed before provider work with the same existing local Codex permission-profile error: bubblewrap attempts `activity.fifo/.git` and reports “Not a directory.” This matched baseline failure remains open; it is not a native remote continuity pass. Full post-run tree and build-stamp revalidation also passed after the compiled native case. The installed component trees are A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`. Controlled PostgreSQL qualification passed. Baseline/candidate/candidate/baseline workloads used 300 controllers, 10,000 historical tasks, real snapshot/encrypted PRP writes and unchanged pools. PostgreSQL and client were each limited to two CPUs and 1536 MiB on an isolated eight-vCPU Linux host. \| Workload \| PG CPU per operation \| WAL per operation \| Database request p99 \| \| --- \| ---: \| ---: \| ---: \| \| Steady \| +1.18% \| +1.81% \| +1.91% \| \| Aligned burst \| +3.40% \| +2.56% \| +3.81% \| \| Two logical servers \| +3.11% \| +2.56% \| +9.68% \| All arms passed absolute/correctness gates with unchanged control transaction counts, no dropped operations and no owner loss. Overlap p99 has little margin below the relative threshold. It uses two clients in one Node process, not two HTTP servers. [Closed measurements and limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json) and exact-source reproduction patches are retained. [Earlier contended-host failures](doc/plans/2026-10-30-native-controller-load-evidence.json) remain unchanged. The production DB-claim probe claimed 300 owners in 2.84 seconds and restored/committed controls and snapshots in 8.74 seconds, with zero owner loss or settled-idle SQL; its provider references were synthetic. The measured checkpoint/snapshot/renewal paths retain the same persistence behavior after integration; later accounting and business writes have separate owners. Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487` passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed rows during text intervals; zero settled-idle SQL. Each retained seven control transactions and twelve changed records. Existing renewal SQL was measured separately. The guard/scheduler fixes add no stream path, pool or recurring idle scan. The live fixture used current-source producer SHA-256 `f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a normal locked exact-head build with pinned Rust 1.97.1 in Bookworm. Actual network-isolated, read-only ABI execution passed (GLIBC requirement 2.34 against runtime 2.36). The immutable base image remained `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`, source `a0ef3ce7`. This was an unpublished development producer, explicitly distinct from the base image and admitted before the fresh fixture began. The earlier host build failed ABI preflight before admission. The preceding old-producer fixture completed its initial task but lacked the usage-basis marker required by current master; it remained pending/unpriced and correctly paused. It was independently cleaned without debt or budget changes. Two preceding local Linux setup attempts OOM-killed before provider allocation, and their cleanup passed. These attempts remain separate from the final recovery failure. Failures retained and resolved separately. The earlier `98f05fc` probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that reader and the fresh follow-up passed. Later component cleanup failures were traced by a scoped exception observer to the fixture adapter rejecting directory downloads. A bounded owned-directory transfer with exact-byte/mode preflight corrected the fixture; source and installed strict-cleanup cases then passed. No production guard was relaxed. The initial local integrated control-port run hit its 30-second embedded Postgres startup hook before executing 59 cases; 18 other focused cases passed. A bounded rerun used a fresh owned external Postgres fixture, applied normal migrations before Vitest, and passed all 59 cases with the original timeouts unchanged. Its container and data were removed; the failed attempt remains retained. The additional opt-in full Product E2E Vitest run is not green: 5,313 passed, seven failed and three skipped. An actual `abed921e12` archive with the same dependencies, Node and IPC permissions reproduces all seven failures with the same reasons. Five are unchanged workflow-policy/profile-pin assertions, including Pi profile version 37 versus the test pin 35. Two database suites failed during embedded Postgres startup before their assertions; semaphore exhaustion is confirmed in the observer log. The observer passes its supported external-Postgres rerun with original timeouts and source unchanged. The controller-database client-kill/postmaster assertions remain unexecuted after `initdb` bootstrap failure; no external override is supported. The unit launcher stopped before its Node-test phase, which was then run independently and passed 181/181. The original failed full run is preserved. These results remain distinct from the passing focused checks and normal repository CI. The first local Product E2E typecheck also encountered stale compiled Runner declarations; a genuine package-local TypeScript compile refreshed them and the unchanged check passed, without a dependency install or producer rebuild. Earlier [Daytona campaign 41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900) failed at sandbox creation before any agent, with the underlying provider error unavailable. Its [sanitized report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html) remains a `cleanup_failure`. Both old creation names are absent in the current authenticated account, but the old CI account identity was not retained, so account equivalence is unproven. The subsequent exact `eaef0d5` source Product E2E attempt passed image/agent setup and the initial project, then held the controller offline for 300,003.7 ms. Its Mac supervisor failed owned-process inspection before B readiness; no adoption passed. Full current-company lease/label checks, provider deletion, exact-ID 404s and complete inventory confirmed cleanup of that attempt. Its local server and owned Postgres stopped. The prior AWS campaigns and database qualification VM were terminated with root volumes removed. The old admitted test Runner lacks the newer usage-basis marker introduced on master. The integrated component B turns retained unpriced, incomplete accounting obligations with `costAccountingPending: true` and no settled charge. The compatibility-reader A turn used its legacy receipt interpretation but likewise retained pending accounting and no settlement timestamp. This proves obligation preservation, not a whole-run accounting migration. Runner/receiver release ordering remains governed by the usage owner's contract. At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`, Greptile was 5/5 with zero unresolved threads and CI completed with 54 successful checks and five expected skips. At `0b6afc499f`, Greptile again passed 5/5 with zero unresolved threads, but CI ended with 51 successful checks, five skips and three failures. Server 9/12 fails a deterministic managed-skills maintenance assertion newly inherited from master `198218bf` in merge `c80e00ad`; that test is absent from the PR head, and the storage owner has the explicit-StorageService fix in #18357. Runner 2/2 passed 2,283 unit cases but its runner received a shutdown signal during the long stream probe; its cancelled step and aggregate failure are preserved. No blind retry was used. Current `af9e2d157d` checks are pending; the matcher-only change passes 144 focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI failures and the interrupted unsharded local test run remain recorded in the dated plan. CI runs the repository's normal test partitions. Final verification does not claim Railway rollout, uninterrupted HTTP service or whole-webserver filesystem loss. </details> ## Risks - The task-pointer repair passes its assembled regression. The first live repair exposed pre-acceptance ordering and the fresh review exposed a replay lock-order issue; both are fixed and pass real Postgres tests at `0862062`. The following live attempt exposed the verified-running-versus-suspended preparation gate. Its narrow fix passes 91 focused cases at `64c0b746`. The next live case exposed synthetic attachment IDs; `7a203274` fixes that field selection and passes real Runner authentication. The subsequent queued continuation exposed strict retirement maintenance and a false default-home change; `39a4343` fixes both with real Runner/Postgres and fingerprint regressions. The `39a4343` live continuation failed again despite green CI. `283e05ffaa` repairs the actual native-observed admission ordering and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass targeted real-Postgres regressions and the live active plus queued paths. The definition 6 post-C browser navigation timed out before admission. Explicit definition 7 now verifies native continuation through the ordinary comment API; its strict post-C native gate failed after a real dispatch and remains historical failed evidence. The final `fc68584` repair now passes its fresh strict 49-check Linux/Daytona campaign; the prior failed grades remain unchanged. Browser reconnection is handed to its own workstream with the original failed evidence. No prior failed campaign is reclassified as passing. - Incorrect checkpoint/source binding or stale ownership could duplicate external work. Missing evidence remains a hold; existing receipt and finalization owners keep uncertain effects unresolved. - Recovery must retain maintenance writes after accepted results without allowing a stale business turn to write. Real lock-wait and generation tests cover that boundary. - The current two-connection paired burst profile fails its absolute drain gate in both baseline and candidate, despite passing relative load gates and all correctness checks. A causal capacity diagnostic selected three slots; the final controlled cap-three profiles pass unchanged thresholds. Full application replacement and two-server overlap capacity remain separate. There is no new streaming writer, pool or recurring idle scan. - The remote sandbox and Runner must survive the recovery window. Provider auto-stop is a separate lifetime bound from reconnect grace; the explicit qualification fixture uses a longer idle window. - Workspace and provider backup availability remain external dependencies. This PR does not certify whole-webserver filesystem loss, uninterrupted HTTP service or a Railway rollout. ## Model Used Implementation and initial verification used OpenAI GPT-6 through Codex; the initial hosted snapshot was not exposed. The follow-up qualification used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with repository tools, code execution, CI and scoped test infrastructure. Context-window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and exact-source application/cost checks pass; full forced CI passes; earlier local embedded-PG setup failures remain recorded) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (exact `fc68584` full CI and auxiliary checks pass; forced complete test partitions) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (exact `fc68584` check and edited summary pass; all six threads resolved) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 07:33:52 -05:00
DottaandPaperclip	59da15066a	feat(runner): deliver native Kimi Code (#18391 ) Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-30 22:26:15 -05:00
DottaandPaperclip	0e15c92073	feat(skills): make managed content independent of web-server disk (#18334 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use installed skills and exact version pins during execution. > - Managed skills still depended on mutable directories on the web server. > - A replacement server could lose acknowledged edits or remove an unassigned skill whose directory was absent. > - This pull request makes immutable versions the authority for managed content and installed repository reads. > - The server can rebuild delivery files from durable content after replacement. > - This establishes skills readiness for ephemeral web servers. General server draining and workspace recovery remain separate work. ## Linked Issues or Issue Description Subsystem affected Skills services, shared contracts, database receipts, Skill Studio, and runtime delivery. Problem or motivation A fresh web server must discover, read, edit, and deliver protected installed skills using the same database, object storage, secrets, and supported application artifacts. It must not need its predecessor's files or process memory. Proposed solution Use `managed` ownership with a null source locator. Read installed content by immutable version. Prepare packages before a short publication transaction. Persist execution selections before materialization. Recover exact pinned runtime assets and reject deletion while active or recovering pins exist. Alternatives considered Reconstructing a mutable installed directory preserves the old source-of-truth ambiguity. Permanent dual writes create two competing authorities. This change uses one managed model after a bounded legacy conversion. Roadmap alignment This improves the existing Skills Manager, Studio, version history, and Cloud execution capabilities. The maintainer supplied the implementation contract. This does not add a separate skills product or deploy fleet changes. Related work: #18082 supplies immutable package storage. #17273 addresses edits outside the API for linked local skills; this PR keeps that source category separate. #2648 is a closed earlier ephemeral-filesystem proposal. ## What Changed - Add managed ownership and focused content publication and execution materialization modules. Keep the archive format and package storage tables. - Publish managed create, save, delete, fork, rename, restore, and catalog content with version checks. Add durable create receipts. Preserve file-save receipt responses. - Return file version identities. Keep Studio drafts after a conflict and provide an explicit reload action. A delayed reload cannot overwrite newer typing or navigation. - Persist exact runtime selections, including empty selections. Rebuild pinned assets and verify their admitted digests on a replacement server. - Preserve source access on forks and disconnected copies. Keep the existing workspace policy category for managed skills. - Add bounded conversion, eligibility, and independent verification phases. Load persisted Cloud configuration in the standalone CLI without seeding or changing it. Reject partial current versions against installed inventories; recover only complete saved content or the exact source. Document storage migration blockers and the compatible rollback floor. - Add separate-process PostgreSQL/S3 qualification and update domain, retention, policy, and UI regressions. - Pass the plugin worker manager into Studio execution, admit restricted `skill_test` mode through native Runner contracts, and preserve admitted scope across native and REST tools. Native tests cannot create ordinary tasks, browse company data, or use external tools. - Select the exact durable Studio revision even when unassigned or Beta skills is off. Persist Studio output and audit atomically with native task completion, preserving the result on replay. ## Verification - Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full CI: all 49 workflow jobs completed successfully ([run 41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)). This includes typecheck, test suites, Runner checks, browser checks and build. Greptile review 15 is 5/5 on that same revision, with zero unresolved threads. The PR is ready for review. - Local workspace typecheck and build passed after integrating master. Focused checks passed: 33 execution/completion cases, 52 Runner HTTP cases, 16 managed-content cases, 17 Studio cases, the new real-process CLI regression, map checks and UI token gates. The CLI regression proves inspection does not seed missing Cloud configuration. Server typecheck passed after that fix. - The independent-process PostgreSQL/S3 matrix passed through normal HTTP skill routes. It covers A → B → C replacement, lost commit responses, upload crashes, concurrent saves, binary contents and executable delivery. Metadata listing and warm selected delivery retain zero object downloads. Local timing measurements are recorded; they do not establish hosted latency. - Hosted A → B → C passed on the owned disposable test tenant with external Postgres, S3 and a real remote Paperclip Computer. Predecessor temporary homes were absent. C read and executed B's acknowledged managed save. New runs used repository v1 while v2 was installed. Hashes matched an independently checked exact upstream commit. - Compatible image rollback to `6518b323` passed. Studio retained the unassigned fork's C save. A fresh remote run executed the newer managed content and historical repository pin. The actual shell event confirmed expected hashes, stdout, exit 0 and mode 0555. - Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run 41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)). The final Studio test delivered only the unassigned fork's exact v3, executed its script and persisted a successful output snapshot. Independent command-event inspection confirmed all three hashes, stdout, exit 0 and mode 0555. All 14 installations were version-backed. Independent verification checked all 25 versions: 23 remote objects, two bundled database snapshots, zero blockers. - The deployed image is `9200e3fcb4`. Later master integration, the client-only delayed-reload draft guard and the CLI bootstrap have separate automated proof. The CLI bootstrap was also exercised against the tenant's stored configuration and storage through the same read-only services. This PR does not claim that the deployed image is the final PR revision. - Live Studio checks exposed worker wiring, admission, access scope, exact selection and completion-publication gaps. They are fixed. Failed attempts remain in the evidence. The full local suite exceeded 80 minutes under host contention and was stopped. Later combined/isolated local attempts also timed out; they are not counted as passes. Full CI covers the integrated suites. Earlier CI found an obsolete restart-scope fixture, which was corrected and passed all 52 HTTP cases. - Reproduction and migration: [`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md). Source revisions, deployments, versions, run IDs, hashes and failure history: [`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md). ## Risks - Legacy conversion requires stopped admission and retired filesystem-authoritative writers. Pre-conversion binaries are not supported rollback targets afterward. - Cloud readiness requires remote object storage. Local provider receipts and provider mismatches remain blockers until objects are copied and verified. - Linked local sources retain filesystem semantics and are outside the cloud guarantee. - Cold reads still fetch the existing archive format. Warm runtime preparation retains the verified-cache path. - Bundled reconciliation preserves an installed head. Existing release snapshots remain selectable. An older server must not replace a newer head with its shipped files. - No production volumes, fleet defaults, or general deployment coordination change in this PR. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and browser use. The exact backend model ID and context-window size are not exposed in this session. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 16:18:27 -05:00
Nicky LeachandPaperclip	6174b4d7c8	fix(runner): allow session reset after confirmed task retirement (#18373 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native warm sessions retain a Runner owner between turns. > - Closing that owner must stop execution and preserve valid recovery state. > - A failed suspension can leave a rejected close promise after the task has stopped. > - A session reset then retries that same promise and blocks every fresh session. > - This PR separates task retirement proof from checkpoint reuse for a new session. > - A reset can proceed after exact stop proof, completed handoff, and file collection. ## Linked Issues or Issue Description Refs #18011. That change preserves checkpoint authority during warm preparation. This change handles a stopped owner whose checkpoint cannot be reused. Related: #14695 covers managed files retained with warm processes. A search found no duplicate stopped-owner reset fix. What happened? A warm Runner can stop and complete its task handoff while close reports `runner did not durably suspend before checkpoint`. The transport caches that rejected close promise. Core retains the warm owner. Each session reset retries the same promise before it can launch a provider. A controller restart clears the in-memory owner and permits recovery. Expected behavior A reset to a new normalized session should release the old owner when its exact task has proved that execution stopped and its handoff completed. Core must collect stopped instruction files before it admits the new session. An unconfirmed owner must continue to block replacement. Steps to reproduce 1. Complete a warm native run through a plugin task provider. 2. Change configuration so Core retires the warm owner. 3. Let the task stop and complete, but fail durable suspension proof. Close keeps its rejected promise. 4. Reset the session and start another run on the same task environment. 5. Observe the old close error before any new provider work starts. ## What Changed - Keep a lease-bound retirement receipt in the plugin task owner. It requires `executionStopped: true` and successful task completion after credential handoff. - Accept an unrecoverable suspension error only during retirement for a different normalized session, with that exact owner's receipt and settled Core tool results. - Collect stopped instruction files before releasing the warm owner. Preserve old checkpoint evidence. - Keep same-session, active-owner, incomplete-handoff, unrelated-owner, unresolved Core tool results, and other close-error fences. - Report Core tool settlement on the close error after ingress retirement and drain. Default missing proof to false. Expose the error through the server's Runner shim. Add regression tests and document the recovery boundary. ## Verification - The reset regression fails with the new handling disabled and passes with it enabled. - Both affected suites passed: 1,032 tests. The final 15 focused owner/reset cases pass after adding the tool-settlement guard. - The assembled regression uses the real task-owner and backend binding. Its controlled task provider proves stop and completion. Close returns the same rejected promise on every call. - Negative cases cover incomplete handoff, other close errors, same-session continuation, provider change with the same session ID, unrelated receipts, unresolved Core tool results, and failed file collection. Collection can later succeed without replaying stop or reading the old checkpoint. - Five real-process transport cases pass. They cover an insufficient suspension deadline, the cached rejected promise, late tool completion within and beyond the deadline, interruption, and result-persistence failure. Unsettled callbacks produce `semanticToolsSettled: false`. - `pnpm -r typecheck` passed with the repository's pinned Rust toolchain on PATH. After the final contract change, Runner TypeScript build and server `tsc --noEmit` passed. - The local full-suite run was stopped at the requester's direction. Full regression and build coverage are delegated to GitHub CI. - Full GitHub CI passed on `ef0040d80af679ce6d4a130468e15f75678c1f4c`. Standard Greptile reviewed the same commit: 5/5, zero comments. Apex then reviewed that same commit: 5/5, zero comments. Final checks confirm no unresolved review threads and no merge conflicts. - Initial full CI on `00c002fd5a` failed one scheduled-retry route test at 18/20 cancellations. Controlled cancellation latency reproduced the short poll-budget failure. Before the next cycle, master advanced with #18322, which coordinates that test at the first completed batch. The separate local timeout commit was dropped during rebase. The upstream test passes locally. The rebased run passed 46 targeted owner/reset and scheduled-retry route cases. This PR has no scheduled-retry source or test diff. - No deployed provider run or tenant rollout is claimed by these source tests. ## Risks The exception applies only to a new normalized session and an exact completed plugin task. The old checkpoint remains evidence and cannot authorize reuse through this path. The original suspension failure can remain visible on its run. This fix prevents repeated startup failures after a reset; it does not hide unconfirmed cleanup or make every suspension succeed. Missing task receipts and failed file collection continue to block replacement. No schema migration, plugin protocol change, Rust Runner binary change, or new polling is required. Tests use a controlled task boundary; they do not certify every external provider implementation. ## Model Used OpenAI Codex, GPT-6 family, with tool use and local code execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal issue or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] Greptile Apex is 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 13:57:12 -07:00
DottaandPaperclip	f53db0c70f	perf(server): overlap OpenCode setup and trace reply recovery (#18367 ) Overlap eligible OpenCode GitHub and assigned MCP preparation. Trace deferred reply discovery and recovery, and forward the native GitHub transport recorder. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-30 15:21:05 -05:00
DottaandPaperclip	9f524ec1a7	fix: persist native Runner usage before transport acknowledgement (#18354 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent execution must preserve usage so Paperclip can price work and enforce budgets. > - Remote Runner execution can outlive its web receiver. > - The receiver previously acknowledged Runner events before its later usage recorder saved the accounting receipt. > - This pull request saves required native usage receipts before that transport acknowledgement. > - A replacement receiver can accept the same retained report and create one ledger charge. ## Linked Issues or Issue Description What happened? Native remote Runner usage could leave its durable outbox before the web receiver saved its accounting receipt. The later recorder required a local receipt directory. A replacement web receiver could lose the only remaining report when the database write failed. Expected behavior The Runner retains unaccepted structured usage. Postgres owns accepted receipts. Repeated delivery preserves one receipt for each source report and one charge for the run. Missing usage keeps accounting pending. Steps to reproduce 1. Start a supported built-in native remote Runner. 2. Make the receipt database insert fail after the provider reports usage. 3. Replace the web receiver and its local directory. 4. Reconnect the same Runner and inspect the saved receipt and ledger. Paperclip version or commit The initial investigation used `ebd05e9f4c88c66af7473db86bbaacfd1544aa0a`. Deployment mode A web server with a remote native Runner and shared Postgres. Related work: #18327 owns output logs and provider traces. This change has no dependency on its S3 export path. #18031 covers interrupted Codex receipt recovery. #18005 covers legacy spool reads. #17516 covers Codex stop-boundary retention. ## What Changed - Add native producer receipt delivery through the existing Runner callback before transport acknowledgement. - Reuse `run_usage_receipts`, stable source sequences, and deterministic report identities. Reject conflicting replay and retry a lost commit response with the same receipt. - Restore per-turn usage from structured receipt facts. Preserve provider turns that share one PRP envelope turn. - Bind retained warm transport callbacks to the current run authority epoch, allowing old attachment control acknowledgements while preserving accounting ownership. - Persist Codex session baselines and cumulative totals in the existing provider checkpoint. Mark qualified whole-run deltas, preserve them across resume/restart/attachment, and leave old unmarked reports incomplete and unpriced. - Normalize the bounded final usage tail before the terminal event. Retain later increases for explicit reconciliation, block further attachment, and fail an affected successor without changing its baseline. - Complete runs from the accepted immutable receipt and its exact saved rate-card amount; preserve existing live-budget enforcement. - Skip the local receipt recorder for supported built-in native remote Runner execution. Keep local, legacy, and external Dot/Muse behavior unchanged. - Keep pricing, reservations, ledger settlement, source fencing, and budget holds with their existing owners. - Put the actual Runner integration test in the existing Rust-cached native test lane. Build before opening Postgres and give cold compilation its own bounded budget. ## Verification - Fresh Greptile review is 5/5 on `e7376a50c8162770da1d581ecf25c8cffac64520`, with no unresolved findings. Parent and independent peer review approved the final changes. - Real PostgreSQL 18.6 and a real Rust Runner exercise failed receipt insertion, receiver-directory loss, replay from the same producer, and exactly one 25-cent ledger charge. The actual accounting spool path stays absent. Optional logs/traces are disabled. The provider is controlled; no paid request is made and the fixture never pre-saves the expected receipt. - Producer checks passed 32 usage/provider tests,37 terminal controls, corrected completion, all 3 previously failing exit cases, and both full transport goal-continuation cases. Rejected active-goal completion retains its 120-token tail before failure. Baseline 100→successor150→33 duplicate old snapshots→old120 retains explicit incomplete reconciliation evidence and positive budget exposure. - Heartbeat executor checks passed 982/982. PostgreSQL tests preserve exact 1.002 USD pricing, the 100.2-cent ledger amount, budget holds, source takeover, partial/final receipts, cost-only corrections, equal rate-card replay, and stale-completion interleaving. Newer incomplete usage remains pending rather than settling an older snapshot. - Runtime replay and strict usage-schema checks passed 2/2; generated contracts and Rust formatting pass. Test-partition contracts passed 32/32. - [Final combined-head evidence](https://github.com/paperclipai/paperclip/pull/18354#issuecomment-6531526158) records full workspace typecheck/build and 41/41 accounting tests against all five cloud PR heads, including this final accounting commit. It includes commands and evidence limits. - Exact-head CI is green: 53successful checks, 6intentional skips, and the legacy Snyk status passed. Typecheck, build, Runner, server, browser, and aggregate gates pass. Runner Vitest1/2 initially failed an unchanged mocked Antigravity login fixture; its focused case passed locally, then one unchanged job retry passed 131 files/1995 tests, including all 11 Antigravity tests. The initial failed attempt remains recorded in CI. Initial local attempts exposed the host SysV semaphore limit and corrected harness path, IPC readiness, and queue-counter defects. A temporary test loader selected an existing POSIX-semaphore PostgreSQL 18.6 build without changing repository dependencies. An optional expanded debug provider run was interrupted at its unrelated 8192-turn rollover stress case; no full-file local pass is claimed. Required failure cases were run directly, and the full Runner CI remains required. ## Risks - The guarantee covers built-in native remote Runner execution. Local native execution, legacy adapters, external Dot/Muse execution, and server-originated model calls retain their existing contracts. - This does not provide traffic switching or general web-server failover. The existing controller identity, checkpoint, and generation gates still apply. - The receipt JSON gains optional structured producer facts. No schema migration or accounting S3 spool is added. Text output and raw trace payloads are excluded. - Codex accounting authority requires a Runner artifact built from this provider baseline implementation and qualified by digest. Update receivers and strict usage-schema readers before releasing that Runner; PRP does not negotiate this payload field. Existing unmarked Codex reports remain pending/unpriced. No deployed Runner artifact is claimed fixed. - Usage arriving beyond the bounded terminal tail may require manual reconciliation or a fresh session. Its evidence remains in the existing producer checkpoint/outbox; no adjustment ledger or recurring scan is added. ## Model Used OpenAI `gpt-6.1-sol` assisted with the implementation, tests, and documentation through code execution and repository tools. The context window size is unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 15:20:42 -05:00
DottaandPaperclip	dbff3747b8	feat(server): persist run logs and provider traces in S3 (#18327 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use run logs and restricted provider traces to inspect agent work. > - Local log and trace files depend on the web server's persistent volume. > - That dependency prevents overlapping volume-backed Railway deployments. > - This PR reuses the instance S3 store for batched log and trace payloads with conditional publication. > - Existing Postgres rows retain their contracts; native export recovery reuses events already committed before Runner acknowledgement. > - S3-configured servers automatically copy and verify retained history before a later deployment cutover. ## Linked Issues or Issue Description Subsystem affected Server run logs, restricted provider traces, storage providers, and migration tooling. Problem or motivation Replacing the web server and its disk must preserve committed log output and retained traces. The current S3 mirror can repeatedly upload a growing file. Moving those payloads into Postgres would add database writes, WAL, replication, and backup work. Proposed solution Reuse the existing instance storage configuration, S3-compatible provider and credential chain. Store new run logs under run-logs/ and opted-in raw traces under provider-traces/, below the existing prefix. Cloud keeps its existing bucket and bucket-scoped credentials per stack. Add no new configuration switches. Store immutable payload segments and bounded binary range indexes directly in S3. Publish each batch with an ETag-conditional head. Keep existing database ownership and lifecycle records. Recover canonical native exports from their existing validated control event source, checkpointing export progress in S3. Add no new payload writes to Postgres. Preserve local defaults and migrate settled history with verified byte counts and digests. Alternatives considered Postgres payload chunks add control-plane load. Repeated whole-file mirrors amplify transfer work. The first index prototype left superseded index nodes; the final binary index keeps all normally published nodes reachable. Roadmap alignment This is a storage prerequisite for replacing the main web/API server. It does not enable deployment overlap, detach a volume, or change native Runner protocol acknowledgements. The maintainer approved this implementation scope. ROADMAP.md has no duplicate implementation item. Related PR: #16770 improves the existing in-flight mirror. This PR preserves that legacy backend and uses the existing instance S3 profile for new journals. ## What Changed - Automatically select S3 segments for new logs and opted-in traces when instance file storage uses S3. Reuse bucket, endpoint, region, prefix, path style and credentials. Remove separate backend switches. Keep local-disk instances local. - Share the existing storage configuration resolver with product files. Preserve config-file settings, environment precedence and the existing root prefix. Keep pre-existing run-log mirror settings for legacy history. - Add bounded buffering, conditional ownership fencing, immutable range reads, stable byte offsets, and completion hashing. - Dispatch all existing log consumers through the recorded backend. Coalesce output progress writes and publish live log messages after storage receipts. - Recover interrupted canonical native exports from their existing durable source. Exclude separate credential, cleanup, and reprojection records. Replay missing callback ranges and deduplicate late delivery. Repair settled archives after controller lease expiry. Bound shared repair work and stop it when all readers cancel. Serve the published prefix when best-effort repair fails. - Add encrypted optional trace channels, replacement reads, durable deletion tombstones, and deadline-driven local/external expiry cleanup independently of heartbeat scheduling. Stream historical source and remote verification. Propagate worker cancellation through trace open, conditional-write probing, upload, restore and reads. Fence deletion even before the first object head exists; abort and drain owned captures. - Retry transient storage errors, distinguish 409 from failed preconditions, and backpressure bounded log admission. Track interrupted trace imports explicitly so ordinary local deletion stays independent of S3. - Automatically migrate retained history through the existing delivery coordinator. Process one log and one trace per batch. Retry unresolved history with backoff and stop scans after completion. Keep the inventory/import command as optional diagnosis and repair. Verify source and remote bytes before switching references. Verify finalized native size and digest before setting its export cursor. Require ordered source coverage when completeness metadata is missing. Preserve original trace expiry. Block active, missing, corrupt, and unproven sources. - Preserve identity and run-secret redaction in live and recovered native logs. Register runtime log redaction secrets once through the existing encrypted registry. Keep agent private keys in their existing owner. - Add storage, migration, retention, UTF-8, and route-privacy coverage, plus operating documentation and qualification tooling. - Add no schema, payload table, per-record SQL, production dependency, or lockfile change. ## Verification - Final fixes at `b3a44c898e` passed all 116 selected startup, retention, attribution and cancellation controls. The disabled-scheduler cleanup and concurrent-repair defects reproduced red before the fixes. A timed-out repair still prevents later storage reads. - The merged source passed 124 storage and real-Postgres Cloud configuration tests. These include automatic history migration, existing environment/file vault keys, source-directory removal, five real-SDK stalled-request cancellation cases, shared capability-probe cancellation and multi-chunk trace ordering. Saved Cloud S3 settings work after the legacy config file is gone. - Workspace typecheck and full build passed at the merged parent `19f702d780`. Server typecheck and build passed again at `b3a44c898e`. - Actual app acceptance: finish a process-agent run through the normal API, stage its retained local-file input, then restart. Automatic migration reports one migrated item and zero blockers. Remove the original web home and restart at `b3a44c898e` with an empty home, the existing S3 profile and the existing vault key file. The normal log API preserves all five producer lines and exactly 788 bytes, with SHA-256 `74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No log-specific storage settings are present. - Actual expiry acceptance: with heartbeat scheduling disabled, a trace remains unexpired after startup and later reaches its fixture deadline. The worker marks the row deleted and tombstones both S3 channel heads. The retained run log stays byte-identical. The process fixture emits no provider protocol frames, so nonempty encrypted payloads are qualified separately through the real SDK round trips. - Isolation tests use identical run identities in two stack buckets. Cleanup in one existing prefix leaves another prefix, another company and product files intact. Read-only Cloud source inspection confirms a bucket and bucket-scoped credentials per stack. Four existing Cloud storage environment-contract tests passed. No Cloud provisioning changed and no new live vendor permission probe was performed. - Earlier qualification passed 1,014 affected storage/native tests and 27 privacy-route tests with isolated Docker Postgres. An abrupt process test commits 1,000 records, receives receipts, dies by SIGKILL and loses its directory; another process reads every byte and continues. The real Rust Runner experiment acknowledges 32 records while the export cursor is zero, then continues the same Runner PID after controller loss. All 64 records export once with exactly 64 existing control rows. Repeated snapshot/Stop commands completed in 162/81 ms. - Emulator workload: 1,000 burst records used two batch PUTs. Eight steady streams produced 240 records with 104 conditional PUTs and 72 GETs over 6.9 seconds. These are synthetic operation counts, not fleet sizing or provider prices. - Failure evidence is retained. Compact native events over 64 KiB, missing file-key support, disabled-scheduler retention and repair failure handling all reproduced before their fixes. Fixture repairs addressed cold module setup, localhost/IPC permissions, the S3 bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm test:run` was stopped after host PostgreSQL semaphore exhaustion; affected suites used isolated Docker databases instead. Native package library symlinks were restored with the package's standard hydration helper. - The `19f702d780` browser session-recovery case timed out while saving a task title. All eight browser shards passed on `b3a44c898e`, including that case, with no UI behavior or assertion change. Fresh-head CI passed at `b3a44c898e`: 55 successful checks, three skipped checks and one neutral check. The branch is conflict-free. - Map and module-boundary checks passed. Current-head Greptile is 5/5 with zero unresolved review threads. Reproduce the opt-in storage suites with `PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated `PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See `doc/run-evidence-storage.md` for operating limits, automatic migration, optional diagnostics and rollout qualification. ## Risks - Keep the existing per-stack bucket, prefix and vault key stable across replacements. Relocating that storage is a separate whole-instance operation. - This release automatically changes new log storage for existing S3-configured instances. Local-disk instances remain local. Canonical native source replay is qualified with a deterministic real Runner; actual remote execution and production-provider qualification remain rollout gates. Generic server-only diagnostics and optional raw trace tails without a surviving source can be incomplete after a crash. This does not establish full zero-downtime deployment. - Before deploying this release, qualify the actual S3 provider. It must enforce conditional writes and pass read, version, deletion, workload, and latency qualification. A capability probe rejects providers which ignore preconditions. Moto is emulator evidence only. - Dirty output is bounded and unconfirmed until S3 publication. Storage failure leaves persistence incomplete while task results and usage can settle. - Trace expiry is not yet represented by Cloud's external wake contract. Outstanding trace deadlines conservatively block sleep. - Automatic history copying requires the original source volume. Keep it until all retained resources pass verified import and clean-filesystem reads. Stop old binaries from creating new local-file runs. Missing or corrupt history remains blocked and visible in server logs. Rollback requires a compatible reader against the same external state. Ordinary object deletion does not erase retained historical object versions or backups. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 15:17:35 -05:00
DottaandPaperclip	ebd05e9f4c	perf: avoid scanning unused provider pack trees during archive upload (#18340 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents need verified provider files in remote sandboxes. > - A fresh sandbox can need the bundled provider-pack archive. > - The upload path also hashes the expanded controller tree, which it does not upload. > - That synchronous scan blocks the server event loop and delays other requests. > - This change verifies the archive source and the installed pack, and keeps source-tree checks for directory uploads. ## Linked Issues or Issue Description What happened? Provider-pack archive fallback scans and hashes the full expanded controller tree before reading the build archive. The archive is the source sent to the sandbox. The extra scan blocks the Node event loop for several seconds. An overlapping asynchronous operation can then appear to take as long as that scan. Expected behavior Verify the bytes used by the selected upload path. Keep full verification of the installed pack against the selected manifest before a provider can use it. Steps to reproduce 1. Use a packaged provider pack with a build archive. 2. Start a remote native run whose sandbox needs that pack. 3. Inspect `provider_pack.archive.verify` and event-loop delay on the controller. Related: #18308 adds timing detail for workspace preparation. A search found no duplicate archive-verification fix. ## What Changed - Use the verified archive without scanning the unused expanded directory. - Keep full directory verification when no archive exists or the remote lacks archive support. - Keep archive digest checks and full verification of installed bytes. - Add `provider_pack.directory.verify` for directory fallback. - Add regression cases for both archive formats, fallback, and final verification failure. ## Verification - Both archive regressions failed before the change and passed after it. - 80 focused archive, selection, and verification cases pass on the rebased branch. - 20 native manifest and admission cases pass. - Server typecheck, server build, and the runtime bundle pass. - A disposable sandbox from the recorded image exercised the real uploader and full installed-pack verifier. It accepted the unchanged build bytes and rejected an altered proxy file. No model was called. The sandbox was deleted and its absence was verified. - A local baseline/archive/archive/baseline probe used the same unchanged public build pack. Preparation took 30.673 s, 84 ms, 106 ms, and 5.371 s. The slow baseline outlier is retained. This measures one controller phase; it does not establish a whole-turn or hosted speedup. ## Risks The archive path no longer checks expanded files that it does not send. Archive checks still fail closed, and the installed files must pass the same full manifest verification. Directory fallback keeps its source-tree check. Installs without archives retain that synchronous scan. The local sandbox probe tests upload and installed-content checks; it does not qualify admission of an older pack by a newer controller. ## Model Used OpenAI Codex, GPT-6 family, with tool use and local code execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 11:47:21 -05:00
Nicky LeachandPaperclip	1d571de5f9	Add shared native turn attribution boundaries (#18213 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - Native execution crosses the plugin worker, Runner, sidecar, and durable event boundaries. > - Existing outer spans leave repeated preparation and admission work unattributed. > - Optimizations need a common baseline with accurate ownership and event meanings. > - This change extends the existing opt-in tracing contract with bounded timing receipts. > - It preserves execution policy and keeps provider internals opaque where the API provides no timing proof. ## Linked Issues or Issue Description What existing behavior does this improve? Attribution of native task preparation, retained/cold Runner handoff, and completion-to-text work. Current behavior Worker spans use a flat provider contract. Workspace calls lack a consistent dispatch envelope. Submitted-to-started timing is named as a provider queue even though it includes other work. The transport does not expose a shared breakdown of its waits. Proposed improvement Add invocation-scoped nesting, bounded numeric remote receipts, current-run transport timing, durable receipt intervals, and distinct completion/text milestones. Reuse the existing exporter. No latency optimization is included. Related public work: #17842 consolidates existing startup tracing tests; it does not add this runtime coverage. ## What Changed - Extend the optional SDK tracer with nested async scopes and completed timing receipts. Keep the host responsible for parent identity, capability checks, attributes, clocks, depth, and count limits. - Add dispatch and bounded host-read timings for environment and workspace providers. - Resolve transport parents on each call so retained sessions do not retain an earlier run's trace. - Split cold/resume/attach/prepare/registration/command/started-event waits. Add opt-in numeric ACPX Runner/sidecar status and prompt-start receipts. - Distinguish first completion request, validated tool receipt, final result commit, first nonempty text, and event persistence. Rename the submitted-to-started interval to match its meaning. - Link completed earlier remote lifecycles to their host-owned observer rather than placing them under a later polling request. Preserve sampling, timestamp limits and budgets. - Document the versioned coverage map, causal clock limits, and baseline acceptance requirements. ## Verification - Current head: GitHub CI is fully green. Standard Greptile and Apex are both 5/5, with no unresolved review threads. Full-suite validation was performed in CI; follow-up local validation was limited to affected packages and tests. - Review follow-up: early rejection, missing provider capability and startup-parent regressions were reproduced and fixed. The affected worker/SDK/timing set passes 128 tests; 28 timing/context tests, SDK typecheck and direct server TypeScript checking pass. The server typecheck wrapper reached an unrelated Runner rebuild and stopped because Cargo was absent from PATH; full Runner/image validation remains in CI. - CI failure accounting: the earlier Turbo cache/force conflict and Linux OpenCode shell assertion were reproduced and diagnosed. A direct proc probe ruled out permissions; a portable shell comparison identified Bash skipping BASH_ENV with stripped SHLVL and socket stdin. Upstream #18230 now supplies the CI and ignored-stdin fixture fixes; this PR adopts them without duplicate CI changes. The strict Linux OOM assertion passed before this source-equivalent merge. - Retry follow-up: the three insertion/replay/failure timing cases and server typecheck pass. The existing database regression now asserts a single marker across identical result replay; the current-head CI integration gate passes. - Upstream CI alignment: all three real-Turbo 2.11.10 integration cases pass. The repository policy gate rejected committing a generated lockfile; the final PR contains no lockfile edit and uses the maintained inline stale-lockfile recovery. - Linked-lifecycle follow-up: 44 focused host/export tests pass (three optional SDK-dependent cases skip); SDK and server typechecks pass. The API contract test verifies an explicit root context, host-owned observer link and unsampled suppression. - SDK typecheck and focused worker/invocation tests: 30 passed, including clock changes, bounded legacy IPC, and manual-span creation-scope parentage. - Focused reviewed host/native/transaction-boundary checks: 44 passed. The 86-test worker-manager suite passes after updating its exact assertions for the host-owned timing budget found by CI. Earlier preparation/first-text checks also passed. - Affected server and Runner TypeScript checks pass after building local workspace declarations. - Merge follow-up: the profile/candidate startup subset passes 94 tests, the four Grok source-identity checks pass, and Runner TypeScript checking passes. A subsequent execution-factory merge passes 21 focused timing/trace tests and server TypeScript checking. - Pinned Rust build and the ACPX Rust subset pass. The hostile receipt projection test passes. Source profile compatibility checks: 127 passed, one existing skip. Grok identity checks: four passed. - Broad transport/ACPX suite: 410 passed after building all fake provider binaries. Earlier attempts failed on missing Runner/fake-provider outputs; building the required fixtures resolved those failures. - The first CI run correctly rejected stale source attestations. After integrating current master, Grok advances 13→14 and Copilot 84→85; master’s historical fixtures and Pi 35 stay byte-identical. Provider dependencies, models and permission defaults are unchanged. Identity generation/checks now pass. - The added real-Postgres result lock regression could not initialize its local embedded cluster. The insertion/replay/failure unit regressions pass. The current-head CI integration gate passes. - Comprehensive repository checks and image builds are delegated to CI, as requested for this instrumentation pass. No deployment, model run, collector-ingestion proof, or latency improvement is claimed. ## Risks The new SDK methods are optional. Legacy span calls and endpoint-off behavior remain supported. The host accepts only closed timing names and bounded attributes. Remote durations are anchored at local receipt; transport/scheduling uncertainty remains explicit. Trace trees have count/depth limits. The first rollout must inspect actual collector exports before treating the baseline as accepted. Fresh live qualification for the new source identities remains a rollout requirement. Revert these commits to remove the new measurements; no schema migration is required. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and local code/tool execution. The exact runtime model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Standard Greptile and Apex are both 5/5, with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 17:45:56 -07:00
DottaandPaperclip	61bd4ac691	feat(agents): set up Antigravity with saved Google subscriptions (#18189 ) ## Thinking Path > - Paperclip manages agents and their work. > - Users choose providers and managed accounts while adding agents. > - Antigravity was missing from the ordinary additional-agent picker and its saved Google path did not complete native readiness. > - A user needs a successful Test, an enabled task action, and a stable account binding. > - This pull request connects the ordinary setup flow to the native candidate. > - It also preserves retired credential refreshes across database failures and server restarts. ## Linked Issues or Issue Description Built on merged #18188 and #18187. Refs: #12503, #14938 for related legacy CLI proposals. Agent or provider Google Antigravity ACP 1.4.3. Why this adapter is useful Users need to set up an Antigravity agent and run real work through their managed Google subscription. How the agent is invoked Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact operator admission remains required; the local candidate result does not enable general release. ## What Changed - Show Google Antigravity in the ordinary adapter grid and preserve the draft choice when setup resumes. - Reuse the same user’s saved personal default Google subscription and discover the exact low model through the native account. - Run a prompt-free native Test and wait for authoritative readiness before enabling task assignment. - Carry the managed account through native runs and recover retired refreshes through the existing encrypted, fsynced journal with exact generation checks. - Retain the latest rotated credential through verification, storage, and partial home-removal failures. Retry the exact account’s retained cleanup before another provider use. - Reconcile cleanup before taking the credential mutation lock, fence owners arriving during lock acquisition, and support nested lock holders without a join cycle. - Bound sign-in retirement outside database transactions and preserve claims for the existing cleanup owner; one pending sign-in does not block unrelated accounts. - Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model editing usable, and keep adapter discovery available after candidate expiry. - Skip taskless native timer work before run or credential admission. Preserve current master monitor and deadline scheduling. ## Verification - The registered local saved-Google-subscription journey passed all 16 matchers in 196,520 ms using eval source `12aba` against actual serving source `4a37ee7c`. Both turns stayed in one conversation, produced exact downloaded files of 61 and 119 bytes after reload, and used distinct native primary processes. All observed owners retired, leases released, and credential fences settled. Independent readback confirmed Done work, no active owned runs, a connected saved account, and the retained agent paused for inspection. - The measured runtime is production-equivalent to pre-restack `65ef31dc` except its test-only transport fixture correction. Current-master head `ff7c9249` preserves all 99 product paths: 98 exact blobs and only eight inherited plugin-panel lines in `agents.ts`. Its native binary was rebuilt for inherited Codex-only diagnostic changes; the changed Rust behavior is outside Antigravity's ACPX execution path. Shared native/UI changes have focused regression coverage. The recorded live result is not represented as a measured replay of `ff7c9249`. - Full workspace build and typecheck passed on `ff7c9249`. All 376 focused tests in 14 files passed, covering Google credential/retirement controls and the inherited native-tool, redaction, route, and UI recovery changes. Token gates, maps, and diff checks passed. The required-close fixture was separately demonstrated red, then passed all 317 selected provider-wiring cases. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972) is green: 55 successful checks, two skipped workflow checks, and an optional neutral review check. Required `ci / verify` and `ci / e2e` passed. Greptile and Paperclip Review both report 5/5 on exact `ff7c9249`, with zero unresolved review threads. - This qualifies the selected operator-authorized local saved-subscription candidate. The original exploratory journey remains historical evidence; the repeatable fixed-driver eval is in the next PR. ## Risks - Google refresh recovery must preserve the exact grant, account, and credential generation. An uncertain native owner or failed encrypted handoff retains the private home and blocks cleanup. - A successful local saved-subscription candidate does not qualify fresh OAuth, API keys, Cloud, Linux, other models, or general release. - Timer filtering changes only taskless native Runner admissions. Existing task-bound monitors and legacy adapters retain their owners. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 16:51:50 -05:00
Devin FoleyandPaperclip	5d79d88d68	fix(server): let settled native finalizations permit idle sleep (#18184 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs retain finalization records for recovery and audit history. > - Idle sleep checks durable work before it stops an instance. > - The current check treats every retained finalization as pending work. > - A committed record can remain after its run and status effects have finished. > - This pull request recognizes complete records without deleting them. > - Instances can sleep while unfinished execution and recovery work remain protected. ## Linked Issues or Issue Description What happened? A retained `native_run_finalizations` row prevents idle sleep even after the run has finished and its status decision has been applied. The current gate checks whether any row exists. Expected behavior Complete finalization history should remain available without counting as pending work. Active leases, retries, accounting, recovery, and incomplete status effects must still prevent sleep. Steps to reproduce 1. Finish a native run and retain its committed finalization. 2. Confirm that its accepted result, assessment, applied decision, and delivered effects agree. 3. Request an idle report under a valid quiet admission hold. 4. Observe that the retained row reports background work before this change. Paperclip version or commit Reproduced on master at `cd0c86e5e1d427c8fa8aa8c3a9499391ee9b425b` with a local PostgreSQL fixture. Deployment mode Server with PostgreSQL. No provider calls are needed to reproduce the gate failure. Related PR and issue searches found no duplicate for this check. ## What Changed - Replace the blanket finalization-row check with a completion proof bound to the same company, issue, run, accepted result, assessment, and applied decision. - Require the owning run's successful workspace-finalization receipt; retain unresolved leases, retries, accounting, recovery, and status effects. - Share final-reply candidate discovery with the presentation recovery worker, so a completed run cannot hide a required reply. The pure query stays with native presentation; idle inspection uses it in the existing read-only transaction and five-second statement limit. - Register open warm sessions and in-flight close/checkpoint promises with local idle inspection. Historical controller/provider process metadata remains preserved. - Add PostgreSQL and local-warm regressions and document the retained-history rule. ## Verification - PostgreSQL sleep/finalization and chat-monitor continuation suites: 300 tests passed, including 56 finalization cases. - Local-warm regression: passed; an open session and its pending close/checkpoint both block sleep until retirement completes. - Server typecheck and server build: passed after review fixes. - Full workspace typecheck and build: passed before the review additions; hosted checks cover the updated revision. - An earlier local full invocation encountered two chat recovery failures that passed in isolation, and two existing Git workspace fixture failures caused by Xcode system Git initializing `main`. Those two pass with `init.defaultBranch=master` scoped to the test subprocess. That superseded full invocation was stopped for review revisions; no full local pass is claimed. No machine configuration or unrelated source was changed. - Fresh Apex review: 5/5 on `6846203fe2383bae1ede888fbaaedf83e9946e73`, with all findings addressed. - Hosted CI: fully green on `6846203fe2383bae1ede888fbaaedf83e9946e73`, including the final verify gate. Twelve jobs were interrupted by a simultaneous runner shutdown; rerunning the failed jobs completed successfully. - No hosted deployment of this change is claimed. ## Risks This changes an automatic-stop gate. An incomplete proof could stop an instance with pending work. The SQL fails closed and retains the existing admission, local work, plugin, schedule, backup, and provider-allocation gates. PostgreSQL cases cover unresolved leases, retry/recovery intent, accounting, projections, rejected results, unapplied decisions, and pending effects. No migration or history deletion is required. Reverting the predicate restores the conservative gate. ## Model Used OpenAI GPT-6 through Codex, with repository tools and local PostgreSQL verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 11:11:22 -07:00
DottaandPaperclip	6f278dd6d7	fix(runtime): settle recovery lease and connection lifecycles (#18167 ) ## Thinking Path > - Paperclip manages AI agents and their work. > - A remote runner can survive a controller restart. > - Recovery renews its execution lease during slow preparation. > - The native executor then shares that renewal timer. > - Preparation kept its reference until all finalization finished. A tick after lease release treated completion as ownership loss and detached the retained runner. > - A Linux continuity test also found that buffered wire events could arrive before their connection was registered. Cleanup then kept a completed promise under an undefined key and never finished. > - This pull request transfers renewal ownership after execution acquires its reference. It also registers each connection before wire callbacks can run. > - Recovered turns can complete, retain the warm runner, and settle connection cleanup. ## Linked Issues or Issue Description What happened? A recovered remote run could complete, release its coordinator lease, then detach during instruction collection. The next turn could find the old runner still using the ingress port. Recovery then stopped because it could not prove process ownership. Expected behavior Preparation keeps the lease alive until execution owns renewal. Execution stops renewal before releasing the lease. Actual ownership loss still detaches the runner. Buffered wire callbacks must use the registered connection, and cleanup must finish. Steps to reproduce 1. Recover an in-flight remote runner after a controller restart. 2. Let the recovered turn complete. 3. Hold instruction collection open through the next lease renewal tick. 4. Send a follow-up that needs the retained runner. Related: #17439 introduced remote controller recovery. #18011 preserves checkpoint authority through warm preparation. This change fixes the lifetime of preparation's renewal reference. ## What Changed - Pass an internal lease-acquired callback from recovery through heartbeat dispatch. - Release preparation's renewal reference after the executor registers its own. Keep final cleanup for exits before dispatch. - Add a PostgreSQL regression for completion while finalization is still pending. Cover callback forwarding, acquire-before-release ordering, and failure before dispatch. - Register the authority connection before subscribing to wire events. Handle an immediate close before accepting buffered data. - Add regressions for a message or close delivered during subscription. - Extend Linux continuity coverage with preparation renewal and delayed instruction collection after recovery. - Update the continuity guide and session verification recipes. ## Verification - The new recovery regression failed before the fix because renewal detached the runner after lease release. The 76 recovery, renewal, and dispatch tests then passed. The final ordering and early-exit checks passed again after replacing fixed sleeps with controlled interval ticks. - Both buffered-wire regressions failed before the connection fix. All 170 control-plane tests passed afterward. Existing tests cover stale authority, ownership loss, pending operations, and closed ingress. - Linux ARM64 acceptance passed at `a7daefacceb8bd994d0b79d284cdf54f3e3829d1`, with Node 24.19.0, PostgreSQL, the real Rust runner, TLS ingress, and a deterministic Codex protocol fixture. The test interrupted the first turn, recovered it, held finalization open for a renewal interval, and completed two follow-ups. It retained one provider process, three turns in one history, and the same runner identity. Final suspension and connection cleanup completed. The ordinary warm-reuse case also passed three turns with one provider process. - Earlier Linux attempts completed the turns but hung during cleanup. A debugger found a fulfilled promise keyed by an undefined connection. The regression reproduces that defect. One later setup attempt timed out while staging an unstripped 373 MB debug binary. The passing run used the same binary with debug symbols removed. - Reproduce Linux acceptance with the `same-server` case in `doc/testing-native-remote-continuity.md`. - `pnpm -r typecheck`, `pnpm build`, map validation, and `git diff --check` passed at the final revision. The full local `pnpm test:run` command was attempted, but the serial run had not finished after 13 minutes. It was stopped after the equivalent CI lanes passed. It is not counted as a full local pass. - Greptile and the Paperclip review both scored the final revision 5/5 with no findings. All Runner verification lanes passed. The first CI attempt hit timing failures in voice recovery, agent YAML configuration, and the task-title outage browser test. The full YAML and voice-recovery files passed locally (88 and 19 tests). These paths were unchanged and passed on the prior revision. All three passed on the single retry. Final current-head status: 55 successful checks, one neutral check, and two skipped Storybook checks. No pending or failed checks remain. - Live Daytona/model acceptance and recovery of the hosted task remain unverified. This PR does not deploy the change. ## Risks - Releasing preparation before execution starts renewal could leave a gap in authority. The callback runs after the executor registers its reference. - Preparation errors must not leak renewal. The recovery promise retains idempotent final cleanup. - Wire callbacks can run during subscription. Message and close regressions check that registration is complete before either callback runs. - No schema, API, model, permission, lease duration, or retry policy changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser diagnostics. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 11:18:53 -05:00

1 2 3 4 5