## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents keep remote processes and conversation history across warm turns. > - A completed turn can leave handoff authority only in controller memory. > - Losing that controller can block a follow-up even while its sandbox survives. > - This pull request verifies durable ownership and reconnects the exact prior process before handing it to the next run. > - Healthy follow-ups stay warm. Recovery preserves saved work without replaying a completed turn. ## Linked Issues or Issue Description **What happened?** A remote warm turn can finish without suspended backup authority. A replacement controller rejects that prior owner even when its exact process and completed checkpoint remain. A single unavailable identity probe can also stop an otherwise valid reattachment. **Expected behavior** Keep healthy follow-ups warm. Recover a completed session after proving that its controller stopped and that the successor holds current authority. Reuse compatible processes; otherwise save and resume the conversation. Retry unavailable identity reads within one shared deadline. **Steps to reproduce** Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute real turns through PostgreSQL, TLS ingress, and the Rust runner. They kill a separate controller process, recover old checkpoint metadata, and change process configuration. The provider is deterministic. Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds recovery diagnostics. #14695 covers provider process retention. ## What Changed - Preserve warm process reuse between healthy remote turns. - Verify a committed prior result, stopped controller, matching checkpoint and sandbox, and current task/coordinator ownership before recovering an orphaned warm session. - Reconnect through the prior run route, then use the existing authenticated run attachment to change authority. - Persist a process-configuration digest. Close and save recovered sessions when old or changed configuration cannot support reuse. - Retry unavailable state, marker, and process reads within one 30-second deadline. Keep signals and definitive identity failures single-attempt. - Record closed recovery diagnostics, including unreadable or malformed saved state. - Retire the recovery-only ownership check after verified attachment, so later warm turns use their current authority. - Add real-process Linux regressions and refusal controls. - Expose the existing source-only file-recovery form for retained native holds, with the same board authority, evidence, and exact stop-receipt requirements as legacy holds. Recording recovered files does not change task status or replay a run. ## Verification - `pnpm -r typecheck` passed; the server check was repeated for the third-turn fix. - `pnpm build` passed after the final UI change. The UI typecheck, token gates, and map checks also pass. - Four Linux continuity cases passed: warm reuse, controller loss, legacy checkpoint metadata, and changed configuration. Each creates one conversation and starts exactly three turns, including a healthy follow-up after recovery. The fixture uses expired prior leases; the third turn keeps the runner and provider processes. - Greptile caught the retained recovery gate; the third-turn regression now covers it. An initial test omitted the heartbeat’s saved input and left a test process after failure; the fixture was corrected and its disposable container reset before the passing run. - The current focused executor/restart group passes all 749 tests, including the original failure classifications. - Native file-recovery and UI tests pass (52 tests), including blocked tasks and both native and legacy holds. - After merging current master and retaining its equivalent CI expression check, the CI policy tests pass (15 tests) and all four three-turn Linux continuity cases pass again on `dec418ae55ff21b4fb067132613658cce445f181`. - `pnpm test:run` completed its general server group with 20,572 passes, 103 skips, and three failures: the skill-audit child process and two HTTP test timeouts. The same three suites then passed in isolation (89 tests) without code changes. That local command stopped before later groups; the complete final-head CI, including general, serialized, Runner, build, and browser gates, is green. - Greptile reviewed the combined head `dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable findings; both earlier findings are resolved. All final-head CI gates and the security scan passed. Staging deployment and retained-incident disposition remain deployment acceptance work. ## Risks - Recovery requires exact durable evidence. Live conflicting controllers, replacement sandboxes, Stop/pause, and uncertain effects must remain blocked. - After controller loss, tool-contract or process-configuration changes can restart the provider. Normal healthy follow-ups retain their processes. - The new idle-recovery path has Linux fake-provider evidence. Live Codex/OpenCode and retained quarantined-state recovery still need staging qualification. - No database schema, telemetry, or runner protocol changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution, and browser tools. The session does not expose a more specific model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks and isolated reruns; full-run limitation recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `dec418ae55ff21b4fb067132613658cce445f181` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
14 KiB
Recovery — agent access
Product map: Recovery.
Review date: 2026-10-12. Source commit: 58e16dbddfc9c8d0cf60867f9d9847f0feda055b.
Native REST transport reconciled in this change; see the
shared transport contract.
This is a source-reviewed baseline of representative operations and known gaps. Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the map contract for format, permission checks, evidence and maintenance.
diagnosis
Review: baseline.
API: present — representative operation IDs:
GET /api/issues/{id}/diagnostics/blockers.GET /api/issues/{id}/recovery-actions.
Agent access: available — REST / native call_api; CLI issue diagnostics.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. The server can recover a prepared remote bootstrap only with a durable never-claimed executor receipt and no contradictory execution evidence. This automatic classification grants no new agent action or permission; normal execution admission and durable runner-state verification still apply. The current server's active preparation remains with its existing execution. A preparation that loses the coordinator lease cannot fail the shared run or revoke the new owner's gateway tokens, even after lease expiry. Only normal recovery can establish a takeover claim; no agent permission is added. Older unmarked preparations do not qualify for this automatic path during an upgrade. The fencing marker comes from fresh server-owned identity creation; caller profile data cannot upgrade an existing preparation.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Diagnostics expose recorded cause and boundaries; source-backed reading does not prove successful recovery.
bounded-continuation
Review: baseline.
API: present — representative operation IDs:
POST /api/issues/{id}/recovery-actions/resolve.
Agent access: partial — Eligible agent REST / native call_api / CLI under
source-bound recovery authority.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Eligible resolutions use native call_api. The route retains owner and
management authority, active-run/source-trust checks, board-only outcomes and
reconciliation controls.
Scoped source review: 2026-10-13, recovery correction against
ea6c8eb5e16d61111968f633a66748b397b64a3e. The
recovery service reuses the
existing completed-onboarding handoff validator. It checks the source company,
onboarding origin, Done parent, assignee and Done child, with no unfinished
siblings, before exempting the report from the issue-terminal backstop. Other
terminal work and process-death/controller ownership checks remain enforced.
This internal wake handling adds no recovery-resolution or cross-company agent
authority. Real PostgreSQL regressions cover allowed and rejected bindings;
revised live agent authorization and callback delivery remain not-run.
source-action
The recovery-actions read API includes settled execution reconciliation holds
with an automatic replay: blocked receipt, as well as retained workspace holds.
The existing company and task-read checks remain in force. The board notice
reads this history to explain its current hold; this adds no agent mutation
authority. Route and notice regressions cover the source contract, without
claiming live agent or provider qualification.
Review: baseline.
API: present — representative operation IDs:
GET /api/issues/{id}/recovery-actions.POST /api/issues/{id}/recovery-actions/resolve.
Agent access: partial — Eligible agent REST / native call_api / CLI; source-bound
recovery state.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. Completion or cancellation of a task does not clear its exact unresolved native workspace owner hold. Physical owner release still requires the operator stop-verification procedure.
For a resolved legacy or native action that retains a file hold, the same resolution API supports source-only file repair. Board runtime management and recovery-resolution authority are required. The request identifies the exact source action/run and records recovered-file evidence; the server verifies the original stop receipt. It preserves the current task status and does not wake or replay the agent. The task form fails closed without permission or source bindings. Agent Chat retains the file diagnosis while this hold exists. The repair route regressions and file form cover the local contract; they do not independently recover files or qualify a live provider.
Evidence: source-only — implementation at
the reviewed commit above. Exact owner hold retention in the
recovery state helper
was reviewed at 89770ba707fb1ebd4cabd184eec2e7fee226e0c3.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability. Local
recovery route regressions
verify one unchanged hold across reads and sweeps on done/cancelled tasks, no
physical copyback, and cleanup after the receipt is cleared. Exercise this
sequence in a disposable instance; automatic cleanup must use the system actor
and must not create reader participation. No live recovery proof is claimed.
Gaps: Permitted resolutions use native call_api under existing eligibility
and ownership checks. Source/action policy determines whether continuation is allowed.
A recorded operator Stop before legacy runtime selection releases only that run's task locks and preserves deferred input for normal later admission. This does not grant an agent a new retry or provider-stop authority. Local database regressions cover the release boundary and a competing active routine cycle; they are not live provider-stop proof.
exhaustion
Review: baseline.
API: present — representative operation IDs:
GET /api/issues/{id}/recovery-actions.GET /api/heartbeat-runs/{runId}.
Agent access: partial — REST / native call_api reads; operator escalation flow.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Reading exhaustion is supported. Continuing past the recorded gate requires the specified owner/action.
workspace
Review: baseline.
API: present — representative operation IDs:
POST /api/issues/{id}/recovery-actions/retry-workspace-export.POST /api/issues/{id}/recovery-actions/resume-workspace-finalization.POST /api/issues/{id}/recovery-actions/resolvefor retained-source file repair.POST /api/execution-workspaces/{id}/reconcile-branch.
Agent access: restricted — Board/runtime management APIs; agent reads diagnosis and
performs permitted workspace work.
Policy: Export retry and branch reconciliation require board runtime management.
Retained legacy file-only repair requires the board, exact stopped-source receipts,
and recovered-file evidence. It can record repair for a successful legacy turn
without changing its result or authorizing replay. Queued and running sources
remain denied. Source-only reconciliation regressions cover this boundary.
Physical-owner stop confirmation uses the separate runtime:confirm_owner_stop
action. It requires active company owner/admin membership, trusted local board
access, or instance administration. Agents and ordinary runtime managers are denied.
The exact source task, running native run, accepted result and owner receipt must
match; active copyback locks, live local writers, and newer runs prevent admission.
An authorized administrator must first verify through the deployment platform that
the exact controller and all copyback children stopped, then attest that evidence.
Elapsed time, an expired coordinator lease or a lost advisory connection is not proof.
Only the exact owner receipt is released; normal reconciliation performs export and
commitment without another provider turn or bypassing existing task waits.
Evidence: source-only — route,
administrator authorization, and
owner admission
reviewed for this change. Local authorization, route, UI and finalization regressions
cover member/agent denial, exact-owner admission and the real reconciler preserving
monitor and approval waits. This is not live provider-stop proof.
Verification: not-run for live agent/deployment execution — assert the documented agent denial with an owned test fixture;
verify any listed agent request/result path separately. A delegated operation needs its
own policy and access review.
Gaps: Export retry and branch reconciliation require board runtime:manage at their HTTP routes. The exact-owner operator fallback additionally requires runtime:confirm_owner_stop. Native call_api preserves these agent denials. Automatic managed-cloud recovery uses independently verified, signed platform proof through the finalization reconciler; agents cannot attest owner shutdown.
unknown-outcome
Review: baseline.
API: present — representative operation IDs:
POST /api/chat-endpoints/{endpointId}/publications/{publicationId}/resolve.
Agent access: restricted — Board provider-outcome resolution; agent may inspect
assigned provider tools.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — assert the documented agent denial with an owned test fixture;
verify any listed agent request/result path separately. A delegated operation needs its
own policy and access review.
Gaps: Uncertain-side-effect resolution is operator/source-specific; do not replay to obtain apparent parity.
user-continuation
Review: baseline.
API: present — representative operation IDs:
POST /api/issues/{id}/comments.
Agent access: partial — User comment API and server-owned wake reconsideration; native
agent consumes bound wake.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Receiving the continuation is supported; impersonating a saved user message or retry control is not.
reconnect
Review: baseline.
API: present — representative operation IDs:
GET /api/companies/{companyId}/events/ws.GET /api/issues/{id}.
Agent access: partial — Existing event protocol; REST refresh of persisted task state.
Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.
Evidence: source-only — implementation at
the reviewed commit above.
Verification: not-run — exercise a listed permitted operation with actual agent
credentials/run context; inspect its result and applicable persistence/attribution.
Assert the relevant denied actor, owner, company or mode from Policy. Review the
remaining operations before qualifying the whole capability.
Gaps: Streaming reconnect needs its protocol client. Polling/read APIs remain available; replay is not proof of a fresh successful action.