Files
PaperClipAI/maps/agent-access-map/recovery.md
T
DottaandPaperclip adb69e8f09 feat: default new Codex agents to Paperclip Runner on Linux and macOS (#15683)
## Thinking Path

> - Paperclip is the open source control plane for agent companies.
> - People choose a harness when creating an agent.
> - The server should resolve that harness to a qualified runner.
> - Codex is the first harness in the staged default rollout.
> - Saved agents must retain their execution choices.
> - New Codex agents therefore use Paperclip Runner on qualified
macOS/Linux targets, with an explicit Legacy override.

## Linked Issues or Issue Description

Refs #16028, #15555, #15673, #15422, #15631, #15926, #15932, #16149.

The runtime/install prerequisite and release repair are merged. This
slice owns Codex creation, setup, configuration and preservation. Other
harness defaults remain unchanged. It adds no database migration;
incoming master migrations, required maps and regression coverage are
retained.

## What Changed

- Resolve `runner: auto | paperclip | legacy` at the common creation
boundary across onboarding, API/CLI, hires, built-ins, plugins and
imports. Freeze pending hires and export resolved execution choices.
- Preserve existing runners on upgrades, unrelated edits and import
updates, including compatible accounts, workspace settings, Codex effort
and intentional clears.
- Use the underlying harness picker and branding; put the runner
override in Advanced. Use the shared dropdown and update
production-component stories and creation contracts.
- Support qualified Apple Silicon, Intel Mac, Linux x64 and Linux ARM64.
Linux requires GNU libc 2.39+. Disabled, external, unsupported and
unqualified combinations retain their applicable Legacy execution.
- Test the selected runtime/account/model/environment. Ordinary Connect
probes eligible immutable managed targets and refreshes scoped
discovery. Creation/discovery do not provision. Auth, dependency,
incompatible-setting and cleanup failures remain actionable without
changing runners.
- Keep setup probes read-only, preserve local/SSH isolation and resolve
packaged assets before staging credentials. Bound cleanup reads by the
remaining absolute deadline; physically close every owned ingress socket
before endpoint release. Unconfirmed cleanup retains recovery state.
- Preserve validated completed-onboarding callbacks through startup
recovery. First file-producing work belongs to the child task and uses
the existing dependency/block handoff and task-scoped registration.
- Reuse existing E2E fixtures, artifact checks, reporting and controls.
Keep Docker/Cargo/native compilation remote. Paperclip tarballs contain
no Codex payloads; official optional npm platform dependencies remain.

## Verification

Current tested head: `1c03a8815672d82c264a9b08d4ea1984e8fae107`.
**Merge-ready: required CI, fresh review, package qualification, Mac
core acceptance and Test task/follow-up acceptance pass.** The existing
attachment and native-continuity oracles independently verify both Test
downloads, run attribution and unchanged native/provider/workspace
identities. The maintainer authorized a code-owner-review exception for
this PR only. The first Test setup failure and normal Board
completion-review friction are retained below. Actual post-merge
packages and private Cloud distribution remain separate qualification
gates.

- [Current
CI](https://github.com/paperclipai/paperclip/actions/runs/38564241625):
all 48 ordinary jobs and required verify/E2E pass. Attempt 1 retained
six simultaneous host-shutdown cancellations. One failed-only retry
passes those six lanes and two aggregates, preserving 40 successes.
Actual integration checkout `53104c1` combines this head with master
`c80`; it is not relabeled as the branch tree. Fresh exact-head Greptile
5/5, 30 resolved threads, no actionable findings.
- [Exact-source package
qualification](https://github.com/paperclipai/paperclip/actions/runs/38565287239):
nine applicable jobs pass, nine conditional jobs skip. Four remote
native producers; Linux x64 clean install; Intel Mac, Apple Silicon and
Linux ARM64 native startup/physical cleanup controls. All 18
tarballs/four daemon source/hash/manifest bindings and 343 migrations
through 0344 verify. No Codex payload is packed. These hosted tarballs
are not a published npm canary.
- Normal Apple Silicon installation and credential-free native
startup/cleanup pass using the packaged daemon, with no source build or
provider turn. Fresh installed DB initialization passes. Upgrade
651a→1c03 preserves all three complete execution configs, accounts, ten
recorded run profiles, paused budgets, active incidents, cost summaries
and two historical unpriced events. Historical recovery state is
retained.
- Cleanup regressions: four delayed-read baseline failures become 15
passing deadline/authority controls. No-acknowledgement and prior-error
socket ownership regressions become 12 passing controls. Direct
runtime/server types, maps: 35 areas / 176 IDs, diff checks and
independent reviews pass. Permission/auth/reconnect policy and the
10-second setup budget remain intact.
- Existing source coverage includes creation/selection/imports, pending
approvals, recovery, account/lifecycle handling, mounted forms,
dropdowns and E2E support. Full typechecks/tests/build/token and
applicable Storybook checks pass in the recorded current CI; no
duplicate suite or weakened grader.
- Literal 9546 live Mac proof: ordinary Auto Codex onboarding, useful
document, real completed-child callback/reply, seven successful native
runs, manual follow-up and independently downloaded1,101-byte child
file. Existing Legacy title edit/reload preserves execution; two real
Legacy runs produce independently checked15/34-byte files. These remain
historical source records.
- Current literal 1c03 Mac core acceptance passes independent checks:
ordinary Auto native Codex onboarding without runner override; only
timeout 180 seconds/concurrency 1 saved; six successful native runs and
six serialized turns (maximum overlap 1); both same-task continuity
checks and five mechanical callback checks; actual rendered parent reply
and exact child downloads. The useful handoff advances from revision 1
to 2. Both files are 48 bytes, SHA-256
`8b742bd4e8123a7c078f0a8d9f3449fbfa8a320d7d90be641a2818db3c53d92c` and
`f42db9774fa06e804ab7d596fc8b922a2b638921c418b87d0111151d290a6afe`;
exactly one byte changes and old history remains. The agent is paused
with zero active runs/errors. Both artifact work products have
`isPrimary=false`, and descriptive titles fail a stricter title-helper
assumption; these retained results are scoped editorial follow-up, not
new merge gates. Six source-bound screenshots are uploaded and verified
in the private primary QA report.
- [Literal 9546 managed
proof](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-38485488621-1/index.html):
original 19 assertions and completed judge pass, four native turns,
child-bound240-byte file and session/workspace continuity. Checkout
reports a verified generated target-lock mutation; full final porcelain
was not retained. Signed execution-image identity remains separate from
Core source.
- Actual 651a Test deployment succeeds on one owned/pinned target. One
ordinary Auto API-key Connect then fails native cleanup before hire.
Controller turn/drain/suspend completion and service-recorded
environment cleanup are observed; they do not certify physical deletion
or native authority retirement. Failure/recovery/unknown holds are
retained. No Legacy switch or unchanged paid retry. The two source
repairs above require actual treatment.
- [Current immutable Core
preview](https://github.com/paperclipai/paperclip/actions/runs/38570365725)
and matching DB/shared artifacts pass. One owned, pinned Test target
serves exact `1c03`; identical migration history correctly skips the
migrator. Ordinary Auto Codex Connect succeeds and hires native; saved
native selection and QA bounds survive reload. The initial child
preparation attempt fails before provider work, with actual stdout
identifying `renewal_deadline`; its trigger remains unknown and the
failure is retained. Normal Board **Try again** and **Resume** recover
the child to Done, producing a browser-verified 44-byte file, SHA-256
`0a8458b1efcc2d7a0d815689458c4caab039a46c7c7a3e5805de6bc8b1a96ecc`. The
actual parent callback reports the completed child document/file. The
existing validator requires a parent-owned attachment, so recovery
enters normal Review; after byte checking, normal **Approve completion**
reaches parent Done. No forced status or duplicate parent file is used.
The validator is unchanged from integrated and current master.
Autonomous initial callback-to-Done is not proven. The same-child
follow-up completes document revision 2 and publishes a second
independently downloaded 44-byte file, SHA-256
`22022257300fae762784632d08d3bbe0bab61e2ce81f91864a14dcd7aa464591`; only
the revision digit changes. The unchanged existing native-continuity and
attachment oracles pass. All ten recorded runs are terminal: nine native
successes and the retained pre-provider setup failure. The QA agent is
paused with zero active runs. The pre-existing parent evidence scope
mismatch is a separate follow-up, with ordinary completion review
exercised rather than a forced status or relaxed validation. This
unsigned Core preview remains distinct from post-merge private Cloud
composition.

## Risks

- Saved agents are not migrated. Explicit runner changes use normal
configuration/session rules; historical runs retain their recorded
execution.
- Native readiness depends on verified assets and server-owned target
qualification. A Legacy probe, HTTP200 or successful account save cannot
prove native readiness.
- Recovery exceptions remain restricted to validated
completed-onboarding reports; process-loss, ownership and terminal-state
safeguards remain active. Unexecuted live journeys stay open.
- Linux ARM paid tasks, Windows, Alpine and 32-bit ARM remain deferred.
Cold Cargo cache misses are recorded; no warm-cache speedup is claimed.
- Historical invoices/physical cleanup remain unconfirmed and held.
After merge, exact published npm canary, signed standard image/migrator,
private composition and serving/live acceptance still require
qualification. Production deployment/default promotion is not
authorized.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tools, code execution and
parallel agents. Exact serving snapshot and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or PRs above
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run the stated focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green on the current candidate
- [x] Greptile is 5/5 with no open findings on the current candidate
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 11:33:02 -05:00

12 KiB

Recovery — agent access

Product map: Recovery.

Review date: 2026-10-12. Source commit: 58e16dbddfc9c8d0cf60867f9d9847f0feda055b. Native REST transport reconciled in this change; see the shared transport contract.

This is a source-reviewed baseline of representative operations and known gaps. Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the map contract for format, permission checks, evidence and maintenance.

diagnosis

Review: baseline.

API: present — representative operation IDs:

  • GET /api/issues/{id}/diagnostics/blockers.
  • GET /api/issues/{id}/recovery-actions.

Agent access: available — REST / native call_api; CLI issue diagnostics.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Diagnostics expose recorded cause and boundaries; source-backed reading does not prove successful recovery.

bounded-continuation

Review: baseline.

API: present — representative operation IDs:

  • POST /api/issues/{id}/recovery-actions/resolve.

Agent access: partial — Eligible agent REST / native call_api / CLI under source-bound recovery authority.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Eligible resolutions use native call_api. The route retains owner and management authority, active-run/source-trust checks, board-only outcomes and reconciliation controls.

Scoped source review: 2026-10-13, recovery correction against ea6c8eb5e16d61111968f633a66748b397b64a3e. The recovery service reuses the existing completed-onboarding handoff validator. It checks the source company, onboarding origin, Done parent, assignee and Done child, with no unfinished siblings, before exempting the report from the issue-terminal backstop. Other terminal work and process-death/controller ownership checks remain enforced. This internal wake handling adds no recovery-resolution or cross-company agent authority. Real PostgreSQL regressions cover allowed and rejected bindings; revised live agent authorization and callback delivery remain not-run.

source-action

Review: baseline.

API: present — representative operation IDs:

  • GET /api/issues/{id}/recovery-actions.
  • POST /api/issues/{id}/recovery-actions/resolve.

Agent access: partial — Eligible agent REST / native call_api / CLI; source-bound recovery state.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. Completion or cancellation of a task does not clear its exact unresolved native workspace owner hold. Physical owner release still requires the operator stop-verification procedure.

For a resolved legacy action that retains a file hold, the same resolution API supports source-only file repair. Board runtime management and recovery-resolution authority are required. The request identifies the exact source action/run and records recovered-file evidence; the server verifies the original stop receipt. It preserves the current task status and does not wake or replay the agent. The task form fails closed without permission or source bindings. Agent Chat retains the file diagnosis while this hold exists. The repair route regressions and file form cover the local contract; they do not independently recover files or qualify a live provider.

Evidence: source-only — implementation at the reviewed commit above. Exact owner hold retention in the recovery state helper was reviewed at 89770ba707fb1ebd4cabd184eec2e7fee226e0c3.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability. Local recovery route regressions verify one unchanged hold across reads and sweeps on done/cancelled tasks, no physical copyback, and cleanup after the receipt is cleared. Exercise this sequence in a disposable instance; automatic cleanup must use the system actor and must not create reader participation. No live recovery proof is claimed.

Gaps: Permitted resolutions use native call_api under existing eligibility and ownership checks. Source/action policy determines whether continuation is allowed.

exhaustion

Review: baseline.

API: present — representative operation IDs:

  • GET /api/issues/{id}/recovery-actions.
  • GET /api/heartbeat-runs/{runId}.

Agent access: partial — REST / native call_api reads; operator escalation flow.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Reading exhaustion is supported. Continuing past the recorded gate requires the specified owner/action.

workspace

Review: baseline.

API: present — representative operation IDs:

  • POST /api/issues/{id}/recovery-actions/retry-workspace-export.
  • POST /api/issues/{id}/recovery-actions/resume-workspace-finalization.
  • POST /api/issues/{id}/recovery-actions/resolve for retained-source file repair.
  • POST /api/execution-workspaces/{id}/reconcile-branch.

Agent access: restricted — Board/runtime management APIs; agent reads diagnosis and performs permitted workspace work.

Policy: Export retry and branch reconciliation require board runtime management. Retained legacy file-only repair requires the board, exact stopped-source receipts, and recovered-file evidence. It can record repair for a successful legacy turn without changing its result or authorizing replay. Queued and running sources remain denied. Source-only reconciliation regressions cover this boundary. Physical-owner stop confirmation uses the separate runtime:confirm_owner_stop action. It requires active company owner/admin membership, trusted local board access, or instance administration. Agents and ordinary runtime managers are denied. The exact source task, running native run, accepted result and owner receipt must match; active copyback locks, live local writers, and newer runs prevent admission. An authorized administrator must first verify through the deployment platform that the exact controller and all copyback children stopped, then attest that evidence. Elapsed time, an expired coordinator lease or a lost advisory connection is not proof. Only the exact owner receipt is released; normal reconciliation performs export and commitment without another provider turn or bypassing existing task waits.

Evidence: source-only — route, administrator authorization, and owner admission reviewed for this change. Local authorization, route, UI and finalization regressions cover member/agent denial, exact-owner admission and the real reconciler preserving monitor and approval waits. This is not live provider-stop proof.

Verification: not-run for live agent/deployment execution — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Export retry and branch reconciliation require board runtime:manage at their HTTP routes. The exact-owner operator fallback additionally requires runtime:confirm_owner_stop. Native call_api preserves these agent denials. Automatic managed-cloud recovery uses independently verified, signed platform proof through the finalization reconciler; agents cannot attest owner shutdown.

unknown-outcome

Review: baseline.

API: present — representative operation IDs:

  • POST /api/chat-endpoints/{endpointId}/publications/{publicationId}/resolve.

Agent access: restricted — Board provider-outcome resolution; agent may inspect assigned provider tools.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Uncertain-side-effect resolution is operator/source-specific; do not replay to obtain apparent parity.

user-continuation

Review: baseline.

API: present — representative operation IDs:

  • POST /api/issues/{id}/comments.

Agent access: partial — User comment API and server-owned wake reconsideration; native agent consumes bound wake.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Receiving the continuation is supported; impersonating a saved user message or retry control is not.

reconnect

Review: baseline.

API: present — representative operation IDs:

  • GET /api/companies/{companyId}/events/ws.
  • GET /api/issues/{id}.

Agent access: partial — Existing event protocol; REST refresh of persisted task state.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability.

Gaps: Streaming reconnect needs its protocol client. Polling/read APIs remain available; replay is not proof of a fresh successful action.