0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-21 00:05:49 +02:00

Files

T

cafa25029c fix(native-runtime): recover unclaimed remote startup (#16349 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runtime preserves task execution across server restarts.
> - Remote workspace preparation can finish before the executor claims
its first provider attempt.
> - The recovery classifier required a provider checkpoint even when the
durable coordinator proved that execution had never started.
> - A deployment during this interval therefore left safe startup work
on a permanent recovery hold.
> - This pull request admits that narrow startup state to the existing
same-run bootstrap path. Existing execution and ownership checks remain
in force.
> - The original preparation yields when recovery owns the coordinator.
It cannot fail or clean up the recovered run.
> - Tasks can continue after this interruption without an operator retry
or a duplicate provider turn.

## Linked Issues or Issue Description

**What happened?**

A server deployment interrupted a prepared remote workspace before the
native executor's first claim. The coordinator had attempt zero,
controller generation zero, and no owner, process, provider event, or
checkpoint. Restart classification returned
`ambiguous_remote_recovery_evidence` and blocked the task.

**Expected behavior**

The server should recognize this durable never-claimed state and
continue the original run through normal execution admission. A prior
claim, uncertain ownership, or contradictory execution evidence must
retain the recovery hold.

**Steps to reproduce**

1. Prepare a remote workspace for a native run.
2. Stop the controller before the executor commits its first claim.
3. Restart the controller and classify the running native task.
4. Observe that the old classifier blocks it because no provider
checkpoint exists.

Related: #16341 preserves remote recovery failure reasons. #16344
strengthens remote restart qualification. Neither changes admission for
an untouched remote startup.

## What Changed

- Require a server-owned fencing version written only with fresh native
identities. Older unmarked preparations stay held during rolling
upgrades; caller profile data cannot add that marker to an existing run.
- Leave the current server's active preparation with its existing
execution.
- Yield through the controller-detachment path when a different
execution owns the coordinator lease. Lease expiry does not grant
takeover authority.
- Preserve run-scoped gateway tokens during detachment so stale
preparation cannot revoke the recovered worker's tools.
- Admit a valid prepared remote workspace only when the locked
coordinator proves attempt and controller generation zero, no owner, and
no prior session or execution evidence.
- Check launch receipts and provider source events before granting
bootstrap recovery.
- Keep the original run and session. Preserve atomic recovery claims,
executor durable-state checks, and normal admission gates.
- Add real PostgreSQL regression tests for competing controllers and
seven contradictory evidence cases.
- Update recovery documentation, the run-log reason, and both capability
maps.

## Verification

- The competing-controller regressions failed before the ownership fix
and pass for both live and expired foreign leases.
- On the final revision, 77 runtime-selection, restart recovery, real
PostgreSQL, and heartbeat boundary tests pass with no skips, including
all local runner process fixtures. The full native executor file passes
all 698 tests.
- Another 91 live-run route tests pass locally. An earlier CI timeout in
that unrelated file passed on retry; the complete final-head CI passed
without that failure.
- `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and
`git diff --check`: passed locally. The map check verifies structure
only.
- Local `pnpm test:run` did not finish: an early attempt had
build-related collection failures, and later full-suite execution was
stopped after complete CI passed. Targeted suites passed after build
preparation. This is not a local full-suite pass.
- The complete [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/38669996667)
passed on `2baffb3652bfeb4705a30ca3706fbda6c3d72aa4`, including all test
shards, build/typecheck, runner verification, and browser E2E. All 55
applicable review/security/CI checks succeeded; two Storybook checks
were skipped.
- [Greptile
Apex](https://github.com/paperclipai/paperclip/pull/16349#issuecomment-6180848957):
5/5 on the same final head, with no unresolved findings.
- No live remote provider restart is claimed by the database regression.

## Risks

- A bare executor can no longer take a foreign coordinator lease after
expiry. A verified recovery claim must transfer authority first.
- Existing preparations without the server-owned fencing marker retain
their recovery hold during rolling upgrades.
- Incorrect admission could duplicate provider work. This path requires
a positive durable never-claimed receipt, rejects contradictory launch
evidence, and keeps the executor's existing durable-root verification.
- Previously claimed or incomplete remote recovery evidence remains
blocked. This change does not clear historical terminal holds.
- No schema, API, dependency, or UI changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code execution,
and browser inspection. The exact serving snapshot ID and context window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>

2026-10-14 19:32:39 -07:00

14 KiB

Raw Blame History

Recovery — agent access

Product map: Recovery.

Review date: 2026-10-12. Source commit: 58e16dbddfc9c8d0cf60867f9d9847f0feda055b. Native REST transport reconciled in this change; see the shared transport contract.

This is a source-reviewed baseline of representative operations and known gaps. Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the map contract for format, permission checks, evidence and maintenance.

`diagnosis`

Review: baseline.

API: present — representative operation IDs:

GET /api/issues/{id}/diagnostics/blockers.
GET /api/issues/{id}/recovery-actions.

Agent access: available — REST / native call_api; CLI issue diagnostics.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. The server can recover a prepared remote bootstrap only with a durable never-claimed executor receipt and no contradictory execution evidence. This automatic classification grants no new agent action or permission; normal execution admission and durable runner-state verification still apply. The current server's active preparation remains with its existing execution. A preparation that loses the coordinator lease cannot fail the shared run or revoke the new owner's gateway tokens, even after lease expiry. Only normal recovery can establish a takeover claim; no agent permission is added. Older unmarked preparations do not qualify for this automatic path during an upgrade. The fencing marker comes from fresh server-owned identity creation; caller profile data cannot upgrade an existing preparation.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Diagnostics expose recorded cause and boundaries; source-backed reading does not prove successful recovery.

`bounded-continuation`

Review: baseline.

API: present — representative operation IDs:

POST /api/issues/{id}/recovery-actions/resolve.

Agent access: partial — Eligible agent REST / native call_api / CLI under source-bound recovery authority.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Eligible resolutions use native call_api. The route retains owner and management authority, active-run/source-trust checks, board-only outcomes and reconciliation controls.

Scoped source review: 2026-10-13, recovery correction against ea6c8eb5e16d61111968f633a66748b397b64a3e. The recovery service reuses the existing completed-onboarding handoff validator. It checks the source company, onboarding origin, Done parent, assignee and Done child, with no unfinished siblings, before exempting the report from the issue-terminal backstop. Other terminal work and process-death/controller ownership checks remain enforced. This internal wake handling adds no recovery-resolution or cross-company agent authority. Real PostgreSQL regressions cover allowed and rejected bindings; revised live agent authorization and callback delivery remain not-run.

`source-action`

The recovery-actions read API includes settled execution reconciliation holds with an automatic replay: blocked receipt, as well as retained workspace holds. The existing company and task-read checks remain in force. The board notice reads this history to explain its current hold; this adds no agent mutation authority. Route and notice regressions cover the source contract, without claiming live agent or provider qualification.

Review: baseline.

API: present — representative operation IDs:

GET /api/issues/{id}/recovery-actions.
POST /api/issues/{id}/recovery-actions/resolve.

Agent access: partial — Eligible agent REST / native call_api / CLI; source-bound recovery state.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board. Completion or cancellation of a task does not clear its exact unresolved native workspace owner hold. Physical owner release still requires the operator stop-verification procedure.

For a resolved legacy action that retains a file hold, the same resolution API supports source-only file repair. Board runtime management and recovery-resolution authority are required. The request identifies the exact source action/run and records recovered-file evidence; the server verifies the original stop receipt. It preserves the current task status and does not wake or replay the agent. The task form fails closed without permission or source bindings. Agent Chat retains the file diagnosis while this hold exists. The repair route regressions and file form cover the local contract; they do not independently recover files or qualify a live provider.

Evidence: source-only — implementation at the reviewed commit above. Exact owner hold retention in the recovery state helper was reviewed at 89770ba707fb1ebd4cabd184eec2e7fee226e0c3.

Verification: not-run — exercise a listed permitted operation with actual agent credentials/run context; inspect its result and applicable persistence/attribution. Assert the relevant denied actor, owner, company or mode from Policy. Review the remaining operations before qualifying the whole capability. Local recovery route regressions verify one unchanged hold across reads and sweeps on done/cancelled tasks, no physical copyback, and cleanup after the receipt is cleared. Exercise this sequence in a disposable instance; automatic cleanup must use the system actor and must not create reader participation. No live recovery proof is claimed.

Gaps: Permitted resolutions use native call_api under existing eligibility and ownership checks. Source/action policy determines whether continuation is allowed.

A recorded operator Stop before legacy runtime selection releases only that run's task locks and preserves deferred input for normal later admission. This does not grant an agent a new retry or provider-stop authority. Local database regressions cover the release boundary and a competing active routine cycle; they are not live provider-stop proof.

`exhaustion`

Review: baseline.

API: present — representative operation IDs:

GET /api/issues/{id}/recovery-actions.
GET /api/heartbeat-runs/{runId}.

Agent access: partial — REST / native call_api reads; operator escalation flow.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Reading exhaustion is supported. Continuing past the recorded gate requires the specified owner/action.

`workspace`

Review: baseline.

API: present — representative operation IDs:

POST /api/issues/{id}/recovery-actions/retry-workspace-export.
POST /api/issues/{id}/recovery-actions/resume-workspace-finalization.
POST /api/issues/{id}/recovery-actions/resolve for retained-source file repair.
POST /api/execution-workspaces/{id}/reconcile-branch.

Agent access: restricted — Board/runtime management APIs; agent reads diagnosis and performs permitted workspace work.

Policy: Export retry and branch reconciliation require board runtime management. Retained legacy file-only repair requires the board, exact stopped-source receipts, and recovered-file evidence. It can record repair for a successful legacy turn without changing its result or authorizing replay. Queued and running sources remain denied. Source-only reconciliation regressions cover this boundary. Physical-owner stop confirmation uses the separate runtime:confirm_owner_stop action. It requires active company owner/admin membership, trusted local board access, or instance administration. Agents and ordinary runtime managers are denied. The exact source task, running native run, accepted result and owner receipt must match; active copyback locks, live local writers, and newer runs prevent admission. An authorized administrator must first verify through the deployment platform that the exact controller and all copyback children stopped, then attest that evidence. Elapsed time, an expired coordinator lease or a lost advisory connection is not proof. Only the exact owner receipt is released; normal reconciliation performs export and commitment without another provider turn or bypassing existing task waits.

Evidence: source-only — route, administrator authorization, and owner admission reviewed for this change. Local authorization, route, UI and finalization regressions cover member/agent denial, exact-owner admission and the real reconciler preserving monitor and approval waits. This is not live provider-stop proof.

Verification: not-run for live agent/deployment execution — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Export retry and branch reconciliation require board runtime:manage at their HTTP routes. The exact-owner operator fallback additionally requires runtime:confirm_owner_stop. Native call_api preserves these agent denials. Automatic managed-cloud recovery uses independently verified, signed platform proof through the finalization reconciler; agents cannot attest owner shutdown.

`unknown-outcome`

Review: baseline.

API: present — representative operation IDs:

POST /api/chat-endpoints/{endpointId}/publications/{publicationId}/resolve.

Agent access: restricted — Board provider-outcome resolution; agent may inspect assigned provider tools.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Verification: not-run — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review.

Gaps: Uncertain-side-effect resolution is operator/source-specific; do not replay to obtain apparent parity.

`user-continuation`

Review: baseline.

API: present — representative operation IDs:

POST /api/issues/{id}/comments.

Agent access: partial — User comment API and server-owned wake reconsideration; native agent consumes bound wake.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Receiving the continuation is supported; impersonating a saved user message or retry control is not.

`reconnect`

Review: baseline.

API: present — representative operation IDs:

GET /api/companies/{companyId}/events/ws.
GET /api/issues/{id}.

Agent access: partial — Existing event protocol; REST refresh of persisted task state.

Policy: Task/run visibility, run ownership and source/action policy; workspace export retry and some operator controls require board.

Evidence: source-only — implementation at the reviewed commit above.

Gaps: Streaming reconnect needs its protocol client. Polling/read APIs remain available; replay is not proof of a fresh successful action.

14 KiB Raw Blame History

Recovery — agent access

diagnosis

bounded-continuation

source-action

exhaustion

workspace

unknown-outcome

user-continuation

reconnect

14 KiB

Raw Blame History

`diagnosis`

`bounded-continuation`

`source-action`

`exhaustion`

`workspace`

`unknown-outcome`

`user-continuation`

`reconnect`