Files
PaperClipAI/server
cafa25029c fix(native-runtime): recover unclaimed remote startup (#16349)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runtime preserves task execution across server restarts.
> - Remote workspace preparation can finish before the executor claims
its first provider attempt.
> - The recovery classifier required a provider checkpoint even when the
durable coordinator proved that execution had never started.
> - A deployment during this interval therefore left safe startup work
on a permanent recovery hold.
> - This pull request admits that narrow startup state to the existing
same-run bootstrap path. Existing execution and ownership checks remain
in force.
> - The original preparation yields when recovery owns the coordinator.
It cannot fail or clean up the recovered run.
> - Tasks can continue after this interruption without an operator retry
or a duplicate provider turn.

## Linked Issues or Issue Description

**What happened?**

A server deployment interrupted a prepared remote workspace before the
native executor's first claim. The coordinator had attempt zero,
controller generation zero, and no owner, process, provider event, or
checkpoint. Restart classification returned
`ambiguous_remote_recovery_evidence` and blocked the task.

**Expected behavior**

The server should recognize this durable never-claimed state and
continue the original run through normal execution admission. A prior
claim, uncertain ownership, or contradictory execution evidence must
retain the recovery hold.

**Steps to reproduce**

1. Prepare a remote workspace for a native run.
2. Stop the controller before the executor commits its first claim.
3. Restart the controller and classify the running native task.
4. Observe that the old classifier blocks it because no provider
checkpoint exists.

Related: #16341 preserves remote recovery failure reasons. #16344
strengthens remote restart qualification. Neither changes admission for
an untouched remote startup.

## What Changed

- Require a server-owned fencing version written only with fresh native
identities. Older unmarked preparations stay held during rolling
upgrades; caller profile data cannot add that marker to an existing run.
- Leave the current server's active preparation with its existing
execution.
- Yield through the controller-detachment path when a different
execution owns the coordinator lease. Lease expiry does not grant
takeover authority.
- Preserve run-scoped gateway tokens during detachment so stale
preparation cannot revoke the recovered worker's tools.
- Admit a valid prepared remote workspace only when the locked
coordinator proves attempt and controller generation zero, no owner, and
no prior session or execution evidence.
- Check launch receipts and provider source events before granting
bootstrap recovery.
- Keep the original run and session. Preserve atomic recovery claims,
executor durable-state checks, and normal admission gates.
- Add real PostgreSQL regression tests for competing controllers and
seven contradictory evidence cases.
- Update recovery documentation, the run-log reason, and both capability
maps.

## Verification

- The competing-controller regressions failed before the ownership fix
and pass for both live and expired foreign leases.
- On the final revision, 77 runtime-selection, restart recovery, real
PostgreSQL, and heartbeat boundary tests pass with no skips, including
all local runner process fixtures. The full native executor file passes
all 698 tests.
- Another 91 live-run route tests pass locally. An earlier CI timeout in
that unrelated file passed on retry; the complete final-head CI passed
without that failure.
- `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and
`git diff --check`: passed locally. The map check verifies structure
only.
- Local `pnpm test:run` did not finish: an early attempt had
build-related collection failures, and later full-suite execution was
stopped after complete CI passed. Targeted suites passed after build
preparation. This is not a local full-suite pass.
- The complete [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/38669996667)
passed on `2baffb3652bfeb4705a30ca3706fbda6c3d72aa4`, including all test
shards, build/typecheck, runner verification, and browser E2E. All 55
applicable review/security/CI checks succeeded; two Storybook checks
were skipped.
- [Greptile
Apex](https://github.com/paperclipai/paperclip/pull/16349#issuecomment-6180848957):
5/5 on the same final head, with no unresolved findings.
- No live remote provider restart is claimed by the database regression.

## Risks

- A bare executor can no longer take a foreign coordinator lease after
expiry. A verified recovery claim must transfer authority first.
- Existing preparations without the server-owned fencing marker retain
their recovery hold during rolling upgrades.
- Incorrect admission could duplicate provider work. This path requires
a positive durable never-claimed receipt, rejects contradictory launch
evidence, and keeps the executor's existing durable-root verification.
- Previously claimed or incomplete remote recovery evidence remains
blocked. This change does not clear historical terminal holds.
- No schema, API, dependency, or UI changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code execution,
and browser inspection. The exact serving snapshot ID and context window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-10-14 19:32:39 -07:00
..