fix(runtime): reclaim failed remote checkpoint retries (#18877)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must preserve the original execution and provider
state.
> - A failed remote run can retain its provider PID after heartbeat
releases the task execution lock.
> - Normal resumption excludes that PID, while restart recovery rejects
the released lock.
> - Recovery must reclaim that exact execution only after current
admission and durable ownership checks pass.
> - Failed retries and admission holds need separate finite budgets.
Neither can prove a provider stopped.

## Linked Issues or Issue Description

**What happened?**

A failed remote native run remains in retryable failure without a task
execution owner. Normal resumption and restart recovery both reject it
before inspecting the preserved remote provider. The current remote
workspace reference format also misses the due-retry routing path.

**Expected behavior**

A due exact-checkpoint retry reclaims its original task execution only
when current ownership and safety gates permit it. Newer task input
prevents an old claim. Lost or ambiguous provider state still prevents
replay or replacement. Ineligible recovery eventually becomes a quiet
manual reconciliation obligation.

**Steps to reproduce**

1. Start a remote native run and persist its provider checkpoint and
encrypted Runner authority.
2. Fail with exact-checkpoint recovery and a retained remote PID.
3. Release the task execution lock at heartbeat completion.
4. Run ordinary resumption and restart recovery after the retry
deadline.
5. Observe that both reject the same eligible run before remote
inspection.

**Paperclip version or commit**

Reproduced from source at `d59791828482`; integrated with upstream
`db9be4c5a6`.

**Deployment mode**

Remote workspaces with PostgreSQL. Local regressions use disposable
PostgreSQL and encrypted checkpoint persistence.

Related public repairs #18846, #18854, #18860 and #18841 cover finished
per-turn owners, pre-provider failures, unstarted follow-ups and storage
migration. They do not reclaim this failed run's released execution
lock. Merged #18879 also adds scheduled-resume lock retention and
reclaim. This PR keeps that compatibility and requires encrypted Runner
authority, canonical source ordering and a charged failed retry before
reclaim. Its scheduled-resume regression now uses a real encrypted
authority and retains a snapshot-only refusal control.

## What Changed

- Share normal admission checks and one guarded execution-binding
restore/event helper between normal and remote resumption.
- Keep agent-name normalization in a module with no imports. Preserve
heartbeat's existing export. Native admission no longer loads heartbeat
orchestration for a string key.
- Recheck company, assignment, agent eligibility, Stop, pause, approval,
dependencies, execution and checkout ownership before reclaim.
- Reject newer task input using PostgreSQL creation-time and ID
ordering, preserving timestamp precision.
- Require the recorded remote target, encrypted Runner authority, exact
checkpoint identity and existing controller fence.
- Commit task binding, controller lease, run status and recovery event
atomically. Recognize both remote workspace reference formats.
- Charge failed reattachments against the existing
three-provider-attempt budget, including retained-lock crash windows.
Active controller adoption preserves its charged attempt.
- Use the existing five-retry/30-minute system-recovery policy for
admission holds. Ended tasks and newer sources exhaust it immediately.
Preserve PID, checkpoint and task bindings, suppress resource
settlement, and assign manual reconciliation with executor stop/state
explicitly unverified.
- Register recovery intent before deadline writes; preserve live/future
deadlines quietly and roll back zero-row writes. Record changed hold
reasons once and terminal dispositions separately.
- Keep historical provider-budget exhaustion behind current
admission/source/controller/authority checks. Make an existing
orphan-cleanup fixture explicitly due.
- Include exact plugin task leases that have no workspace reference.
Reuse every reclaim and retry-budget guard, with a bounded
company/task/run lease projection.
- Share strict manual ownership proof across stale-lock cleanup,
adoption, queue release and wake admission. Require termination receipts
for every exact remote lease before normal cleanup can release either
binding.
- Preserve governing Stop through the existing audited cancellation
path. Its acknowledgement does not release an unverified provider.
Ordinary Retry can wait for exact Stop and file preservation, then queue
one continuation.

## Verification

- Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its
[current CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858)
is terminal and clean: 54 successful, three skipped and two neutral
contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on
this exact head with no open findings. No fresh Paperclip 5/5 assessment
is claimed.
- The [prior complete CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388)
passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and
Paperclip both 5/5 and no open threads.
- All 233 tests across the six affected heartbeat, restart-isolation,
remote-admission, scheduled-resume and session-binding families pass on
production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under
Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and
plugin task leases, competing controllers, exact rollback, Stop,
assignment/company boundaries, attempts, work-intent ordering, finite
backoff and quiet held state.
- The merge preserves #18931's per-run audit-failure isolation and this
PR's unchanged-transition hold. Two initial composition cases failed
because their setup or expected result described the older contract. The
rollback test now expects the exact per-run deferral while retaining
full coordinator equality and no work intent. The routine fixture writes
a real encrypted checkpoint through the normal owner before reaching the
actual routine unique constraint. No production admission guard or
rollback assertion was weakened. A separate agent reviewed this exact
merge and fixture delta without finding a blocker.
- Full recursive `pnpm -r typecheck` and `pnpm build` pass on production
merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits
change only three test files. Both the two-file `a5104fa` and final
`5d7baf` server test-layer typechecks pass.
- Four production-identical `a5104fa` probes pass with the rebuilt
actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL
checkpoints under Node 24.22.2. Workspace-reference and plugin-lease
admission each cover surviving and lost providers. Replacement preserves
one provider launch, one original side effect, the charged attempt and
restored task execution binding. Lost providers reject snapshot recovery
without replacement. This is protocol-boundary evidence, not a live
Claude or hosted provisioning claim.
- Initial `a5104fa` CI found one additional routine integration fixture
missing the persisted `exact_checkpoint_resume` mode. Its encrypted
checkpoint was already present. The exact case fails locally before
repair and passes after recording that mode and expecting the charged
provider attempt. Every exact routine conflict, rollback, checkpoint,
task binding, successor-count and sibling-startup assertion remains.
Final head changes only that fixture. The correct native-server group
passes all nine files: 175 executed cases and 14 container-only skips,
matching the now-successful native-server CI gate.
- The earlier canonical HTTP Retry acceptance passed on the prior
reviewed head with real PostgreSQL, the recovery worker and real files.
Pending Stop protects both bindings and starts no successor. Exact
termination and copyback queue one preserved-workspace continuation. No
browser or live-model outcome is inferred from that test.
- The predecessor's full local `pnpm test:run` failed in an unchanged
Discord Gateway fixture with a 200 ms lease and a 422 initial ownership
error. Focused predecessor and untouched-base controls pass; that
full-run cause remains unproved. The required command was also started
on production-identical `a5104fa`. All six local chat shards passed,
covering 1,850 cases. The duplicate local invocation was interrupted
with SIGINT while the general-server group was still running after this
head's complete CI and the combined repair qualification had passed. The
wrapper exited 1 on cancellation, no process retained its test log, and
no full local pass is claimed. An earlier attempt to select the native
lane through the CI wrapper instead selected local package sharding:
2,631 cases passed and 13 failed. Those separate failures are retained
without causal attribution or unrelated source changes; that run is not
counted as native-server qualification. The direct native group and
current complete CI matrix pass.
- Live Claude Opus 5.5 continuation, hosted plugin provisioning,
deployment and recovery of historical tasks remain unverified. No
original task was retried or mutated for this verification.

## Risks

- Restoring a task binding grants remote inspection; it does not prove
that the provider survived.
- Unknown effects, lost providers and missing semantic results retain
their existing reconciliation requirements.
- Exhausted admission holds preserve execution bindings and resource
evidence for manual reconciliation. They do not authorize a fresh
executor or provider/resource release.
- The source guard is conservative: newer task input prevents recovery
of an older failed source.
- No database migration or provider profile change is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and
shell execution. The runtime does not expose the exact model snapshot or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/modules/wake-queue/adapters/postgres.ts

@@ -14,6 +14,7 @@ import { isCompletedOnboardingHandoffWake } from "../../../services/chat-complet
 import { instanceSettingsService } from "../../../services/instance-settings.js";
 import { currentConversationCommentCondition } from "../../../services/agent-conversations.js";
 import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/execution-blocker.js";
+import { nativeManualAdmissionHoldsExecution } from "../../../services/native-runtime/native-runner-ownership.js";
 import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm";
 import { readReleaseRun, type ReleaseRun } from "./release-run-read.js";
 import type { Db } from "@paperclipai/db";
@@ -1189,6 +1190,9 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd
             : [];
           if (!finalized) return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
         }
+        if (await nativeManualAdmissionHoldsExecution(tx, { companyId: input.companyId, runId: run.id })) {
+          return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
+        }
         if (input.resumeStrandedQueue && await hasPendingRunCleanup(tx, input.companyId, run.id)) {
           return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
         }