Files
PaperClipAI/doc/plugins/PLUGIN_RUNNER_EXECUTION.md
T
DottaandClaude Opus 5.5 c45f54ee74 fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown

## Linked Issues or Issue Description

**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.

**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:

1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.

The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.

**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.

**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.

## What Changed

- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.

## Verification

- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches 7e8901aa60, not current master, so each wave
ran this change cherry-picked onto 7e8901aa60:
- `w-20261111T210049-s6111` (first commit only): the close reason is now
logged: `close failed: Daytona sandbox lease 192606aa… is no longer
active.` The delete stayed 409 with that reason. The new check failed
the wave correctly.
- `w-20261111T222826-s6112` (all commits): two W2Q owners were released
with `sandbox was already destroyed`. The environment was deleted. 0
sandboxes left running. The warm-shutdown check passed. No 409 from a
warm session.
- Not verified: a wave where the graceful close fails and the forced
Launcher stop succeeds on a live sandbox (that path is covered by the
unit test). The wave's task failures (`T:first_run_succeeded`,
`R3:retry_needed`) are other open defects. They are also on master.

## Risks

- Environment delete now stops an idle Runner whose close did not
settle. The delete already has the operator's consent to destroy those
sandboxes. The workspace-export and restore-source guards on sandbox
destruction are unchanged. Instruction files are collected after the
confirmed stop, as in #20844.
- An owner is released without contact only when a lease for the same
company, environment and sandbox has a valid `destroyed` termination
receipt. A `stopped` receipt is not enough, because a stopped sandbox
can be resumed.
- `closeIdleWarmNativeSessions` results have a new `failures` field.
Callers that read only the counts are unchanged.
- No persisted state changes. Warm owners live only in controller
memory, so older saved state and mid-run deploys are not affected.
- The new harness check can fail a warm restart-fault wave that leaves a
warm session unclosed at controller shutdown. That is a real leaked
Runner.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs, chaos waves) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:12:21 -06:00

31 KiB

Plugin Runner execution

Core can run paperclip_runner tasks in a plugin environment. The plugin owns the execution host and its files. Core owns run identity, selected policy, assigned tools, and the PRP journal.

Managed AI connection required

A task-capable environment runs only agents with a managed AI connection. Core does not use an account installed on the plugin's execution host. Before it acquires a lease or calls the plugin, Core fails a new run that has no managed AI connection. A homeMode: "preconfigured" Runner home does not use the agent's AI connection, so Core refuses it in the same way. The run fails as configuration_incomplete with reason ai_connection_required. The task is blocked with a notice that names the fix: choose an AI connection, or switch to an isolated Runner home. Agent verification applies the same rule before it resolves the probe target, so it acquires no lease. The agent Test route also checks before it resolves secrets or allocates a test target. A setup probe without a managed AI connection, or with a preconfigured home, fails the ai_connection_required check and calls no task operation. Verification then waits for the configuration to change. A resumed native run keeps the contract it was admitted under.

Required hooks

Set supportsTasks: true on the environment driver. Declare the environment.drivers.register capability. Implement these hooks:

  • onEnvironmentAcquireLease
  • onEnvironmentTask
  • onEnvironmentReleaseLease
  • onEnvironmentDestroyLease

Task execution does not require onEnvironmentExecute or onEnvironmentRealizeWorkspace. The task API has no shell command or file installation operation. Use a separate workflow to install agent files. Providers that advertise workspaceFileReadVersion: 1 can return task output files through the bounded read_file operation for attachments and deliverables.

Core supports per-turn workspace sessions with Codex, OpenCode, and ACPX. OpenCode and ACPX require the existing qualified provider profiles. Workspace-free providers are not supported here. Drivers opt into warm retention through warmTaskVersion: 1. Core prefers warm execution for new task runs on capable drivers, including ACPX; existing provider admission still determines which harnesses the driver supports. The plugin owns project workspace preparation. Core does not prepare or reuse a Core Git worktree for a task-only environment. Selecting a plain task directory does not add redundant default configuration. Retaining that directory does not itself change the run configuration. For plugin-backed project storage, Core uses the workspace provider contract. The storage plugin must also provide the selected execution environment. Its declared workspace modes and strategies remain in effect. Core passes the pinned workspace binding with each task operation. Core rejects configured workspace services for task-only environments. Fresh retries that import retained files are not supported for plugin storage. An agent-owned workspace cannot move to another agent or environment. Core also rejects reuse of a local workspace in a task-only environment. Keep its original owner, or use the workspace storage provider contract.

Prepare and launch

  1. Return an opaque task ID from lease acquisition as providerLeaseId.
  2. Return the installed Runner version and SHA-256 digest from describe.
  3. Return launchPreparationVersion: 1 from describe.
  4. Handle prepare with the launch preparation contract.
  5. Save the result under its configuration ID.
  6. Handle submit with that ID in preparedLaunchId.
  7. Return an authenticated WSS endpoint from connection.

Preparation must not start a process. It must preserve Core's provider, model, policy, instructions, and resource assignment identities. Resolve assigned resources to execution-host paths. Use bindRunnerLaunchResources and createRunnerProviderLaunch when the built-in descriptor builder is suitable.

Core checks the returned requirements digest and descriptor before it registers PRP or submits the task. A failed check does not enable a fallback launch. Recovery within the same run must return the exact saved configuration.

The plugin supplies executable paths and the provider workspace in the descriptor. It owns the process environment and account installation. For example, a persistent computer can use these locations:

HOME=/home/agent
AGENT_HOME=/home/agent/.agents
CODEX_HOME=/home/agent/.codex
instructions=/home/agent/.agents/AGENTS.md
skills=/home/agent/.agents/skills/<name>/SKILL.md

These are execution-host locations. Core does not copy the home for each run. The plugin must declare projectAuthCheckVersion: 1 for all native tasks. Core requires an allowed check_project_auth result, or a prepared result with projectAuthCheckAtSubmit: true. The latter delegates the check to the trusted plugin's submit handler before credential installation or task start. The plugin must reject conflicting project or parent authentication settings. It must keep the checked configuration fixed until submission. See launch authentication validation.

Core sends authorized project IDs, the PRP identity, launch limits, and the bootstrap ticket through submit. It does not send a process command. The plugin must submit each task at most once. A lost reply requires status reconciliation for the same task.

Credentials and tools

Core can transfer a selected Codex account or a direct Claude credential. Other managed routes fail closed until they have a typed credential contract. Do not put credentials in task metadata, journals, logs, or errors.

Assigned MCP tools run through the Core gateway over PRP. The gateway credentials stay on Core. The plugin must preserve the MCP binding identity in the prepared context.

Core records credential writeback authority before dispatch. After process termination, Core saves refreshed credentials before it completes the task. A failed save retains the task grants for cleanup.

Stop and recovery

A terminal phase does not prove process termination. Return executionStopped: true only after the task and its descendants stop. Return sessionStateAvailable: true only when the suspended state remains usable. Keep retained state when Core completes the task. Core checks managed plugin workspaces through their storage provider before completion, including after a restart. Core does not inspect these folders locally. If this check fails, Core defers the run and continues recovery for other runs. Core keeps known branch and owner error codes with the repair instructions.

The task contract exposes two checkpoint operations:

  • read_state returns the exact Runner durable state for the supplied identity.
  • archive_state moves a suspended Runner authority checkpoint under the supplied archive key. It returns the same state on retry. It keeps provider session files.

These operations do not accept paths. Core checks the active lease and retained source task before it calls the plugin. The existing Runner transport validates state identity and suspension before it rotates authority.

For controller replacement, Core restores its encrypted PRP journal from Postgres and reuses the original active task lease. The task owner checks running status and exact Runner identity, then reacquires the connection. It does not call prepare, submit, or stop. The running artifact remains pinned even if describe reports a newer installed version. Launch preparation files on the old controller are not required to adopt an already running task. The five-minute outage contract and its prerequisites are described in the continuity recipe.

Core checks company, agent, environment, workspace, driver, and termination receipts before it supplies resumeFrom. Historical controller recovery without the durable journal first stops and releases the old task. A failed stop prevents replacement. A new run prepares its current assignments. Provider compatibility checks remain in the existing backends. Missing or incompatible retained state fails closed.

Setup verification

Core creates a private probe lease with an expiry. It runs the same describe, prepare, submit, and WSS/PRP path. The probe checks provider authentication and the selected model with tools denied. Core bounds Runner lifetime to the remaining probe time. It then requires suspension, confirmed termination, and task cleanup. For a new agent-owned computer, the saved-agent lifecycle runs verification after it saves the agent. Core does not require a computer before this save.

Verification

The Runner fixture uses a synthetic Codex provider and local sockets. It does not qualify a deployed plugin, live account, or production WSS proxy.

Workspace completion and maintenance

A storage provider can declare workspaceProvider.workspaceVerificationVersion: 1. Before creating a fresh workspace for an admitted Retry, Core may send the original task-owned workspace's prepare request with verifyExisting: true. Core sends verification through workspaceOperation with a 10-second deadline and a 12-second RPC timeout. The provider must verify a completed checkout without creating or fetching one and return prepared with created: false. Core checks prior writer settlement and continues normal preparation on the retained workspace after verification. A missing or unconfirmed checkout leaves the fresh retry fallback in place. Local workspaces use the same ownership and writer checks with Core's existing local Git verification.

A storage provider can declare workspaceProvider.finalizationVersion: 1. Core then sends workspaceOperation with operation.kind: "finalize", the retained binding, lifecycle generation, and originating runId and agentId. The operation must verify workspace identity and make the run's retained writes durable before returning { kind: "finalized", headSha, branchName }. Git workspaces must still report the assigned branch. Non-Git workspaces return null Git fields.

Completion permits dirty, untracked, ignored, and unpublished work. It does not prove remote recoverability or authorize deletion. The provider chooses how to meet this contract; Core does not prescribe a VM, filesystem, or Git layout. Core gives completion a 30-second deadline. An unavailable execution host or an unconfirmed durability barrier must fail retryably, never return success. Recovery receives the original run and agent context.

The existing agent-independent inspect and remove operations continue to protect storage maintenance. Providers that do not declare finalization version 1 retain the inspection-based completion path. No stored binding migration is required.

A provider whose inspection cannot prove merge ancestry may declare workspaceProvider.supportsMergeAncestry: false. Omission preserves the existing ancestry inspection behavior. For an opted-out provider, automatic cleanup uses its latest successful finalization HEAD as a hint for destination-aware GitHub checks. It discovers associated PRs without requiring a recorded work product. A direct ancestry proof or a merged PR with the exact final HEAD, intended base repository/branch, and reachable integration commit must precede storage inspection. A fresh matching HEAD and branch, remote recoverability, no local data, terminal tasks, and the existing lifecycle fences still govern deletion. Open or unknown delivery for a known commit retains a bounded remote-only retry. Missing candidate identity parks until a relevant write or startup recovery. Explicit close-readiness and pending lifecycle recovery retain their checks.

A provider may declare workspaceProvider.recoveryPullRequestVersion: 1. Core can then include recoveryPullRequest: { number, headSha } in inspect and remove. This is a narrow recovery hint, not permission to delete. The provider must use its trusted source repository, validate that the specific PR recovery reference still identifies headSha, and independently check the actual workspace and all owned data. Arbitrary remote URLs and reference names are not accepted. Absent capability means Core does not send the extra field. Before dispatch, Core persists the negotiated removal payload and its operation ID. Retries replay both unchanged, even if capabilities change, while still enforcing company, provider enablement, binding and generation checks. A downgraded provider can reject an unsupported payload; Core does not silently change its identity. Legacy intents without a recovery hint retain their deterministic ID. A hinted intent without a recorded ID is ambiguous and fails closed. None of these receipts replaces the provider's checks.

A reservation is a workspace whose prepare never returned a binding. When its source task is done or cancelled, no task points at it, and no run for the task is queued or running, Core archives it and sends remove with binding: null and expectedHeadSha: null instead of reconciling it with an agent-free prepare. The provider must resolve its own record for that execution workspace, discard anything the unfinished preparation created, and return { kind: "removed" }, including when nothing was created. The removal is persisted and replayed with the same operation ID like any other removal. A provider that refuses it fails the removal, which backs off like any other cleanup failure. A reservation that a task still points at, or whose task is open, keeps the prepare reconciliation.

When terminal cleanup inspection or removal fails, Core records metadata.terminalCleanupRetry on the workspace with the failure count, the next attempt time, and the failure kind, reason, and Core's message. Retries back off from 5 minutes, doubling up to 4 hours. A successful inspection or removal clears the record, and a reopen, which raises the lifecycle generation, discards it. Other wakes, including provider upgrades, wait for the next attempt time. Core keeps the record through workspace metadata updates.

The optional prepared.destinationRef records the resolved delivery branch as refs/heads/<branch>, up to 266 characters including the prefix. Providers should return null when preparation cannot establish an unambiguous branch destination or its canonical ref exceeds this receipt limit; an optional receipt must not fail an otherwise successful preparation. Core stores this alongside the source snapshot; callers cannot overwrite it through workspace metadata updates. Older providers omit it. Core only falls back to their captured explicit source refs, not a mutable project configuration or a newly chosen default branch. A source SHA or tag may support direct ancestry, but does not imply a PR base branch. Provider upgrades, activation, company re-enablement, and finalization settlement wake cleanup transactionally. No table, independent timer, or background scan of all remote PR refs is required.

Asynchronous workspace preparation

A provider can implement onWorkspacePrepare alongside onWorkspaceOperation. Core calls the optional method with phase: "submit" and a short submission deadline. A provider that cannot prepare asynchronously returns { kind: "unsupported" } before starting work, so Core can use the original operation. Otherwise the provider returns the completed result or a { kind: "pending", deadline } receipt. The provider sets and enforces that operation deadline. Core holds the workspace lifecycle lock during preparation and rejects offers beyond the previous synchronous RPC budget before polling. It honors accepted deadlines as supplied. Replayed submission must not duplicate an active prepare.

Core sends phase: "poll" with the same operation identity and the deadline from the receipt. The provider returns the same receipt while work is pending and the usual scoped prepared result after success. Core rejects late results and stops polling when the deadline expires. Each submission and poll has a short transport bound independent of the operation deadline. A provider can keep reporting pending while it restores recovery snapshots, under the same deadline.

Each invocation has its own scoped host access. Complete scoped host calls inside the submit or poll that owns them. Detached work can use a remote operation already authorized by that invocation. It cannot reuse the invocation to call host services after the reply.

Workspace operation failures

Core separates a failed workspaceOperation call by what it knows about the outcome:

Failure Run error Core behavior
The worker was not running, so no request was sent Workspace storage provider is not running; retry the same workspace A run that has not started provider work is retried as plugin_worker_unavailable.
The call timed out, or the worker exited with the call pending A fixed timeout or stopped-provider message that says to retry the same workspace The operation may have run, so the same workspace and operation identity must be retried.
The provider threw an error with a known reason The fixed message Core keeps for that reason Only transient reasons tell the user to retry.
The provider threw any other error Workspace storage provider reported a failure; retry the same workspace Unchanged setup failure.

To report a reason, throw PluginWorkspaceOperationError from onWorkspaceOperation or onWorkspacePrepare, or any error with an own workspaceFailureReason property. The reason must come from WORKSPACE_OPERATION_FAILURE_REASONS:

Reason Retrying the same workspace can help
unsupported_operation, configuration_unavailable, credential_unavailable, storage_grant_unavailable, binding_mismatch, runtime_connection_invalid No
storage_pending, runtime_unavailable, remote_incomplete, context_changed, deadline_exceeded Yes
import { PluginWorkspaceOperationError } from "@paperclipai/plugin-sdk";

throw new PluginWorkspaceOperationError("storage_grant_unavailable", "Project mount grants unavailable");

The SDK sends only the reason. Core never reads the provider's message, so a message cannot reach the run, the API, or Core logs. The run keeps its setup_failed error code and records resultJson.workspaceProviderFailure with the failure kind, reason, and retryable flag. The heartbeat execution setup failed log carries the same fields. A provider may also record a plugin.workspace.failure span whose plugin.failure_reason is a reason from this set or unknown.

Warm task retention

A driver declares warmTaskVersion: 1, implements onEnvironmentResumeLease, and returns warmTaskVersion: 1 from describe only when its execution host supports authorized run attachment. Newly hired agents targeting such a driver are configured for warm execution. Existing agents keep their explicit policy; plugin runs without an explicit policy prefer warm execution. There is no provider-specific warm gate or additional setting. Existing runs retain their admitted lifecycle policy during restart recovery. The existing runnerIdleTimeoutMs controls bounded retention. Fresh warm admission requires a Core-managed credential identity before lease acquisition. The saved warm preference does not bypass account ownership checks. Drivers without warm capability and setup probes remain ephemeral.

Plugin task execution uses the public, run-scoped GitHub credential broker when the provider advertises its versioned contract. Warm preparation updates the command capability before admitting the new run. Restart recovery retains the original run binding; new work must not reactivate an older capability. Project authorization is still checked for each new run.

A Launcher task owns one Runner process, independently of Core heartbeat runs. Successful warm run completion retains the task and its mounts. The reuse policy is saved with the selected native runtime, independently of workspace copy-back records. Recovery uses that policy when no workspace-sync record exists, so it does not stop a Runner that foreground completion retains. Failed and cancelled runs still stop instead of staying warm. The reuse fingerprint includes the authorized project set; changing it retires the previous task before checkpoint recovery. Adoption and each warm handoff require a current, configuration-bound project-auth check against the exact retained mount scope. Core atomically hands its retained provider identity to one new run lease, revalidates current project authentication, prepares the new run's resources, and uses PRP run.attach. Preparation includes the exact priorIdentity and stable accountIdentity. The provider journals that transition idempotently before allowing the new authority. The original ingress address remains unchanged. Runner accepts connection: { mode: "external" } on warm attachment to preserve the process owner's listener and TLS settings while rotating the run identity.

Core permits connection refresh for the saved successful run while its warm task lease is retained, unexpired, and not retiring or stopped. PRP authentication still applies. A superseded lease cannot acquire a connection. Retention does not authorize preparation or submission of new work through the completed lease.

Idle retirement suspends and stops Runner, saves its exact checkpoint, writes credentials through the durable lease handoff, then completes the provider task. Core permits read-only suspension proof for the exact saved identity of a successful warm run after its reuse lease is retained or superseded. This does not authorize new work or file access through that lease. Temporary credential homes are not needed after the originating run ends. An explicit reset to a different normalized session can retire a warm owner whose close failed only because it could not prove checkpoint suspension. Core requires the exact lease-bound task's executionStopped: true receipt and a successful complete handoff first. When the close left them unproven, the new run requests them itself: one confirmed stop (not resent once confirmed), the credential snapshot, then complete. A warm owner without a task needs its Runner's own durable suspension instead. Unsettled Core tool results do not block this reset: a governed-wait park leaves one by design, its run is already terminal, its provider is stopped, and the new session never replays it. Core then collects stopped instruction files before it releases the in-memory owner. With that task receipt, any close error qualifies, including a controller journal drain that outlived the close grace: the old journal lives in the discarded session's own state root. The old checkpoint remains evidence; it is not reused by the new session. A handoff that still fails, failed file collection, another close error from an owner without a task, or continuation of the same session still blocks replacement. A controller restart is not required for this stopped-owner reset path.

Deleting an environment (with consent to destroy its reusable sandboxes) closes its idle warm owners first. An idle owner whose close cannot settle is retired the same way: one confirmed stop, the credential snapshot, complete, then instruction collection. An owner whose sandbox lease already carries a destroyed termination receipt has no process left to stop, so Core releases it without contacting the sandbox. Only then are the environment's sandboxes destroyed, through the usual workspace-export and restore-source guards. When an owner still cannot be retired, the delete returns 409 and names each run and the close and stop errors. Idle-timeout, restart and sleep sweeps do not force a stop. They log the exact close error for each owner instead of a count.

When retiring the previous owner stays unproven anyway (any close failure, on any of the paths a new run uses to retire it: instruction preparation, configuration change, session reset, or a prior-owner fence on a resumed sandbox), the new run has done no provider work. It fails with the typed pre-provider error native_predecessor_retirement_unproven instead of a provider failure. The owner stays registered for its own sandbox, so no run on that sandbox launches over it. The heartbeat then:

  • marks this run's reusable lease on that sandbox replaceBeforeReuse, so the next acquisition creates a fresh sandbox and hands the old one to the cleanup sweep. That replacement uses the same rule as a failed resume: a clean, committed workspace save and a verified harness backup. Without them the next acquisition resumes the sandbox as before;
  • after the run ends, hands its prepared workspace on at the saved version only through the shared unchanged-files proof (no runner command, workspace operation such as a provisioning command, execution event or result). Without that proof the workspace stays dirty and is preserved as today, and the run rests with Retry;
  • re-queues the run in the uncharged interrupted_preparation lane instead of a native reconciliation hold. A run that answered a task interaction keeps its own wake in that lane, so the retry still carries the answer.

A run on a different sandbox forgets the unretired owner instead of replaying its rejected close. A previous run parked on a durable governed interaction leaves its question's tool input unanswered by design. Once its Runner's sandbox lease has ended elsewhere, that committed park settles the input for retirement.

Core ends a dispatched task lease with stop, status, and complete. When that sequence confirms the stop and completes the same task, Core does not also call onEnvironmentReleaseLease. Core still calls it when the release did not complete a dispatched task, and the cleanup retry calls it after any failure. When onEnvironmentResumeLease returns no provider lease, Core sends status first. It sends stop only when that status does not report executionStopped: true and any required preparationStopped: true proof. An old warm-start preparation marker does not force a repeated stop when both proofs are present. A driver must release task-scoped resources in its status or complete handling, not only in onEnvironmentReleaseLease.

Termination receipts cover all run leases that owned the retired physical task. A stopped task gets a new provider task ID for checkpoint resumption. Inspection precedes in-memory session selection so a replacement cannot inherit the stopped session. Core saves the fresh provider expiry and forwards and enforces any caller-imposed deadline. A missing or unreachable computer never authorizes a replacement. Controller recovery can adopt a verified surviving idle process through its authenticated PRP connection; ambiguous transitions remain fenced by the existing durable recovery checks.

Changes to account, credential generation, workspace, runtime or process policy retire the old owner before recovery. A new authority epoch archives the stopped checkpoint under its stable archive key and preserves provider session files. Warm status monitoring relies on PRP after startup and does not poll an idle Launcher. ACPX uses this task-provider contract; other execution transports retain their provider-specific lifecycle policies.

Operation-time GitHub credentials

A task provider can advertise githubCredentialBrokerVersion: 1 in its describe result. Core then supplies an optional githubBroker: { url, token } in each prepare operation. This is a runtime capability, not a GitHub token. No controller path, executable, script, or command implementation is part of this contract. Plugin authors own how their command environment uses it.

At the start of each authenticated operation, send POST with body {} to <url without trailing /api>/runtime-tools/github/credentials, with Authorization: Bearer <token>. The response is either { status: "available", env: { ... } } or { status: "unavailable", reason: "..." }. Core rechecks the live run, accepted instruction identity, grants, and trust policy on each request. Use the returned environment only in the command and its descendants. Never retain raw GitHub credentials in plugin state, launch descriptors, logs, or files.

A warm task must replace its broker capability before admitting the next run's commands. An absent broker clears the old capability and allows anonymous/local operations only. Isolate ambient host credentials and Git/GitHub configuration; an unavailable broker must never select a host account. Capture once per operation so steering does not split a command's Git descendants across people. The runtime capability may be staged privately for command-time access; it must not enter the returned launch descriptor or entity records. Core's managed launchers are an optional reference implementation. Compatible native helpers, credential helpers, or command proxies can implement the same semantics.

Deploy the updated Core and plugin together. A configured GitHub connection on an older task provider returns an explicit upgrade error. Unconfigured providers continue ordinary work without authenticated GitHub access.

Managed GitHub command providers must retain the qualified provider executable and arguments. Supply helper discovery through the task environment instead of wrapping that executable. Claude ACPX preserves explicit BASH_ENV, ZDOTDIR, GH_CONFIG_DIR, and PAPERCLIP_GITHUB_LAUNCHER_DIR selectors across the sidecar boundary. It does not forward GitHub tokens or broker capabilities to the provider. The helper reads the current task capability at invocation, including after warm reuse. These selectors must come from the trusted environment owner. Reference helpers mark only their child operation and reject re-entry before resolving another executable or capturing credentials. A PATH shim that calls a helper's absolute path fails with launcher_recursion_rejected (exit 126), preventing recursive process growth. Offline Git fixtures must resolve the native binary outside managed helper paths and exclude runtime shell startup hooks.

The pure launch factory exports RUNNER_GITHUB_LAUNCHER_VERSION = 1 for this contract. A provider must check it as well as the host broker schema and its Launcher capability before advertising managed GitHub commands. The installed Runner must match the factory's source revision.

The helper projection is marked by the controller-owned PAPERCLIP_GITHUB_LAUNCHER_DIR. Task overrides cannot supply that marker or the shell selectors. Run preparation strips stored bindings before secret resolution, and configured task-environment projections reject the shell selectors too.