Files
PaperClipAI/doc/plugins/PLUGIN_RUNNER_EXECUTION.md
T
Nicky LeachandPaperclip 247de89eae fix(runner): preserve warm tasks during recovery (#18169)
## Thinking Path

> - Paperclip manages agent work and the lifetime of its execution
resources.
> - A warm native task can keep one Runner alive between successful
turns.
> - Foreground completion selects whether that resource stays running or
stops.
> - Background recovery must use the same selected policy.
> - Recovery currently reads that policy only from workspace copy-back
records.
> - A task without such a record can be stopped immediately after
foreground retention.
> - This pull request saves the resource policy with the native runtime
and permits the retained task to refresh its control connection.

## Linked Issues or Issue Description

**What happened?**

Background native finalization can stop a successful warm Runner when
its run has no Core workspace-sync record. Foreground completion retains
that same task. The stopped Runner cannot receive the later drain and
suspend commands. Cold resume then correctly rejects its unproven
checkpoint.

Hosted diagnostics confirmed the recovery reconciler attempted
environment release immediately after foreground retention. The provider
connection returned HTTP 404 within the same second. A database-backed
regression reproduced the stop without a workspace-sync record.

A separate authorization gap also prevents the current retained task
from refreshing its connection endpoint after the run succeeds.

**Expected behavior**

Foreground completion and recovery must use the selected resource
policy. A successful warm task stays running until its idle shutdown can
suspend it. Failed or cancelled work still stops. The current retained
task can refresh its existing control route, but cannot prepare or
submit new execution.

**Steps to reproduce**

1. Select a warm native runtime on a reusable remote execution
environment.
2. Complete a turn without a Core workspace-sync record.
3. Let native finalization recovery settle the still-active run lease
before foreground retention finishes.
4. Before this change, recovery defaults to `stop_and_retain` and
invokes the provider stop path.
5. Separately, request the retained task's connection endpoint after
successful completion. Before this change, Core rejects that request.

**Paperclip version/commit**

The hosted failure used Core 31b927e301.
Regressions also reproduced on master at 6d59f5ef7.

**Deployment mode**

A remote native execution environment with reusable warm tasks. The
resource policy is generic and applies across harnesses and providers.

Related recovery coverage: Refs #18161. This change preserves the
suspended-checkpoint admission guard. Existing tasks without the newly
saved policy retain the previous conservative fallback; deployments must
drain or recover them explicitly.

## What Changed

- Save the selected native resource disposition in the existing
runtime-selection transaction. Add no startup database round trip or
network request.
- Read this small policy in background settlement and recovery when no
workspace-sync record exists. Preserve valid workspace-sync policies and
export-retry behavior.
- Keep failed and cancelled runs out of warm retention.
- Permit connection refresh for a successful, dispatched warm task on
its current retained reusable lease. Keep retirement, termination,
expiry, saved-run identity, and plugin capability checks.
- Add database-backed retention and selection regressions,
failure/cancellation coverage, and bounded projection checks. Document
the lifecycle and access contracts.

## Verification

- Before the source edits, the no-workspace-sync regression invoked the
driver with `stop_and_retain` instead of retaining the lease. Both
runtime-selection cases failed because the resource policy was absent.
- After the source edits, those regressions and both
run-state/runtime-selection files passed. The three-file run had 77
passes and two new fixture expectation failures. Those fixtures were
corrected. Final-head GitHub CI passed, including the server suites,
Runner checks, typecheck, build, and E2E gates. The local rerun was
deferred at the user's request.
- Commands: `node_modules/.bin/vitest run
server/src/__tests__/native-environment-settlement-work.test.ts
server/src/services/heartbeat/runtime-selection.test.ts
server/src/services/heartbeat/run-state.test.ts`.
- The connection authorization regression failed before its edit. All 34
tests in `server/src/__tests__/environment-task-runtime.test.ts` then
passed.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- No comprehensive local suite was run. Further local tests and VM
reproductions were stopped at the user's request to conserve battery and
mobile data. GitHub CI verifies the final head.
- Hosted verification of this patch requires deployment after merge.
Passing pre-patch retries did not establish repair of the original
failure.

- Standard Greptile and the subsequent requested Apex review both
returned **5/5** on `360bcd7e789d65950c27484af237faec02553666`, with no
inline findings.

## Risks

- This adds one server-owned field to the existing run profile JSON; no
migration is required. Invalid or absent policy keeps the previous
conservative fallback.
- Valid workspace-sync disposition remains authoritative. Pending export
and failed finalization still use stop-and-retain safety behavior.
- Retained connection access requires a successful run, matching saved
run identity, dispatched warm task, unexpired reusable lease, and no
retirement or termination. Superseded leases and new execution remain
denied.
- The cold-resume checkpoint proof is unchanged. This patch does not
make the already-stopped task's unsuspended checkpoint resumable.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The host does not expose a more specific serving model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run targeted local checks; final verification passed in
GitHub CI (local rerun deferred at the user's request)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 09:19:49 -07:00

238 lines
13 KiB
Markdown

# Plugin Runner execution
Core can run `paperclip_runner` tasks in a plugin environment.
The plugin owns the execution host and its files.
Core owns run identity, selected policy, assigned tools, and the PRP journal.
## Required hooks
Set `supportsTasks: true` on the environment driver.
Declare the `environment.drivers.register` capability.
Implement these hooks:
- `onEnvironmentAcquireLease`
- `onEnvironmentTask`
- `onEnvironmentReleaseLease`
- `onEnvironmentDestroyLease`
Task execution does not require `onEnvironmentExecute` or
`onEnvironmentRealizeWorkspace`. The task API has no shell command or file installation
operation. Use a separate workflow to install agent files. Providers that
advertise `workspaceFileReadVersion: 1` can return task output files through the
bounded `read_file` operation for attachments and deliverables.
Core supports per-turn workspace sessions with Codex, OpenCode, and ACPX.
OpenCode and ACPX require the existing qualified provider profiles.
Workspace-free providers are not supported here. Drivers opt into warm retention through `warmTaskVersion: 1`. Core prefers warm execution for new task runs on capable drivers, including ACPX; existing provider admission still determines which harnesses the driver supports.
The plugin owns project workspace preparation. Core does not prepare or reuse
a Core Git worktree for a task-only environment.
Selecting a plain task directory does not add redundant default configuration.
Retaining that directory does not itself change the run configuration.
For plugin-backed project storage, Core uses the workspace provider contract.
The storage plugin must also provide the selected execution environment.
Its declared workspace modes and strategies remain in effect.
Core passes the pinned workspace binding with each task operation.
Core rejects configured workspace services for task-only environments.
Fresh retries that import retained files are not supported for plugin storage.
An agent-owned workspace cannot move to another agent or environment.
Core also rejects reuse of a local workspace in a task-only environment.
Keep its original owner, or use the workspace storage provider contract.
## Prepare and launch
1. Return an opaque task ID from lease acquisition as `providerLeaseId`.
2. Return the installed Runner version and SHA-256 digest from `describe`.
3. Return `launchPreparationVersion: 1` from `describe`.
4. Handle `prepare` with the [launch preparation contract](../runner-launch-preparation.md).
5. Save the result under its configuration ID.
6. Handle `submit` with that ID in `preparedLaunchId`.
7. Return an authenticated WSS endpoint from `connection`.
Preparation must not start a process. It must preserve Core's provider, model,
policy, instructions, and resource assignment identities. Resolve assigned
resources to execution-host paths. Use `bindRunnerLaunchResources` and
`createRunnerProviderLaunch` when the built-in descriptor builder is suitable.
Core checks the returned requirements digest and descriptor before it registers
PRP or submits the task. A failed check does not enable a fallback launch.
Recovery within the same run must return the exact saved configuration.
The plugin supplies executable paths and the provider workspace in the
descriptor. It owns the process environment and account installation.
For example, a persistent computer can use these locations:
```text
HOME=/home/agent
AGENT_HOME=/home/agent/.agents
CODEX_HOME=/home/agent/.codex
instructions=/home/agent/.agents/AGENTS.md
skills=/home/agent/.agents/skills/<name>/SKILL.md
```
These are execution-host locations. Core does not copy the home for each run.
The plugin must declare `projectAuthCheckVersion: 1` for all native tasks.
Core requires an allowed `check_project_auth` result, or a prepared result with
`projectAuthCheckAtSubmit: true`. The latter delegates the check to the trusted
plugin's submit handler before credential installation or task start. The plugin
must reject conflicting project or parent authentication settings. It must keep
the checked configuration fixed until submission. See
[launch authentication validation](PLUGIN_ENVIRONMENT_TASKS.md#launch-authentication-validation).
Core sends authorized project IDs, the PRP identity, launch limits, and the
bootstrap ticket through `submit`. It does not send a process command.
The plugin must submit each task at most once. A lost reply requires status
reconciliation for the same task.
## Credentials and tools
Core can transfer a selected Codex account or a direct Claude credential.
Other managed routes fail closed until they have a typed credential contract.
Do not put credentials in task metadata, journals, logs, or errors.
Assigned MCP tools run through the Core gateway over PRP.
The gateway credentials stay on Core.
The plugin must preserve the MCP binding identity in the prepared context.
Core records credential writeback authority before dispatch.
After process termination, Core saves refreshed credentials before it completes
the task. A failed save retains the task grants for cleanup.
## Stop and recovery
A terminal phase does not prove process termination.
Return `executionStopped: true` only after the task and its descendants stop.
Return `sessionStateAvailable: true` only when the suspended state remains usable.
Keep retained state when Core completes the task.
Core checks managed plugin workspaces through their storage provider before
completion, including after a restart. Core does not inspect these folders locally.
If this check fails, Core defers the run and continues recovery for other runs.
Core keeps known branch and owner error codes with the repair instructions.
The task contract exposes two checkpoint operations:
- `read_state` returns the exact Runner durable state for the supplied identity.
- `archive_state` moves a suspended Runner authority checkpoint under the supplied
archive key. It returns the same state on retry. It keeps provider session files.
These operations do not accept paths. Core checks the active lease and retained
source task before it calls the plugin. The existing Runner transport validates
state identity and suspension before it rotates authority.
For controller replacement, Core restores its encrypted PRP journal from Postgres
and reuses the original active task lease. The task owner checks running status
and exact Runner identity, then reacquires the connection. It does not call
`prepare`, `submit`, or `stop`. The running artifact remains pinned even if
`describe` reports a newer installed version. Launch preparation files on the
old controller are not required to adopt an already running task.
The five-minute outage contract and its prerequisites are described in the
[continuity recipe](../testing-native-remote-continuity.md#replacing-the-main-server).
Core checks company, agent, environment, workspace, driver, and termination
receipts before it supplies `resumeFrom`. Historical controller recovery without
the durable journal first stops and releases the old task. A failed stop prevents replacement.
A new run prepares its current assignments. Provider compatibility checks remain
in the existing backends. Missing or incompatible retained state fails closed.
## Setup verification
Core creates a private probe lease with an expiry.
It runs the same describe, prepare, submit, and WSS/PRP path.
The probe checks provider authentication and the selected model with tools denied.
Core bounds Runner lifetime to the remaining probe time.
It then requires suspension, confirmed termination, and task cleanup.
For a new agent-owned computer, the saved-agent lifecycle runs verification
after it saves the agent. Core does not require a computer before this save.
## Verification
- [Lease and SDK RPC integration](../../server/src/__tests__/plugin-runner-task.integration.test.ts)
- [Real Runner setup and recovery](../../server/src/services/native-runtime/plugin-runner-probe.test.ts)
- [Task ownership and cancellation](../../server/src/services/native-runtime/plugin-runner-task.test.ts)
- [Credential transfer](../../server/src/services/native-runtime/plugin-runner-credentials.test.ts)
The Runner fixture uses a synthetic Codex provider and local sockets.
It does not qualify a deployed plugin, live account, or production WSS proxy.
## Workspace completion and maintenance
A storage provider can declare `workspaceProvider.finalizationVersion: 1`.
Core then sends `workspaceOperation` with `operation.kind: "finalize"`, the retained
binding, lifecycle generation, and originating `runId` and `agentId`. The operation
must verify workspace identity and make the run's retained writes durable before
returning `{ kind: "finalized", headSha, branchName }`. Git workspaces must still
report the assigned branch. Non-Git workspaces return null Git fields.
Completion permits dirty, untracked, ignored, and unpublished work. It does not
prove remote recoverability or authorize deletion. The provider chooses how to
meet this contract; Core does not prescribe a VM, filesystem, or Git layout.
Core gives completion a 30-second deadline. An unavailable execution host or an
unconfirmed durability barrier must fail retryably, never return success.
Recovery receives the original run and agent context.
The existing agent-independent `inspect` and `remove` operations continue to
protect storage maintenance. Providers that do not declare finalization version 1
retain the inspection-based completion path. No stored binding migration is required.
## Warm task retention
A driver declares `warmTaskVersion: 1`, implements `onEnvironmentResumeLease`,
and returns `warmTaskVersion: 1` from `describe` only when its execution host
supports authorized run attachment. Newly hired agents targeting such a driver
are configured for warm execution. Existing agents keep their explicit policy;
plugin runs without an explicit policy prefer warm execution. There is no
provider-specific warm gate or additional setting. Existing runs retain their
admitted lifecycle policy during restart recovery. The existing
`runnerIdleTimeoutMs` controls bounded retention. Fresh warm admission requires
a Core-managed credential identity before lease acquisition. Plugin-owned installed
accounts continue per-turn until such an identity is available; the saved warm
preference does not bypass account ownership checks. Drivers without warm
capability and setup probes remain ephemeral.
Plugin task execution does not create a Core-managed GitHub broker. Warm reuse
and restart recovery use the same broker-ownership rule as initial startup.
Project authorization is still checked for each new run.
A Launcher task owns one Runner process, independently of Core heartbeat runs.
Successful warm run completion retains the task and its mounts. The reuse
policy is saved with the selected native runtime, independently of workspace
copy-back records. Recovery uses that policy when no workspace-sync record exists,
so it does not stop a Runner that foreground completion retains. Failed and
cancelled runs still stop instead of staying warm. The reuse
fingerprint includes the authorized project set; changing it retires the previous
task before checkpoint recovery. Adoption and each warm handoff require a current,
configuration-bound project-auth check against the exact retained mount scope. Core atomically
hands its retained provider identity to one new run lease, revalidates current
project authentication, prepares the new run's resources, and uses PRP
`run.attach`. Preparation includes the exact `priorIdentity` and stable
`accountIdentity`. The provider journals that transition idempotently before
allowing the new authority. The original ingress address remains unchanged.
Runner accepts `connection: { mode: "external" }` on warm attachment to preserve
the process owner's listener and TLS settings while rotating the run identity.
Core permits connection refresh for the saved successful run while its warm task
lease is retained, unexpired, and not retiring or stopped. PRP authentication
still applies. A superseded lease cannot acquire a connection. Retention does
not authorize preparation or submission of new work through the completed lease.
Idle retirement suspends and stops Runner, saves its exact checkpoint, writes
credentials through the durable lease handoff, then completes the provider task.
Core permits read-only suspension proof for the exact saved identity of a
successful warm run after its reuse lease is retained or superseded. This does
not authorize new work or file access through that lease.
Temporary credential homes are not needed after the originating run ends.
Termination receipts cover all run leases that owned the retired physical task.
A stopped task gets a new provider task ID for checkpoint resumption. Inspection
precedes in-memory session selection so a replacement cannot inherit the stopped
session. Core saves the fresh provider expiry and forwards and enforces any
caller-imposed deadline. A missing
or unreachable computer never authorizes a replacement. Controller recovery can
adopt a verified surviving idle process through its authenticated PRP connection;
ambiguous transitions remain fenced by the existing durable recovery checks.
Changes to account, credential generation, workspace, runtime or process policy
retire the old owner before recovery. A new authority epoch archives the stopped
checkpoint under its stable archive key and preserves provider session files.
Warm status monitoring relies on PRP after startup and does not poll an idle
Launcher. ACPX uses this task-provider contract; other execution transports retain
their provider-specific lifecycle policies.