## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice
## Linked Issues or Issue Description
Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)
**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.
**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.
**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.
**Paperclip version or commit**
master at 377e98f1a2.
**Deployment mode**
Authenticated deployment with a plugin environment that provides a
remote Runner.
## What Changed
- Plugin SDK: add `PluginEnvironmentTaskError` and
`ENVIRONMENT_TASK_ERROR_CODES` with one code,
`runner_revision_mismatch`. The worker RPC host sends it as structured
`environmentTask` error data with the schema
`paperclip/environment-task-error-code/v1`
(`environmentTaskErrorCodeData`, read with
`readEnvironmentTaskErrorCode`). This uses the same envelope method as
the other environment RPC errors. Core reads only the code, never the
message. The source refusal diagnostic from #19302 keeps its own schema
and API unchanged, and it takes precedence on a source `submit`.
- `executeEnvironmentTask` still replaces each worker error with a
generic error. After the #19302 diagnostic check, it now keeps only that
closed code.
- Run setup: a Runner mismatch before provider dispatch gets the error
code `runner_revision_mismatch`. Setup schedules a bounded retry with
its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s
apart). The reason is in the same resource-wait lane as workspace and AI
connection waits, so a wait does not spend the failure retry budget. The
guards are the same as for `plugin_worker_unavailable`. Recovery and
legacy reconciliation treat the code like the other bootstrap-only
codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and
budget for a mismatch run when a caller gives no reason, and the
recovery sweeper reports exhaustion from that budget
(`failedRunRetryBudgetSpent`). An answered interaction that hits a
mismatch also waits for the Runner, and its successor keeps the resolved
interaction continuation identity.
- Move the managed project auth classification from `heartbeat.ts` to
`classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a
found `ai_connection_incompatible` conflict becomes a configuration
failure. All other errors keep their own code.
- Recovery notices: `ai_connection_incompatible` now has its own notice
instead of the "secret/env bindings" text. An exhausted Runner wait gets
the notice "Remote Runner is out of date".
- Agent verification: the Runner probe and the managed project auth
precheck report a mismatch as a retryable check.
`assertHarnessTestPassed` maps it to the new lifecycle code
`runner_revision_mismatch`. That code never counts toward the
three-failure park. The agent checks again every 60 s and becomes ready
when the Runner matches.
- Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`.
## Verification
- `cd packages/plugins/sdk && npx vitest run
tests/environment-task-error-code.test.ts
tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the
code crosses the real `environmentTask` RPC. A plain error with the same
message gets no code. Other methods get no envelope. The #19302
diagnostic tests still pass.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped
mismatch gets `runner_revision_mismatch` and the runner retry budget,
with no configuration failure. A started, untrusted (message only) or
native run keeps `setup_failed` with no retry. A real
`ai_connection_incompatible` stays `configuration_incomplete` with no
retry. Other errors, such as an unreachable execution host, are not
reported as an authentication conflict.
- `npx vitest run src/services/execution-recovery-attempt.test.ts`:
Runner waits do not change `failureRetries`, and earlier failures do not
shorten the wait.
- `npx vitest run src/services/heartbeat/retries.test.ts
src/modules/run-dispatch` (embedded Postgres): a recovery caller without
options gets the Runner wait budget and its exhaustion. An answered
interaction waits for the Runner, keeps its continuation identity and
does not spend failure retries.
- `npx vitest run src/__tests__/environment-task-runtime.test.ts`:
sanitization keeps only the code. No worker text or cause survives.
- `npx vitest run
src/modules/agent-lifecycle/__tests__/verification-retry.test.ts`
(embedded Postgres): a `verifying` agent keeps checking after five
mismatches. It never gets `retryRequired`. It becomes ready when the
next check passes.
- `npx vitest run
src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts
src/__tests__/agent-harness-verification-remote.test.ts
src/services/recovery/stranded-notice.test.ts
src/services/recovery/service.pause-durability.test.ts
src/services/legacy-execution-recovery.test.ts`.
- `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`.
- One test in `src/__tests__/heartbeat-process-recovery.test.ts`
("redacts opaque environment-bound credentials from Sentry diagnostics")
fails on my machine. It fails the same way on the base commit without
this change.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them.
## Risks
- Low to medium. The change adds one new setup failure code and one
lifecycle code. Other failures keep their current behavior.
- A Runner mismatch must come from a plugin that throws
`PluginEnvironmentTaskError`. Until an environment plugin does that,
behavior is the same as today.
- The run retry budget is 10 minutes. If a Runner update takes longer,
the task goes to board recovery with the Runner notice.
- A remote Runner run without a managed AI connection reaches the Runner
check only after native dispatch. That configuration is not supported,
so this change does not handle it. Such a run keeps the native session
retry (3 attempts, 30 s apart).
- #19172 edits the same precheck call in `agent-environment-test.ts`.
Whichever PR merges second needs a small rebase. Keep the mismatch check
before its generic retryable check.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, shell and test runs). Reasoning
mode: standard agentic coding.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
18 KiB
Environment task API
An environment driver can provide a Runner task through a worker hook. The task API has no shell command. Its optional file-read operation is scoped to the active task workspace.
This API is a server and worker contract. The native execution integration connects a task-capable environment to a Runner run.
Declare the hook
Set supportsTasks: true on the environment driver in the plugin manifest.
Implement onEnvironmentTask with the SDK types:
import type {
PluginEnvironmentTaskParams,
PluginEnvironmentTaskResult,
} from "@paperclipai/plugin-sdk";
type TaskHook = (
params: PluginEnvironmentTaskParams,
) => Promise<PluginEnvironmentTaskResult>;
Core requires the environment.drivers.register capability, a ready plugin,
the manifest declaration, and the worker's environmentTask method.
The SDK validates each operation and its result before it returns the result.
See the schemas for all fields.
Operations
| Operation | Result | Required behavior |
|---|---|---|
describe |
description |
Return the installed Runner version and SHA-256 digest. |
prepare |
prepared |
Prepare the provider configuration for the exact Core requirements. Do not start a process. |
submit |
accepted |
Start the same opaque task at most once. Use the structured launch inputs. |
copilot_metadata |
copilot_metadata |
Verify a Copilot account and model on a private setup lease. Send no prompt. |
connection |
connection |
Return an authenticated WSS endpoint and its generation. |
status |
status |
Report the phase and observed process termination. |
check_project_auth |
project_auth |
Validate the prepared project configuration before managed credential transfer. |
read_file |
file |
Read one bounded, task-scoped file from the active task. |
read_state |
state |
Read the exact retained Runner checkpoint. |
archive_state |
state |
Archive suspended authority with an idempotent archive key. Keep provider session files. |
credentials |
credentials |
Return the selected account after process termination. |
stop |
accepted |
Stop the task and its descendants. Fence a delayed submission. |
complete |
accepted |
Release task resources after Core saves required results. |
For a refused submit with resumeFrom, a driver can throw an error with
status: 409, code: "SESSION_STATE_UNAVAILABLE", and reason set to
"unresolved_operation" or "source_state_unverified". The SDK worker and
Core preserve only these closed diagnostic fields. They replace the provider
message and omit provider identities and text. Read the validated RPC envelope
with readEnvironmentTaskErrorDiagnostic.
This diagnostic does not authorize a retry, replacement, cleanup or state change. Other task failures keep the existing generic error. An older SDK does not carry this diagnostic. Operation and call identities, when needed, must come from Core's own retained records.
Each result must contain the task ID from the request. Core rejects a result with a different ID or result kind. A WSS URL must have no user information, query, or fragment. Return authentication in the secret headers.
submit contains the PRP version range, Runner identity, bootstrap ticket,
authorized project IDs, and optional launch limits. It can also contain a saved
task ID for session reuse and a typed Codex or Claude credential. Core does not
supply an executable command or a guest path.
Error codes
Core does not read the text of a task error. To report a failure that Core can
act on, throw PluginEnvironmentTaskError with a code from
ENVIRONMENT_TASK_ERROR_CODES. The SDK sends only the code as structured RPC
error data.
| Code | When to use it | Core behavior |
|---|---|---|
runner_revision_mismatch |
The installed Runner does not match the revision that the plugin launches. Throw it from describe or submit before you prepare or start a task. |
A run that has not started provider work is retried for up to 10 minutes. Agent verification checks again every minute. The task is not blocked during this wait. |
import { PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
if (installed.revision !== expectedRevision) {
throw new PluginEnvironmentTaskError("runner_revision_mismatch");
}
Any other error stays an ordinary setup failure. The source refusal diagnostic above is separate: it uses its own envelope and does not authorize a retry.
Launch preparation
A provider can return launchPreparationVersion: 1 from describe.
Core can then send the prepare operation before submit.
The requirements contain the provider, model, policy, run identity, instructions, and runtime context. They do not contain provider credentials or executable choices. Resource paths identify source locations. The plugin must resolve them for its execution environment.
The result contains a configuration ID, requirements digest, and provider
descriptor. The schema is paperclip.runner.launch-requirements.v1.
The digest is SHA-256 over the requirements JSON. Sort object keys in lexical
order at every level. Keep array order. Omit absent optional fields.
The execution owner must preserve Core policy and resource assignments.
Preparation must be idempotent for the task and requirements. The plugin owns
configuration storage and cleanup. A late preparation result must not authorize
execution after cancellation. The submit operation can reference the saved
configuration with preparedLaunchId.
Recovery can supply the prior prepared result. Recovery within the same run
must use the same configuration. A different run supplies its current
requirements. A saved task ID in resumeFrom requires the same retained-session
authority as submission.
Core requires preparation version 1 for native plugin execution. It validates and saves the result before it submits its configuration ID.
Stopped source reconciliation
A plugin’s describe response advertises codexSourceRecoveryVersion: 1 for
Codex App Server and acpxSourceRecoveryVersion: 1 for ACPX. It can advertise
either, both, or neither. Each field means the host can verify and restore a
stopped source for that provider without starting another turn. Core requires
the field for the selected provider and verifies the run’s eligibility.
The authenticated request belongs to
the current run while prepare and submit carry the predecessor Runner identity.
Core derives this relationship from its saved source-consumer claim; a mismatched
run ID alone never grants recovery. Both operations must name the same resumeFrom, exact source Runner identity, and current
accountIdentity. The request and lease still belong to the current run.
Preparation must save this source binding with its new configuration. Submission
must verify the same binding before it starts the source process.
Core requires the source run to have a committed successful result, an encrypted
PRP journal, confirmed physical termination, and no other source consumer. Core
claims the source under the current native controller lease. This does not change
the source run's status or result. A stale source controller cannot write its
journal after this claim. Workspace export and settlement holds still apply.
After source submission, the same current owner can prepare its attachment with
the exact source as priorIdentity. This does not grant ordinary warm reuse.
The provider must verify the original account, workspace, runtime, and source
ownership before launch. The provider must reject saved active goals, pending tools, ambiguous starts, and
uncertain work before launch. Runner verifies that a saved idle provider remains
idle when restored.
Until the existing authenticated run.attach operation changes authority, the
Core retains semantic inputs and sends no fresh predecessor business commands
while the restored source reconciles its journal and known cleanup. Ordinary suspended recovery and live attachment keep their existing
checks. Providers that omit the capability do not support predecessor recovery.
Launch authentication validation
Declare projectAuthCheckVersion: 1 in describe for each native task launch.
Core calls check_project_auth after preparation and before credential transfer.
The request names the configuration ID, provider, and authorized project IDs.
It contains no credential.
A trusted provider can instead return projectAuthCheckAtSubmit: true in its
prepared result. Core then skips the separate check. This flag delegates the
check to submission; it does not claim that preparation checked authentication.
The provider must validate the effective configuration and current project grants
inside submit, before it installs credentials on the execution host or starts
the task. It must reject conflicts and changed preparation before those effects.
Core pins the configuration, provider, and project set on the lease and checks
them again at dispatch. A new preparation invalidates that scope. Providers that
omit the flag keep the separate check. The flag requires a Core version that
supports it; older strict result schemas reject it.
The plugin must check the effective configuration for that prepared launch. Reject project settings, parent settings, helper commands, or environment routing that can replace the selected account or redirect its credential. Keep the checked configuration fixed until submission. If the configuration can change, reject submission and require new preparation and validation.
Return project_auth with the same configuration ID and provider.
Set allowed: true only after the checks pass.
Core saves this result on the task lease. Core requires the exact configuration,
provider, and project set before each submission. This also applies when the
provider uses an installed account.
A new preparation or check removes the prior validation. A failed check does not
authorize credential transfer. The dispatch update also checks that validation
has not changed while Core waited for database work.
Retained checkpoint operations
read_state supplies the exact durable identity and an optional resumeFrom.
archive_state supplies the prior identity, a 24-character lowercase hexadecimal
archive key, and resumeFrom. Both return state with the task ID and Runner
state object. They do not accept a filesystem path.
Core authorizes retained sources with the same checks as preparation and launch. The plugin must read only that task's state. Archive must require process termination and exact suspended identity. It must return the same archived state on retry. It must keep the provider session files. Return an error when the state is absent or its identity differs. Do not return credentials or environment values.
Lease authority
Core calls environmentRuntimeService.task with a company ID, a saved lease ID,
and an operation. It derives the plugin and driver from the lease. A change to
the environment configuration cannot redirect cleanup to a new provider.
Before preparation or submission, Core checks the active lease, run, issue, and every project
in the company. It checks the submitting run ID separately from a retained PRP
identity. Session reuse requires the same agent, environment, workspace, and
provider. Checkpoint resumption also requires a retained source task and proof of termination.
Opt-in Codex warm reuse instead hands off the same retained process through an
atomic lease transition and an exact, authorized priorIdentity preparation.
An internal setup probe can use a private, time-limited lease without a run row. Core checks the saved probe identity, rejects project mounts, and limits the Runner lifetime to the remaining probe time.
Failure and cleanup
A failed submission reply does not prove that the task failed to start. Core saves the dispatch marker before it calls the worker. Retry against the same task ID. Do not allocate another task to resolve an uncertain reply.
An accepted stop does not prove termination. Set executionStopped: true
only after the Runner and all descendants have stopped. A terminal phase alone
does not permit Core to release execution authority.
Core binds each Runner owner to the physical task ID returned by its description. An old owner cannot follow a changed lease to stop or complete its replacement. Late stopped-state, retained-session and credential-handoff writes must still match that same physical task. A new owner can clean up the replacement.
Core fences new submissions when cleanup starts. It stops the saved task, checks termination, saves credential results, and completes the task before it releases or destroys the environment lease. An uncertain result leaves a saved lease for the cleanup sweep. Cleanup can use that lease after the environment or run row is deleted.
Credential writeback
Account selection and credential revision have separate identities. Keep the
selected account fixed for the task. A credential result of null means that
cancellation prevented task admission. It must not mean that an export failed.
Core saves the account grant, secret reference, and session epoch on the lease. It saves no credential bytes or temporary path in lease metadata. After process termination, it can recover a Codex refresh even if Core restarted. It uses the existing account comparison and secret service. A revoked grant, replaced account, or changed session epoch discards the old refresh.
Keep file synchronization and backup outside this API. The provider must retain recoverable credential state until task completion succeeds.
Native environment test
The agent environment test can use a task-capable plugin environment. Core creates a short-lived probe lease with a private run and session identity. The probe has no project mounts or task tools. It prepares the launch, checks authentication routing, and verifies the Runner and provider hello exchange. A passing result requires confirmed cleanup. This test does not enable heartbeat dispatch to plugin environments.
Copilot account test
copilot_metadata accepts the selected token and an optional model ID. Core
permits it only on an active private setup lease. The provider must use the
verified Copilot metadata client. It must send no prompt and use no project
workspace or tool server. Return the metadata and executionStopped: true only
after the probe and its descendants stop. A lost reply keeps the task available
for stop, status, and complete. Core checks the profile and model before
it accepts the account. Do not store the token in task metadata or logs.
For a native hello probe, select a saved AI account or use the installed account. Core rejects local environment overrides when no saved account is selected. This includes draft credentials. The probe uses the installed VM environment. Core does not test a different installed account in their place. Only Core can supply the selected account identity and local credential home.
Fresh attempts
A workspace provider opts in with workspaceProvider.freshWorkspaceVersion: 1.
A task environment also declares environmentDrivers[].freshWorkspaceVersion: 1
and returns freshWorkspaceVersion: 1 from describe.
Core supplies a server-authorized fresh intent on workspace preparation and on
launch-related task calls. attemptId is stable across interrupted preparation.
previousWorkspaceId identifies retained storage; it is not permission to change
or remove it. The provider prepares a new working directory from the authorized
source and base. It may share repository objects or other immutable storage.
Core does not require a particular Git or filesystem layout.
recoverySnapshots contains verified, Core-staged archive and manifest files.
Each file descriptor contains a trusted host path, SHA-256, and byte size. These
paths are host-authored transfer sources, never user-supplied paths or URLs.
Providers verify the bytes and safely materialize each snapshot separately at
.paperclip-recovered/<operationId> beneath the new working directory. Preserve
file links safely and include provenance. Never overlay recovered files onto the
clean working tree or overwrite an earlier attachment. Core includes those
relative attachment paths in the new attempt's continuation context.
A provider reconciles the same attempt and workspace identity after an ambiguous reply. It must not replace the failed workspace or allocate another workspace when preparation is repeated. Providers without this version remain unavailable for fresh attempts; ordinary preparation and reuse retain their existing contract.
Core keeps transfer files while preparation is uncertain. Confirmed task completion and provider workspace removal release these disposable copies. The descriptors remain stable: a later preparation downloads the same verified files again.
Task workspace files
Core uses read_file for call_api file uploads and register_deliverable
when a plugin owns execution and advertises workspaceFileReadVersion: 1 in
describe. The request includes the submitted Runner identity,
validated project IDs, a canonical relative or absolute path, and a byte limit (at most 10 MiB).
The provider resolves paths within locations granted to the active task. Relative
paths resolve from its prepared workspace. Absolute paths can select another
granted location such as agent home, but never a supervisor path or another
agent's files. Core does not prescribe provider filesystem layout.
The provider must reject inactive tasks, path escapes, symlinks, hard links,
non-regular files, protected credentials, oversized files, and files that change
while being read. Return canonical base64, byte size, and a SHA-256 digest in a
file result for the same task. Core validates the result and rechecks authority
before publishing it. Deliverables additionally verify the caller's expected
size and digest. Providers without this operation fail closed; Core never falls
back to reading remote paths on the server. Existing artifact uploads are unchanged.