From a6ab6a301cba75c220e573b805d0abf1055a9a97 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 5 Nov 2026 14:34:23 -0800 Subject: [PATCH] fix(server): retry a run when its remote Runner is out of date (#19326) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can run on a remote Runner that an environment plugin provides > - A release can update the app before the remote Runner, so for a short time the app sees an older Runner > - In that window the plugin can refuse the task, but Core cannot tell this refusal from other errors > - Today the run fails as `setup_failed` with no retry, and the task goes to board recovery. Before #19246, it was also reported as a project authentication conflict > - Agent verification also counts the refusal as a failed harness test, and it parks the agent after three tries > - This pull request gives the plugin a stable error code for a Runner mismatch, and Core waits and retries when it gets that code > - The benefit is that a release no longer blocks tasks or parks agents, and a real authentication conflict keeps its own clear notice ## Linked Issues or Issue Description Refs #19246 (it stopped reporting every precheck error as a project authentication conflict) Refs #19249 (the same bounded bootstrap retry for a stopped plugin worker) Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope is unchanged) Refs #19172 (related open PR: it reports an undescribable plugin Runner as a retryable check in the same precheck) Refs #18667 (a real `ai_connection_incompatible` conflict, which now gets its own notice) **What happened?** A run started while its remote Runner was older than the app. The environment plugin refused the task because the installed Runner did not match the revision it expects. Core recorded a generic setup failure, scheduled no retry, and blocked the task. Before #19246, Core reported this as a project authentication conflict. An agent in `verifying` counted the same refusal as a failed harness test. After three tries it stopped verification until a person chose Retry. **Expected behavior** Core treats a Runner mismatch as temporary. A run retries without blocking its task. A `verifying` agent checks again until the Runner matches. Only a real `ai_connection_incompatible` result blocks the task, with a notice that names the conflict. **Steps to reproduce** 1. Use an agent on a plugin environment whose installed remote Runner revision differs from the revision the plugin expects. 2. Wake the agent on a task, or put a new agent through verification. 3. The run fails with `setup_failed` and the task is blocked. Verification parks after three tries. **Paperclip version or commit** master at 377e98f1a2. **Deployment mode** Authenticated deployment with a plugin environment that provides a remote Runner. ## What Changed - Plugin SDK: add `PluginEnvironmentTaskError` and `ENVIRONMENT_TASK_ERROR_CODES` with one code, `runner_revision_mismatch`. The worker RPC host sends it as structured `environmentTask` error data with the schema `paperclip/environment-task-error-code/v1` (`environmentTaskErrorCodeData`, read with `readEnvironmentTaskErrorCode`). This uses the same envelope method as the other environment RPC errors. Core reads only the code, never the message. The source refusal diagnostic from #19302 keeps its own schema and API unchanged, and it takes precedence on a source `submit`. - `executeEnvironmentTask` still replaces each worker error with a generic error. After the #19302 diagnostic check, it now keeps only that closed code. - Run setup: a Runner mismatch before provider dispatch gets the error code `runner_revision_mismatch`. Setup schedules a bounded retry with its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s apart). The reason is in the same resource-wait lane as workspace and AI connection waits, so a wait does not spend the failure retry budget. The guards are the same as for `plugin_worker_unavailable`. Recovery and legacy reconciliation treat the code like the other bootstrap-only codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and budget for a mismatch run when a caller gives no reason, and the recovery sweeper reports exhaustion from that budget (`failedRunRetryBudgetSpent`). An answered interaction that hits a mismatch also waits for the Runner, and its successor keeps the resolved interaction continuation identity. - Move the managed project auth classification from `heartbeat.ts` to `classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a found `ai_connection_incompatible` conflict becomes a configuration failure. All other errors keep their own code. - Recovery notices: `ai_connection_incompatible` now has its own notice instead of the "secret/env bindings" text. An exhausted Runner wait gets the notice "Remote Runner is out of date". - Agent verification: the Runner probe and the managed project auth precheck report a mismatch as a retryable check. `assertHarnessTestPassed` maps it to the new lifecycle code `runner_revision_mismatch`. That code never counts toward the three-failure park. The agent checks again every 60 s and becomes ready when the Runner matches. - Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. ## Verification - `cd packages/plugins/sdk && npx vitest run tests/environment-task-error-code.test.ts tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the code crosses the real `environmentTask` RPC. A plain error with the same message gets no code. Other methods get no envelope. The #19302 diagnostic tests still pass. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped mismatch gets `runner_revision_mismatch` and the runner retry budget, with no configuration failure. A started, untrusted (message only) or native run keeps `setup_failed` with no retry. A real `ai_connection_incompatible` stays `configuration_incomplete` with no retry. Other errors, such as an unreachable execution host, are not reported as an authentication conflict. - `npx vitest run src/services/execution-recovery-attempt.test.ts`: Runner waits do not change `failureRetries`, and earlier failures do not shorten the wait. - `npx vitest run src/services/heartbeat/retries.test.ts src/modules/run-dispatch` (embedded Postgres): a recovery caller without options gets the Runner wait budget and its exhaustion. An answered interaction waits for the Runner, keeps its continuation identity and does not spend failure retries. - `npx vitest run src/__tests__/environment-task-runtime.test.ts`: sanitization keeps only the code. No worker text or cause survives. - `npx vitest run src/modules/agent-lifecycle/__tests__/verification-retry.test.ts` (embedded Postgres): a `verifying` agent keeps checking after five mismatches. It never gets `retryRequired`. It becomes ready when the next check passes. - `npx vitest run src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts src/__tests__/agent-harness-verification-remote.test.ts src/services/recovery/stranded-notice.test.ts src/services/recovery/service.pause-durability.test.ts src/services/legacy-execution-recovery.test.ts`. - `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`. - One test in `src/__tests__/heartbeat-process-recovery.test.ts` ("redacts opaque environment-bound credentials from Sentry diagnostics") fails on my machine. It fails the same way on the base commit without this change. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - Low to medium. The change adds one new setup failure code and one lifecycle code. Other failures keep their current behavior. - A Runner mismatch must come from a plugin that throws `PluginEnvironmentTaskError`. Until an environment plugin does that, behavior is the same as today. - The run retry budget is 10 minutes. If a Runner update takes longer, the task goes to board recovery with the Runner notice. - A remote Runner run without a managed AI connection reaches the Runner check only after native dispatch. That configuration is not supported, so this change does not handle it. Such a run keeps the native session retry (3 attempts, 30 s apart). - #19172 edits the same precheck call in `agent-environment-test.ts`. Whichever PR merges second needs a small rebase. Keep the mismatch check before its generic retryable check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (code search, shell and test runs). Reasoning mode: standard agentic coding. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md | 21 ++++++ .../plugins/sdk/src/environment-task-error.ts | 45 ++++++++++++ packages/plugins/sdk/src/index.ts | 2 + packages/plugins/sdk/src/worker-rpc-host.ts | 5 +- .../tests/environment-task-error-code.test.ts | 69 +++++++++++++++++++ .../agent-harness-verification-remote.test.ts | 13 ++++ .../environment-task-runtime.test.ts | 15 +++- .../__tests__/verification-retry.test.ts | 19 +++++ .../agent-lifecycle/application/failure.ts | 1 + .../run-dispatch/domain/wake-context.test.ts | 7 ++ .../run-dispatch/domain/wake-context.ts | 2 + server/src/modules/run-dispatch/index.ts | 1 + server/src/services/agent-environment-test.ts | 7 +- .../services/agent-harness-verification.ts | 4 +- .../src/services/environment-task-runtime.ts | 5 +- .../execution-recovery-attempt.test.ts | 28 +++++++- .../services/execution-recovery-attempt.ts | 9 ++- server/src/services/heartbeat.ts | 15 ++-- server/src/services/heartbeat/retries.test.ts | 35 +++++++++- server/src/services/heartbeat/retries.ts | 21 ++++-- .../services/heartbeat/run-completion.test.ts | 60 +++++++++++++++- .../src/services/heartbeat/run-completion.ts | 8 ++- .../src/services/heartbeat/run-preparation.ts | 8 +++ .../legacy-execution-recovery.test.ts | 2 +- .../src/services/legacy-execution-recovery.ts | 4 +- ...lugin-runner-probe-runner-mismatch.test.ts | 26 +++++++ .../native-runtime/plugin-runner-probe.ts | 2 + .../recovery/service.pause-durability.test.ts | 2 +- server/src/services/recovery/service.ts | 2 + .../services/recovery/stranded-notice.test.ts | 22 ++++++ .../src/services/recovery/stranded-notice.ts | 22 +++++- .../src/services/runner-revision-mismatch.ts | 19 +++++ 32 files changed, 468 insertions(+), 33 deletions(-) create mode 100644 packages/plugins/sdk/tests/environment-task-error-code.test.ts create mode 100644 server/src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts create mode 100644 server/src/services/runner-revision-mismatch.ts diff --git a/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md b/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md index 557f3a3c46a..18a5fdd8882 100644 --- a/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md +++ b/doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md @@ -67,6 +67,27 @@ authorized project IDs, and optional launch limits. It can also contain a saved task ID for session reuse and a typed Codex or Claude credential. Core does not supply an executable command or a guest path. +## Error codes + +Core does not read the text of a task error. To report a failure that Core can +act on, throw `PluginEnvironmentTaskError` with a code from +`ENVIRONMENT_TASK_ERROR_CODES`. The SDK sends only the code as structured RPC +error data. + +| Code | When to use it | Core behavior | +| --- | --- | --- | +| `runner_revision_mismatch` | The installed Runner does not match the revision that the plugin launches. Throw it from `describe` or `submit` before you prepare or start a task. | A run that has not started provider work is retried for up to 10 minutes. Agent verification checks again every minute. The task is not blocked during this wait. | + +```ts +import { PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk"; + +if (installed.revision !== expectedRevision) { + throw new PluginEnvironmentTaskError("runner_revision_mismatch"); +} +``` + +Any other error stays an ordinary setup failure. The source refusal diagnostic above is separate: it uses its own envelope and does not authorize a retry. + ## Launch preparation A provider can return `launchPreparationVersion: 1` from `describe`. diff --git a/packages/plugins/sdk/src/environment-task-error.ts b/packages/plugins/sdk/src/environment-task-error.ts index 21455ef9ec1..98aae8a0674 100644 --- a/packages/plugins/sdk/src/environment-task-error.ts +++ b/packages/plugins/sdk/src/environment-task-error.ts @@ -33,3 +33,48 @@ export function readEnvironmentTaskErrorDiagnostic(error: unknown): PluginEnviro const diagnostic = field(data, "diagnostic"); return sanitize(field(diagnostic, "httpStatus"), field(diagnostic, "errorCode"), field(diagnostic, "reason")); } + +export const ENVIRONMENT_TASK_ERROR_CODES = ["runner_revision_mismatch"] as const; +export type PluginEnvironmentTaskErrorCode = typeof ENVIRONMENT_TASK_ERROR_CODES[number]; + +const CODE_SCHEMA = "paperclip/environment-task-error-code/v1"; + +function ownField(value: unknown, name: string): unknown { + if (!value || typeof value !== "object") return undefined; + try { return Object.getOwnPropertyDescriptor(value, name)?.value; } + catch { return undefined; } +} + +function validateCode(value: unknown): PluginEnvironmentTaskErrorCode | null { + return ENVIRONMENT_TASK_ERROR_CODES.includes(value as PluginEnvironmentTaskErrorCode) + ? value as PluginEnvironmentTaskErrorCode : null; +} + +export class PluginEnvironmentTaskError extends Error { + readonly errorCode: PluginEnvironmentTaskErrorCode; + + constructor(errorCode: PluginEnvironmentTaskErrorCode) { + const safe = validateCode(errorCode); + if (!safe) throw new TypeError("Unknown environment task error code"); + super("The installed Runner does not match the expected revision"); + this.name = "PluginEnvironmentTaskError"; + this.errorCode = safe; + } +} + +export function environmentTaskErrorCodeData(error: unknown): { schema: string; errorCode: PluginEnvironmentTaskErrorCode } | undefined { + if (!(error instanceof PluginEnvironmentTaskError)) return undefined; + const errorCode = validateCode(ownField(error, "errorCode")); + return errorCode ? { schema: CODE_SCHEMA, errorCode } : undefined; +} + +export function readEnvironmentTaskErrorCode(error: unknown): PluginEnvironmentTaskErrorCode | null { + for (let current = error, depth = 0; current && depth < 5; current = ownField(current, "cause"), depth++) { + const data = ownField(current, "data"); + if (ownField(data, "schema") === CODE_SCHEMA) { + const code = validateCode(ownField(data, "errorCode")); + if (code) return code; + } + } + return null; +} diff --git a/packages/plugins/sdk/src/index.ts b/packages/plugins/sdk/src/index.ts index 221cdecf043..d8399427968 100644 --- a/packages/plugins/sdk/src/index.ts +++ b/packages/plugins/sdk/src/index.ts @@ -464,6 +464,8 @@ export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readE export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js"; export { PluginEnvironmentRunnerIngressError, environmentRunnerIngressErrorData, readEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js"; export type { PluginEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js"; +export { ENVIRONMENT_TASK_ERROR_CODES, PluginEnvironmentTaskError, environmentTaskErrorCodeData, readEnvironmentTaskErrorCode } from "./environment-task-error.js"; +export type { PluginEnvironmentTaskErrorCode } from "./environment-task-error.js"; export { PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "./environment-stop-error.js"; export type { PluginEnvironmentStopFailure, PluginEnvironmentStopFailureReason } from "./environment-stop-error.js"; export { environmentResumeErrorData, readEnvironmentResumeErrorDiagnostic } from "./environment-resume-error.js"; diff --git a/packages/plugins/sdk/src/worker-rpc-host.ts b/packages/plugins/sdk/src/worker-rpc-host.ts index 7c34deaf807..706fe9da497 100644 --- a/packages/plugins/sdk/src/worker-rpc-host.ts +++ b/packages/plugins/sdk/src/worker-rpc-host.ts @@ -9,7 +9,7 @@ import { environmentSyncErrorData, withEnvironmentSyncErrorCapture } from "./env import { environmentRunnerIngressErrorData } from "./environment-runner-ingress-error.js"; import { environmentStopErrorData } from "./environment-stop-error.js"; import { environmentResumeErrorData } from "./environment-resume-error.js"; -import { environmentTaskErrorData } from "./environment-task-error.js"; +import { environmentTaskErrorCodeData, environmentTaskErrorData } from "./environment-task-error.js"; import { createInvocationTracer } from "./invocation-tracer.js"; import { createPluginIdleDrain } from "./idle-drain.js"; /** @@ -1654,7 +1654,8 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost : method === "environmentSyncOut" ? environmentSyncErrorData(err) : method === "environmentRunnerIngressEndpoint" ? environmentRunnerIngressErrorData(err) : method === "environmentStopLease" ? environmentStopErrorData(err) - : method === "environmentResumeLease" ? environmentResumeErrorData(err) : taskError)); + : method === "environmentResumeLease" ? environmentResumeErrorData(err) + : method === "environmentTask" ? taskError ?? environmentTaskErrorCodeData(err) : undefined)); } finally { done?.(); } diff --git a/packages/plugins/sdk/tests/environment-task-error-code.test.ts b/packages/plugins/sdk/tests/environment-task-error-code.test.ts new file mode 100644 index 00000000000..fc6a4cbeb4b --- /dev/null +++ b/packages/plugins/sdk/tests/environment-task-error-code.test.ts @@ -0,0 +1,69 @@ +import { createInterface } from "node:readline"; +import { PassThrough } from "node:stream"; +import { describe, expect, it } from "vitest"; +import { definePlugin } from "../src/define-plugin.js"; +import { environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode } from "../src/environment-task-error.js"; +import { createRequest, isJsonRpcResponse, JsonRpcCallError, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js"; +import { startWorkerRpcHost } from "../src/worker-rpc-host.js"; + +async function invoke(method: string, error: unknown) { + const stdin = new PassThrough(); + const stdout = new PassThrough(); + const lines = createInterface({ input: stdout }); + const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({ + async setup() {}, + async onEnvironmentTask() { throw error; }, + async onEnvironmentProbe() { throw error; }, + }) }); + try { + const response = new Promise((resolve) => { + lines.on("line", (line) => { + const message = parseMessage(line); + if (isJsonRpcResponse(message)) resolve(message); + }); + }); + stdin.write(serializeMessage(createRequest(method, { taskId: "task-1", operation: { kind: "describe" } }, "task-test"))); + return await response; + } finally { + worker.stop(); lines.close(); stdin.destroy(); stdout.destroy(); + } +} + +describe("environment task error codes", () => { + it("does not read the source refusal diagnostic envelope as a code", () => { + expect(readEnvironmentTaskErrorCode({ data: { schema: "paperclip/environment-task-error/v1", errorCode: "runner_revision_mismatch" } })).toBeNull(); + }); + it("carries runner_revision_mismatch across the real environmentTask RPC", async () => { + const response = await invoke("environmentTask", new PluginEnvironmentTaskError("runner_revision_mismatch")); + if (!("error" in response) || !response.error) throw new Error("Expected RPC error"); + expect(response.error.data).toEqual({ schema: "paperclip/environment-task-error-code/v1", errorCode: "runner_revision_mismatch" }); + const callError = new JsonRpcCallError(response.error); + expect(callError.message).toBe("The installed Runner does not match the expected revision"); + expect(readEnvironmentTaskErrorCode(callError)).toBe("runner_revision_mismatch"); + expect(readEnvironmentTaskErrorCode(new Error("wrapped", { cause: new Error("again", { cause: callError }) }))) + .toBe("runner_revision_mismatch"); + }); + + it("never derives a code from message text", async () => { + const response = await invoke("environmentTask", new Error("The installed Runner does not match the expected revision")); + expect("error" in response && response.error).not.toHaveProperty("data"); + if (!("error" in response) || !response.error) throw new Error("Expected RPC error"); + expect(readEnvironmentTaskErrorCode(new JsonRpcCallError(response.error))).toBeNull(); + }); + + it("only attaches the envelope to environmentTask calls", async () => { + const response = await invoke("environmentProbe", new PluginEnvironmentTaskError("runner_revision_mismatch")); + expect("error" in response && response.error).not.toHaveProperty("data"); + }); + + it("refuses codes outside the closed set", () => { + expect(() => new PluginEnvironmentTaskError("private" as never)).toThrow(TypeError); + expect(environmentTaskErrorCodeData(Object.assign(new Error("forged"), { errorCode: "runner_revision_mismatch" }))).toBeUndefined(); + }); + + it.each([null, {}, { data: { schema: "other", errorCode: "runner_revision_mismatch" } }, + { data: { schema: "paperclip/environment-task-error-code/v1", errorCode: "private" } }])( + "rejects malformed envelopes %j", (value) => { + expect(readEnvironmentTaskErrorCode(value)).toBeNull(); + }); +}); diff --git a/server/src/__tests__/agent-harness-verification-remote.test.ts b/server/src/__tests__/agent-harness-verification-remote.test.ts index db5c5c01ed1..3e6ecfcdbfa 100644 --- a/server/src/__tests__/agent-harness-verification-remote.test.ts +++ b/server/src/__tests__/agent-harness-verification-remote.test.ts @@ -65,3 +65,16 @@ describe("remote harness verification recovery", () => { expect(() => assertHarnessTestPassed(recovered)).not.toThrow(); }); }); + +describe("Runner mismatch during verification", () => { + const context = { stage: "host" as const, phase: "verifying" as const, verificationFailures: 100 }; + it("waits for the Runner update instead of parking verification", async () => { + const { runnerRevisionMismatchTestResult } = await import("../services/runner-revision-mismatch.js"); + let error: unknown; + try { assertHarnessTestPassed(runnerRevisionMismatchTestResult("paperclip_runner")); } catch (caught) { error = caught; } + expect(error).toMatchObject({ code: "runner_revision_mismatch" }); + expect(lifecycleVerificationNeedsRetry(context, error)).toBe(false); + expect(lifecycleFailureMessage(context, error)).toContain("will check again automatically"); + expect(lifecycleFailureMessage(context, error)).not.toContain("Automatic verification is paused"); + }); +}); diff --git a/server/src/__tests__/environment-task-runtime.test.ts b/server/src/__tests__/environment-task-runtime.test.ts index 38d455c57d4..0c3a9181193 100644 --- a/server/src/__tests__/environment-task-runtime.test.ts +++ b/server/src/__tests__/environment-task-runtime.test.ts @@ -1,7 +1,8 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { PgDialect } from "drizzle-orm/pg-core"; import { executionWorkspaces, environmentLeases, type Db } from "@paperclipai/db"; -import { environmentTaskOperationSchema, parseEnvironmentTaskResult, JsonRpcCallError, readEnvironmentTaskErrorDiagnostic } from "@paperclipai/plugin-sdk"; +import { environmentTaskErrorCodeData, environmentTaskOperationSchema, parseEnvironmentTaskResult, JsonRpcCallError, PluginEnvironmentTaskError, + readEnvironmentTaskErrorCode, readEnvironmentTaskErrorDiagnostic } from "@paperclipai/plugin-sdk"; import { executeEnvironmentTask, settleDispatchedEnvironmentTask } from "../services/environment-task-runtime.js"; import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js"; @@ -512,6 +513,18 @@ describe("plugin-provided Runner execution", () => { expect(error.message).toBe("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"); expect(readEnvironmentTaskErrorDiagnostic(error)).toBeNull(); }); + it("keeps only the plugin's stable task error code through worker error sanitization", async () => { + const workers = worker(); + workers.call.mockRejectedValue(Object.assign(new JsonRpcCallError({ code: -32002, message: "private-credential", + data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) }), { extra: "private-credential" })); + const error = await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit }).catch(caught => caught); + expect(error.message).toBe("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"); + expect(readEnvironmentTaskErrorCode(error)).toBe("runner_revision_mismatch"); + expect(JSON.stringify({ ...error, message: error.message })).not.toContain("private-credential"); + expect(error.cause).toBeUndefined(); + workers.call.mockRejectedValue(new Error("The installed Runner does not match the expected revision")); + expect(readEnvironmentTaskErrorCode(await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit }).catch(caught => caught))).toBeNull(); + }); it("accepts PRP identity characters and length limits", () => { for (const field of ["runnerId", "leaseId", "runId", "sessionId", "turnId", "itemId"]) { for (const value of ["session:part.1", "a".repeat(160)]) { diff --git a/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts b/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts index 93cc524613d..5ff5f3f4faa 100644 --- a/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts @@ -133,6 +133,25 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu expect((await current(agent.id)).lifecycleOperation).toMatchObject({ verificationFailures: 3, retryRequired: true }); }); + it("keeps checking an outdated remote Runner past the failure budget and becomes ready once it matches", async () => { + const { agent, work, runHost } = await failed("runner_revision_mismatch"); + for (let check = 0; check < 5; check++) { + const waiting = await current(agent.id); + expect(waiting).toMatchObject({ lifecycleState: "verifying", status: "paused" }); + expect(waiting.lifecycleOperation?.retryRequired).toBeUndefined(); + expect(waiting.lifecycleOperation?.verificationFailures).toBeUndefined(); + expect(waiting.lifecycleOperation?.retryAt).toEqual(expect.any(String)); + expect(waiting.lifecycleError).toContain("Runner is being updated"); + expect(waiting.lifecycleError).not.toContain("Automatic verification is paused"); + await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id)); + await work.process(agent.id); + } + runHost.mockImplementation(async () => "complete"); + await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id)); + await work.process(agent.id); + expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle", lifecycleError: null }); + }); + it("recovers from a generic failed test before the automatic retry budget is exhausted", async () => { const { agent, work, runHost } = await failed("harness_probe_timed_out"); runHost.mockRejectedValueOnce(Object.assign(new Error("temporary test failure"), { code: "harness_test_failed" })); diff --git a/server/src/modules/agent-lifecycle/application/failure.ts b/server/src/modules/agent-lifecycle/application/failure.ts index 55b6e5dc7c6..897d7440b9e 100644 --- a/server/src/modules/agent-lifecycle/application/failure.ts +++ b/server/src/modules/agent-lifecycle/application/failure.ts @@ -9,6 +9,7 @@ const reasons = { harness_cwd_invalid: "The working directory is unavailable. Check the working directory in Configuration, then retry.", harness_probe_timed_out: "The execution test timed out. Paperclip will retry automatically. If this continues, run the agent test in Configuration.", harness_probe_unexpected_output: "The execution test did not return the expected response. Run the agent test in Configuration to inspect the failed checks.", + runner_revision_mismatch: "The remote Runner is being updated to match this Paperclip version. Paperclip will check again automatically.", test_environment_unavailable: "The test environment is unavailable. Check the agent's environment in Configuration. Paperclip will retry automatically.", required_plugin_unavailable: "A required plugin is unavailable. Check the plugin's status in Settings. Paperclip will retry automatically.", invalid_lifecycle_result: "A required plugin returned an invalid result. Check the plugin's status in Settings. Paperclip will retry automatically.", diff --git a/server/src/modules/run-dispatch/domain/wake-context.test.ts b/server/src/modules/run-dispatch/domain/wake-context.test.ts index f8d196d21ce..02f73e5d1df 100644 --- a/server/src/modules/run-dispatch/domain/wake-context.test.ts +++ b/server/src/modules/run-dispatch/domain/wake-context.test.ts @@ -16,6 +16,13 @@ describe("wake context", () => { expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionKind: "request_confirmation" })).toBe(false); expect(isResolvedInteractionContinuationWakeContext({ ...context, connectionIntentResolution: undefined })).toBe(false); }); + it("keeps a resolved interaction continuation through a Runner update wait", () => { + const context = { interactionId: "interaction-1", interactionStatus: "answered", mutation: "interaction", retryReason: "runner_revision_wait", wakeReason: "runner_revision_retry" }; + expect(isResolvedInteractionContinuationWakeContext(context)).toBe(true); + expect(isResolvedInteractionContinuationWakeContext({ ...context, mutation: undefined })).toBe(false); + expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionStatus: "pending" })).toBe(false); + expect(isResolvedInteractionContinuationWakeContext({ ...context, retryReason: "transient_failure" })).toBe(false); + }); it("requires the subscription-specific non-assignee receipt for a subscription wait", () => { expect(isNonAssigneeWorkspaceBusyRetry("ai_connection_busy", { aiConnectionBusyDeferredWhileAssignee: false, diff --git a/server/src/modules/run-dispatch/domain/wake-context.ts b/server/src/modules/run-dispatch/domain/wake-context.ts index a941139bc80..91545b4bdb3 100644 --- a/server/src/modules/run-dispatch/domain/wake-context.ts +++ b/server/src/modules/run-dispatch/domain/wake-context.ts @@ -19,6 +19,7 @@ export const MAX_TURN_CONTINUATION_RETRY_REASON = "max_turns_continuation"; export const WORKSPACE_BUSY_RETRY_REASON = "workspace_busy"; export const AI_CONNECTION_POOL_WAIT_RETRY_REASON = "ai_connection_pool_wait"; export const AI_CONNECTION_BUSY_RETRY_REASON = "ai_connection_busy"; +export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait"; export const INTERACTION_CONTINUATION_INFRA_RETRY_REASON = "interaction_continuation_infra_retry"; export const INTERACTION_CONTINUATION_INFRA_WAKE_REASON = "interaction_continuation_infra_retry"; export const WAKE_COMMENT_IDS_KEY = "wakeCommentIds"; @@ -182,6 +183,7 @@ export function isResolvedInteractionContinuationWakeContext(contextSnapshot: un const retryReason = readNonEmptyString(context.retryReason); return ( (mutation === "interaction" && wakeReason === "issue_commented") || + (mutation === "interaction" && retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) || wakeReason === INTERACTION_CONTINUATION_INFRA_WAKE_REASON || retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON ); diff --git a/server/src/modules/run-dispatch/index.ts b/server/src/modules/run-dispatch/index.ts index 4c17b386c8c..2cd18c72398 100644 --- a/server/src/modules/run-dispatch/index.ts +++ b/server/src/modules/run-dispatch/index.ts @@ -14,6 +14,7 @@ export { WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, + RUNNER_REVISION_WAIT_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_WAKE_REASON, WAKE_COMMENT_IDS_KEY, diff --git a/server/src/services/agent-environment-test.ts b/server/src/services/agent-environment-test.ts index f63358d7004..c60fd3e5a14 100644 --- a/server/src/services/agent-environment-test.ts +++ b/server/src/services/agent-environment-test.ts @@ -16,6 +16,7 @@ import { environmentService } from "./environments.js"; import { environmentRuntimeService } from "./environment-runtime.js"; import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js"; import { prepareManagedAiRuntime, assertManagedAiProjectAuth } from "./ai-connection-runtime.js"; +import { isRunnerRevisionMismatch, runnerRevisionMismatchTestResult } from "./runner-revision-mismatch.js"; import { aiConnectionService } from "./ai-connections.js"; import { validateAiApiKey } from "./ai-api-key-test.js"; import { probeCopilotExecutionTarget } from "./copilot-connection-probe.js"; @@ -573,7 +574,11 @@ export function agentEnvironmentTestService(db: Db, pluginWorkerManager?: Plugin return { adapterType, status: "fail" as const, testedAt: new Date().toISOString(), checks: [{ code: error instanceof HttpError ? String(asRecord(error.details)?.code ?? "COPILOT_REQUEST_FAILED") : "COPILOT_REQUEST_FAILED", level: "error" as const, message: error instanceof HttpError ? error.message : "Copilot metadata verification failed." }] }; } } - await assertManagedAiProjectAuth(context.config, binding.provider, context.executionTarget); + try { await assertManagedAiProjectAuth(context.config, binding.provider, context.executionTarget); } + catch (error) { + if (!isRunnerRevisionMismatch(error)) throw error; + return runnerRevisionMismatchTestResult(adapterType); + } const target = context.executionTarget; const result = adapterType === "paperclip_runner" && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask ? await probePluginRunnerEnvironment(context, { identity: managed.identity, accountIdentity: managed.sessionIdentity, diff --git a/server/src/services/agent-harness-verification.ts b/server/src/services/agent-harness-verification.ts index c8f7e21561a..4ebc7da16d5 100644 --- a/server/src/services/agent-harness-verification.ts +++ b/server/src/services/agent-harness-verification.ts @@ -13,6 +13,7 @@ import { secretService } from "./secrets.js"; import { instanceSettingsService } from "./instance-settings.js"; import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import { museRunnerBroker } from "./muse-runner-broker.js"; +import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js"; import { dotRunnerBroker } from "./dot-runner-broker.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; import type { LifecycleAgent } from "../modules/agent-lifecycle/index.js"; @@ -26,7 +27,8 @@ export function assertHarnessTestPassed(result: AdapterEnvironmentTestResult) { const incomplete = !verified && (result.status === "warn" || result.checks.some(check => check.code.includes("hello_probe"))); if (result.status === "fail" || authenticationFailed || incomplete) { const has = (suffix: string) => result.checks.some(check => check.code.endsWith(suffix)); - const code = result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable" + const code = result.checks.some(check => check.code === RUNNER_REVISION_MISMATCH_CODE) ? RUNNER_REVISION_MISMATCH_CODE + : result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable" : authenticationFailed ? "harness_auth_failed" : has("_hello_probe_model_unavailable") || has("_model_invalid") ? "harness_model_unavailable" : has("_command_unresolvable") ? "harness_command_unavailable" diff --git a/server/src/services/environment-task-runtime.ts b/server/src/services/environment-task-runtime.ts index 64af59a9ee2..0cb1f2c9f42 100644 --- a/server/src/services/environment-task-runtime.ts +++ b/server/src/services/environment-task-runtime.ts @@ -10,6 +10,7 @@ import { and, eq, inArray, sql } from "drizzle-orm"; import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db"; import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult, environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES, + environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode, type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk"; import { pluginRegistryService } from "./plugin-registry.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; @@ -461,7 +462,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage data: environmentTaskErrorData({ status: diagnostic.httpStatus, code: diagnostic.errorCode, reason: diagnostic.reason }), }); // Worker errors and schema diagnostics can contain request credentials. - throw new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"); + const code = readEnvironmentTaskErrorCode(error); + throw Object.assign(new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"), + code ? { data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError(code)) } : {}); } } diff --git a/server/src/services/execution-recovery-attempt.test.ts b/server/src/services/execution-recovery-attempt.test.ts index c434e7b60a2..8667b2e3222 100644 --- a/server/src/services/execution-recovery-attempt.test.ts +++ b/server/src/services/execution-recovery-attempt.test.ts @@ -1,5 +1,7 @@ import { describe, expect, it } from "vitest"; -import { executionFailureRetryCount } from "./execution-recovery-attempt.js"; +import { accountingForScheduledRetry, executionFailureRetryCount, executionRetryAttemptCount } from "./execution-recovery-attempt.js"; +import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js"; +import { RUNNER_REVISION_WAIT_RETRY_REASON as DISPATCH_RUNNER_REVISION_WAIT_RETRY_REASON } from "../modules/run-dispatch/index.js"; describe("failure attempts across resource waits", () => { it("preserves prior failures through subscription waits without trusting unrelated context", () => { @@ -27,7 +29,7 @@ describe("failure attempts across resource waits", () => { }); describe("persisted independent accounting", () => { - it.each(["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait"])("%s cannot erase prior infrastructure debits or spend more", scheduledRetryReason => { + it.each(["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait"])("%s cannot erase prior infrastructure debits or spend more", scheduledRetryReason => { expect(executionFailureRetryCount({ scheduledRetryReason, scheduledRetryAttempt: 20, contextSnapshot: { executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 1 } }, })).toBe(2); @@ -43,3 +45,25 @@ describe("persisted independent accounting", () => { })).toBe(2); }); }); + +describe("Runner update waits", () => { + it("uses the same retry reason as run dispatch", () => { + expect(RUNNER_REVISION_WAIT_RETRY_REASON).toBe(DISPATCH_RUNNER_REVISION_WAIT_RETRY_REASON); + }); + const ledger = (failureRetries: number) => ({ executionRetryAccounting: { version: 1, failureRetries, maxTurnContinuations: 0 } }); + it("never spend the failure retry budget", () => { + const waited = { scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 9, contextSnapshot: ledger(0) }; + expect(executionRetryAttemptCount(waited, "runner_revision_wait")).toBe(9); + expect(accountingForScheduledRetry(waited, "runner_revision_wait", 10)).toMatchObject({ failureRetries: 0 }); + expect(executionFailureRetryCount(waited)).toBe(0); + expect(executionRetryAttemptCount(waited, "transient_failure")).toBe(0); + }); + it("keep failures recorded before the wait and are not shortened by them", () => { + const failed = { scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 1, contextSnapshot: ledger(1) }; + expect(executionRetryAttemptCount(failed, "runner_revision_wait")).toBe(0); + const waited = { scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 3, + contextSnapshot: { ...ledger(1), executionRetryAccounting: accountingForScheduledRetry(failed, "runner_revision_wait", 3) } }; + expect(executionFailureRetryCount(waited)).toBe(1); + expect(executionRetryAttemptCount(waited, "transient_failure")).toBe(1); + }); +}); diff --git a/server/src/services/execution-recovery-attempt.ts b/server/src/services/execution-recovery-attempt.ts index dec0d847cf7..d63f9706181 100644 --- a/server/src/services/execution-recovery-attempt.ts +++ b/server/src/services/execution-recovery-attempt.ts @@ -1,3 +1,5 @@ +import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js"; + type RetryRun = { scheduledRetryAttempt?: number | null; scheduledRetryReason?: string | null; @@ -42,7 +44,7 @@ function historicalFailureCount(run: RetryRun): number { export function executionRetryAccounting(run: RetryRun): ExecutionRetryAccounting { const saved = savedAccounting(run); - const nonFailureLane = ["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait"].includes(run.scheduledRetryReason ?? ""); + const nonFailureLane = ["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", RUNNER_REVISION_WAIT_RETRY_REASON].includes(run.scheduledRetryReason ?? ""); return { version: 1, failureRetries: Math.max(saved?.failureRetries ?? 0, saved && nonFailureLane ? 0 : historicalFailureCount(run)), @@ -57,7 +59,7 @@ export function executionFailureRetryCount(run: RetryRun): number { } export function executionRetryAttemptCount(run: RetryRun, reason: string): number { - if (reason === "workspace_busy" || reason === "ai_connection_busy" || reason === "ai_connection_pool_wait") { + if (reason === "workspace_busy" || reason === "ai_connection_busy" || reason === "ai_connection_pool_wait" || reason === RUNNER_REVISION_WAIT_RETRY_REASON) { return run.scheduledRetryReason === reason ? count(run.scheduledRetryAttempt) ?? 0 : 0; } const accounting = executionRetryAccounting(run); @@ -67,6 +69,7 @@ export function executionRetryAttemptCount(run: RetryRun, reason: string): numbe export function accountingForScheduledRetry(run: RetryRun, reason: string, attempt: number): ExecutionRetryAccounting { const accounting = executionRetryAccounting(run); if (reason === "max_turns_continuation") accounting.maxTurnContinuations = attempt; - else if (reason !== "workspace_busy" && reason !== "ai_connection_busy" && reason !== "ai_connection_pool_wait") accounting.failureRetries = attempt; + else if (reason !== "workspace_busy" && reason !== "ai_connection_busy" && reason !== "ai_connection_pool_wait" + && reason !== RUNNER_REVISION_WAIT_RETRY_REASON) accounting.failureRetries = attempt; return accounting; } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 97f58880e84..accceff958c 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -98,7 +98,7 @@ import { createHeartbeatRecovery, } from "./heartbeat/recovery.js"; import { - BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS, + failedRunRetryBudgetSpent, isWorkspaceBusyDeferral, createHeartbeatRetries, } from "./heartbeat/retries.js"; @@ -147,6 +147,7 @@ export { import { ConfigurationIncompleteFailure, + classifyManagedProjectAuthFailure, deriveTaskKey, type WakeupOptions, mergeCoalescedContextSnapshot, @@ -289,7 +290,6 @@ import { retryChatControlAdmission } from "./chat-control-admission-retry.js"; import { prepareManagedAiRuntime, assertManagedAiProjectAuth, - isAiConnectionIncompatible, } from "./ai-connection-runtime.js"; import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared"; import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js"; @@ -307,7 +307,6 @@ import { adapterExecutionControls, createAdapterExecutionControl, } from "./adapter-execution-control.js"; -import { executionFailureRetryCount } from "./execution-recovery-attempt.js"; import { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status-payload.js"; export { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status-payload.js"; import { StaleExecutionContinuationError } from "./execution-continuation.js"; @@ -1002,9 +1001,7 @@ export function heartbeatService( // Mirrors scheduleBoundedRetryForRun's transient budget check: a failed // or interrupted run that has already consumed every bounded transient // attempt cannot be retried again through this lane. - transientRetryBudgetSpent: (run) => - executionFailureRetryCount(run) >= - BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS, + transientRetryBudgetSpent: failedRunRetryBudgetSpent, }); const runDispatch = createRunDispatch(db); const { @@ -2825,11 +2822,7 @@ export function heartbeatService( }); if (managedAiRuntime && aiBinding) { try { await assertManagedAiProjectAuth({ ...resolvedConfig, cwd: executionWorkspace.cwd }, aiBinding.provider, executionTarget); } - catch (error) { - // Only a found conflict is the board's to fix; an unfinished check stays an ordinary failure. - if (!isAiConnectionIncompatible(error)) throw error; - throw new ConfigurationIncompleteFailure(error.message, { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } }); - } + catch (error) { throw classifyManagedProjectAuthFailure(error, agent.id); } } const remoteExecution = realizationResult.remoteExecution; if (nativeChatWorkspaceScope && materializeNativeChatWorktree) { diff --git a/server/src/services/heartbeat/retries.test.ts b/server/src/services/heartbeat/retries.test.ts index 6c928006482..5ee40c701ac 100644 --- a/server/src/services/heartbeat/retries.test.ts +++ b/server/src/services/heartbeat/retries.test.ts @@ -6,7 +6,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest import { writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; import { HttpError } from "../../errors.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; -import { createRunDispatch, type PostCommitEffect } from "../../modules/run-dispatch/index.js"; +import { createRunDispatch, isResolvedInteractionContinuationWakeContext, type PostCommitEffect } from "../../modules/run-dispatch/index.js"; import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; import * as legacy from "../heartbeat.js"; import { createHeartbeatRunState } from "./run-state.js"; @@ -112,6 +112,22 @@ describe("heartbeat retry module callbacks", () => { expect(database.access).not.toHaveBeenCalled(); }); + it("gives every caller the Runner wait budget for a Runner mismatch and reports its exhaustion", async () => { + const database = guardedDatabase(); + const deps = callbacks(); + const run = { ...unitRun(), status: "failed", errorCode: "runner_revision_mismatch", scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 10, + contextSnapshot: { issueId: "issue", executionRetryAccounting: { version: 1, failureRetries: 0, maxTurnContinuations: 0 } } } as Run; + const retry = await extracted.createHeartbeatRetries(database.db, deps).scheduleBoundedRetryForRun(run, { adapterType: "codex_local" } as Agent); + expect(retry).toMatchObject({ outcome: "retry_exhausted", attempt: 11, maxAttempts: 10 }); + expect(deps.appendRunEvent).toHaveBeenCalledWith(run, expect.objectContaining({ + retryExhaustion: { retryReason: "runner_revision_wait", scheduledRetryAttempt: 10, maxAttempts: 10 }, + })); + expect(extracted.failedRunRetryBudgetSpent(run)).toBe(true); + expect(extracted.failedRunRetryBudgetSpent({ ...run, scheduledRetryAttempt: 9 })).toBe(false); + expect(extracted.failedRunRetryBudgetSpent({ ...run, errorCode: "setup_failed" })).toBe(false); + expect(database.access).not.toHaveBeenCalled(); + }); + it("leaves successor work to the committed case handoff without retry escalation", async () => { const database = guardedDatabase(); const deps = callbacks(); @@ -273,6 +289,23 @@ describePostgres("heartbeat retry module database wiring", () => { } }); + it("waits for a Runner update without spending failure retries and keeps an answered interaction", async () => { + const { run, agent } = await fixture(); + const snapshot = { ...run.contextSnapshot, interactionId: "interaction-1", interactionStatus: "answered", interactionKind: "ask_user_questions", + mutation: "interaction", wakeReason: "issue_commented", executionRetryAccounting: { version: 1, failureRetries: 1, maxTurnContinuations: 0 } }; + const [failed] = await db.update(heartbeatRuns).set({ errorCode: "runner_revision_mismatch", contextSnapshot: snapshot }) + .where(eq(heartbeatRuns.id, run.id)).returning(); + const result = await boundRetries().retries.scheduleBoundedRetryForRun(failed!, agent, { now }); + expect(result.outcome).toBe("scheduled"); + const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id)); + expect(successor).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 1 }); + expect(successor!.contextSnapshot).toMatchObject({ mutation: "interaction", interactionId: "interaction-1", + executionRetryAccounting: { failureRetries: 1 } }); + expect(isResolvedInteractionContinuationWakeContext(successor!.contextSnapshot)).toBe(true); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, successor!.wakeupRequestId!)); + expect(wake!.payload).toMatchObject({ mutation: "interaction", interactionId: "interaction-1", retryReason: "runner_revision_wait" }); + }); + it("resolves current skills on a retry while retaining the failed attempt snapshot", async () => { const {run,agent}=await fixture(); const snapshot={...run.contextSnapshot,skillVersions:{"company/review":"v1"},skillSelectionSnapshot:true,shadowedSkills:[{key:"old",shadowedBy:"company/review"}]}; diff --git a/server/src/services/heartbeat/retries.ts b/server/src/services/heartbeat/retries.ts index 40cbaf4731a..122a1c63787 100644 --- a/server/src/services/heartbeat/retries.ts +++ b/server/src/services/heartbeat/retries.ts @@ -24,6 +24,7 @@ import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-sched import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js"; import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js"; import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js"; +import { RUNNER_REVISION_MISMATCH_CODE, RUNNER_REVISION_MISMATCH_RETRY, RUNNER_REVISION_WAIT_RETRY_REASON } from "../runner-revision-mismatch.js"; import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js"; import { randomUUID } from "node:crypto"; import { @@ -136,12 +137,19 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry"; export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean { return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated || code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE - || code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE; + || code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE + || code === RUNNER_REVISION_MISMATCH_CODE; } export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS = BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length; +export function failedRunRetryBudgetSpent(run: Parameters[0] & { errorCode?: string | null }): boolean { + return run.errorCode === RUNNER_REVISION_MISMATCH_CODE + ? executionRetryAttemptCount(run, RUNNER_REVISION_WAIT_RETRY_REASON) >= RUNNER_REVISION_MISMATCH_RETRY.maxAttempts + : executionFailureRetryCount(run) >= BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS; +} + const INTERACTION_CONTINUATION_INFRA_MAX_ATTEMPTS = 2; const MAX_TURN_CONTINUATION_LIVE_RUN_STATUSES = [ @@ -343,7 +351,8 @@ function isRetryableInteractionContinuationInfrastructureFailure( run.errorCode === WORKSPACE_VALIDATION_FAILURE_CODE || run.errorCode === "process_lost" || run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE || - run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE + run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE || + run.errorCode === RUNNER_REVISION_MISMATCH_CODE ) { return true; } @@ -419,6 +428,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen delayMs?: number; }, ) { + if (!opts?.retryReason && run.errorCode === RUNNER_REVISION_MISMATCH_CODE) opts = { ...RUNNER_REVISION_MISMATCH_RETRY, ...opts }; if (await hasCaseHandoffForRun(run)) return { outcome: "not_scheduled" as const, reason: "The durable case handoff owns successor work.", issueId: readNonEmptyString(run.contextSnapshot?.issueId) }; if (["openai_dot", "muse"].includes(String(parseObject(agent.adapterConfig).provider)) @@ -615,8 +625,10 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen } const taskKey = deriveTaskKeyWithHeartbeatFallback(contextSnapshot, null); const sessionBefore = await resolveSessionBeforeForWakeup(agent, taskKey); + const interactionRunnerWait = retryReason === RUNNER_REVISION_WAIT_RETRY_REASON + && isResolvedInteractionContinuationWakeContext(run.contextSnapshot); const interactionContinuationPayload = - retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON + retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON || interactionRunnerWait ? { mutation: "interaction", interactionId: readNonEmptyString(contextSnapshot.interactionId), @@ -646,6 +658,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen retryOfRunId: run.id, wakeReason, retryReason, + ...(interactionRunnerWait ? { mutation: "interaction" } : {}), ...(retryReason === WORKSPACE_BUSY_RETRY_REASON ? { failureRetriesBeforeWorkspaceWait: @@ -1001,7 +1014,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen const scheduledRunId = randomUUID(); if (contextSnapshot.explicitUserContinuation) { - const continuation = issueId && retryReason === "transient_failure" ? await admitExplicitContinuationRetry({ + const continuation = issueId && (retryReason === "transient_failure" || retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) ? await admitExplicitContinuationRetry({ db: tx as unknown as Db, companyId: run.companyId, issueId, agentId: run.agentId, parentRunId: run.id, successorRunId: scheduledRunId, now, }) : null; diff --git a/server/src/services/heartbeat/run-completion.test.ts b/server/src/services/heartbeat/run-completion.test.ts index f24711c4e6e..524f9c0fc8b 100644 --- a/server/src/services/heartbeat/run-completion.test.ts +++ b/server/src/services/heartbeat/run-completion.test.ts @@ -30,8 +30,11 @@ import { RemoteWorkspaceOperationError, RemoteWorkspaceOperationTimeoutError, Re import { EnvironmentRunError } from "../environment-run-orchestrator.js"; import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js"; import { PluginWorkerUnavailableError } from "../plugin-worker-unavailable-error.js"; +import { environmentTaskErrorCodeData, JsonRpcCallError, PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk"; +import { RUNNER_REVISION_MISMATCH_RETRY } from "../runner-revision-mismatch.js"; +import { unprocessable } from "../../errors.js"; import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js"; -import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure } from "./run-preparation.js"; +import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure, classifyManagedProjectAuthFailure } from "./run-preparation.js"; import { createHeartbeatRunState, getAdapterSessionCodec } from "./run-state.js"; import { buildEffectiveRunSessionConfigMetadata } from "./workspaces.js"; import { createHeartbeatRunCompletion, type CompleteHeartbeatRunInput, type FailHeartbeatRunInput, type HeartbeatRunCompletionDependencies } from "./run-completion.js"; @@ -966,6 +969,61 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)( } else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled(); }); + const runnerMismatch = () => new JsonRpcCallError({ code: -32002, message: "The installed Runner does not match the expected revision", + data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) }); + + it.each(["fresh", "wrapped", "started", "untrusted", "retained"] as const)("waits for the Runner update after a %s Runner mismatch", async scenario => { + const f = await fixture({ runtimeMode: scenario === "retained" ? "native" : "legacy" }); + const refusal = runnerMismatch(); + const error = scenario === "untrusted" ? new Error(refusal.message) + : scenario === "wrapped" ? new Error("Plugin Runner task failed", { cause: refusal }) + : classifyManagedProjectAuthFailure(refusal, f.agent.id); + f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null }); + await f.completion.failRunSetup(f.setup(error, scenario === "started")); + const failed = await read(f.run.id); + const retried = scenario === "fresh" || scenario === "wrapped"; + expect(failed).toMatchObject({ status: "failed", errorCode: retried ? "runner_revision_mismatch" : "setup_failed" }); + expect(failed.resultJson).not.toHaveProperty("configurationIncomplete"); + if (retried) { + expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }); + expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent, RUNNER_REVISION_MISMATCH_RETRY); + } else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled(); + }); + + it("waits for the Runner update on an answered interaction instead of its two-attempt continuation retry", async () => { + const f = await fixture({ contextSnapshot: { interactionId: "interaction-1", interactionStatus: "answered", interactionKind: "ask_user_questions", + mutation: "interaction", wakeReason: "issue_commented" } }); + f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null }); + await f.completion.failRunSetup(f.setup(runnerMismatch())); + expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "runner_revision_mismatch" }); + expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent, RUNNER_REVISION_MISMATCH_RETRY); + expect(f.deps.scheduleInteractionContinuationInfrastructureRetryIfEligible).not.toHaveBeenCalled(); + }); + + it("keeps a found project authentication conflict a configuration gap that blocks for the board", async () => { + const f = await fixture(); + const conflict = unprocessable("This agent's managed AI connection conflicts with project authentication settings in .claude/settings.json.", + { code: "ai_connection_incompatible" }); + await f.completion.failRunSetup(f.setup(classifyManagedProjectAuthFailure(conflict, f.agent.id))); + expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "configuration_incomplete", + resultJson: { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${f.agent.id}/runtime` } } }); + expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled(); + }); + + it.each([ + ["an unavailable Runner identity", new Error("Runner identity unavailable")], + ["an unreachable execution host", Object.assign(new Error("connect ECONNREFUSED"), { code: "ECONNREFUSED" })], + ["another 422", unprocessable("Select a saved AI account", { code: "ai_connection_busy" })], + ])("does not report %s as a project authentication conflict", async (_name, error) => { + const f = await fixture(); + const classified = classifyManagedProjectAuthFailure(error, f.agent.id); + expect(classified).toBe(error); + await f.completion.failRunSetup(f.setup(classified)); + const failed = await read(f.run.id); + expect(failed.errorCode).not.toBe("configuration_incomplete"); + expect(failed.resultJson).not.toHaveProperty("configurationIncomplete"); + }); + it("routes a lease acquisition refused by a stopped provider worker through bounded bootstrap recovery", async () => { const f = await fixture(); const refusal = new PluginWorkerUnavailableError({ pluginId: "plugin-1", workerStatus: null, diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index c0ecf1746d3..00622710994 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -13,6 +13,7 @@ import { RemoteWorkspacePreparationTimeoutError, REMOTE_WORKSPACE_RESTORE_FAILED import { escalateRemoteWorkspaceRestoreFailure } from "../remote-workspace-restore-recovery.js"; import { findReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js"; import { findPluginWorkerUnavailableError, PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js"; +import { isRunnerRevisionMismatch, RUNNER_REVISION_MISMATCH_CODE, RUNNER_REVISION_MISMATCH_RETRY } from "../runner-revision-mismatch.js"; import { type RunSessionOutcome, resolveNextSessionState, normalizeUsageTotals, resolveCacheAdjustedCostUsd, resolveLedgerBiller, resolveLedgerCostStatus, normalizeLedgerBillingType } from "./run-state.js"; import { readGitConnectionFailure } from "../git-connection-failure.js"; import { CONFIGURATION_INCOMPLETE_FAILURE_CODE, ConfigurationIncompleteFailure } from "./run-preparation.js"; @@ -1376,12 +1377,15 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC // bootstrap budget; the worker is restarted or replaced meanwhile. const pluginWorkerUnavailableFailure = !providerDispatchStarted && run.runtimeMode === "legacy" && run.nativeSessionId === null && run.runnerInstanceId === null ? findPluginWorkerUnavailableError(outerErr) : null; + const runnerRevisionMismatch = !providerDispatchStarted && run.runtimeMode === "legacy" + && run.nativeSessionId === null && run.runnerInstanceId === null && isRunnerRevisionMismatch(outerErr); const setupFailureErrorCode = (controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ?? remoteWorkspacePreparationTimeout?.code ?? reusableSandboxResumeFailure?.code ?? remoteWorkspaceRestoreFailure?.code ?? pluginWorkerUnavailableFailure?.code ?? + (runnerRevisionMismatch ? RUNNER_REVISION_MISMATCH_CODE : null) ?? workspaceGitScanFailure?.code ?? workspaceValidationSetupFailure?.code ?? configurationIncompleteSetupFailure?.code ?? @@ -1544,7 +1548,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC const interactionOwnsRetry = livenessRun.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE && isResolvedInteractionContinuationWakeContext(livenessRun.contextSnapshot); await (isTransientWorkspacePreparationCode(livenessRun.errorCode) && !interactionOwnsRetry - ? scheduleBoundedRetryForRun(livenessRun, failedAgent) + ? livenessRun.errorCode === RUNNER_REVISION_MISMATCH_CODE + ? scheduleBoundedRetryForRun(livenessRun, failedAgent, RUNNER_REVISION_MISMATCH_RETRY) + : scheduleBoundedRetryForRun(livenessRun, failedAgent) : scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent) ).catch((retryError) => { logger.warn( diff --git a/server/src/services/heartbeat/run-preparation.ts b/server/src/services/heartbeat/run-preparation.ts index ea292d8f4e9..95f6ad89e4e 100644 --- a/server/src/services/heartbeat/run-preparation.ts +++ b/server/src/services/heartbeat/run-preparation.ts @@ -69,6 +69,7 @@ import { toolProfiles, } from "@paperclipai/db"; import { HttpError } from "../../errors.js"; +import { isAiConnectionIncompatible } from "../ai-connection-runtime.js"; import type { ReviewedChatBindingDenialReason } from "../reviewed-chat-binding-diagnostics.js"; import { filterResolvedGitHubConnectionsForRun } from "../git-credentials.js"; import type { @@ -162,6 +163,13 @@ export class ConfigurationIncompleteFailure extends Error { } } +export function classifyManagedProjectAuthFailure(error: unknown, agentId: string): unknown { + if (!isAiConnectionIncompatible(error)) return error; + return new ConfigurationIncompleteFailure(error.message, { + configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agentId}/runtime` }, + }); +} + const ISSUE_RESPONSIBLE_USER_WAKE_REASONS = new Set([ "issue_assigned", "issue_checked_out", diff --git a/server/src/services/legacy-execution-recovery.test.ts b/server/src/services/legacy-execution-recovery.test.ts index 2d632370260..365a89cfbed 100644 --- a/server/src/services/legacy-execution-recovery.test.ts +++ b/server/src/services/legacy-execution-recovery.test.ts @@ -10,7 +10,7 @@ const stopped = { }, }; -it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => { +it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable", "runner_revision_mismatch"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => { const run = { runtimeMode: "legacy", status: "failed", errorCode, scheduledRetryAttempt: 2, resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }; expect(legacyExecutionNeedsReconciliation(run)).toBe(false); diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index df6cfaac8f8..1059a7978a5 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -9,6 +9,7 @@ import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wai import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js"; import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js"; import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "./plugin-worker-unavailable-error.js"; +import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js"; import { hasWorkspaceRestoreFailure } from "@paperclipai/shared"; import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js"; import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js"; @@ -79,7 +80,8 @@ export function legacyExecutionNeedsReconciliation( // that the bootstrap evidence proves never started. Keep unknown outcomes held. if ((run.errorCode === "workspace_git_scan_timeout" || run.errorCode === "workspace_git_scan_saturated" || run.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE - || run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE) && + || run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE + || run.errorCode === RUNNER_REVISION_MISMATCH_CODE) && evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false) return false; if (executionFailureRetryCount(run) >= 2) return true; return !( diff --git a/server/src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts b/server/src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts new file mode 100644 index 00000000000..3b1f3829a55 --- /dev/null +++ b/server/src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts @@ -0,0 +1,26 @@ +import { describe, expect, it } from "vitest"; +import { environmentTaskErrorCodeData, JsonRpcCallError, PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk"; +import { probePluginRunnerEnvironment } from "./plugin-runner-probe.js"; +import { assertHarnessTestPassed } from "../agent-harness-verification.js"; + +const probe = (runnerTask: () => Promise) => probePluginRunnerEnvironment({ + companyId: "fixture-company", adapterType: "paperclip_runner", config: { provider: "codex", model: "fixture-model" }, + executionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "probe-lease", + runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session" }, runnerTask }, +} as Parameters[0]); + +describe("plugin Runner probe with an outdated remote Runner", () => { + it("reports a retryable Runner mismatch from the plugin's stable code", async () => { + const refusal = new JsonRpcCallError({ code: -32002, message: "The installed Runner does not match the expected revision", + data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) }); + const result = await probe(async () => { throw refusal; }); + expect(result).toMatchObject({ status: "fail", checks: [{ code: "runner_revision_mismatch", level: "error", retryable: true }] }); + expect(() => assertHarnessTestPassed(result)).toThrow(expect.objectContaining({ code: "runner_revision_mismatch" })); + }); + + it("keeps other plugin failures a non-retryable failed probe, even with the same message", async () => { + const result = await probe(async () => { throw new Error("The installed Runner does not match the expected revision"); }); + expect(result).toMatchObject({ status: "fail", checks: [{ code: "runner_hello_probe_failed", level: "error" }] }); + expect(result.checks[0]).not.toHaveProperty("retryable"); + }); +}); diff --git a/server/src/services/native-runtime/plugin-runner-probe.ts b/server/src/services/native-runtime/plugin-runner-probe.ts index 4bf9f3cc022..f337c8348b6 100644 --- a/server/src/services/native-runtime/plugin-runner-probe.ts +++ b/server/src/services/native-runtime/plugin-runner-probe.ts @@ -11,6 +11,7 @@ import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js"; import { createPluginRunnerTaskOwner } from "./plugin-runner-task.js"; import { resolvePaperclipRunnerProviderProfile } from "./provider-profile.js"; import { resolveAcpxQualification } from "./acpx-qualification.js"; +import { isRunnerRevisionMismatch, runnerRevisionMismatchTestResult } from "../runner-revision-mismatch.js"; /** Core verifies a real Runner; providers only implement their task API. */ export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTestContext, @@ -70,6 +71,7 @@ export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTe checks: [{ code: "runner_hello_probe_passed", level: "info", message: `Runner verified ${result.providerDriver} with model ${result.effectiveModel}.` }] }; } catch (error) { if (error instanceof NativeEnvironmentProbeCleanupError) throw error; + if (isRunnerRevisionMismatch(error)) return runnerRevisionMismatchTestResult("paperclip_runner", testedAt); // Provider failures may contain credentials or private addressing. return { adapterType: "paperclip_runner", status: "fail", testedAt, checks: [{ code: "runner_hello_probe_failed", level: "error", message: "The selected plugin environment did not complete the Runner hello probe and confirmed cleanup. Check its runtime, account, and model access." }] }; diff --git a/server/src/services/recovery/service.pause-durability.test.ts b/server/src/services/recovery/service.pause-durability.test.ts index 7d9e7407f97..187dee0821a 100644 --- a/server/src/services/recovery/service.pause-durability.test.ts +++ b/server/src/services/recovery/service.pause-durability.test.ts @@ -5,7 +5,7 @@ const run = (errorCode: string | null) => ({ errorCode } as unknown as Parameters[0]); describe("pause durability: continuation retry classification", () => { - it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not grant %s another recovery budget", (code) => { + it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable", "runner_revision_mismatch"])("does not grant %s another recovery budget", (code) => { expect(classifyContinuationFailure(run(code))).toMatchObject({ kind: "non_retryable", maxAttempts: 0 }); }); it("agent_paused is retryable so work resumes (Option A: Resume Continues Work)", () => { diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 39fd3e77adc..f4697c129f9 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -10,6 +10,7 @@ import { boundRunIssueId } from "../run-task-binding.js"; import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js"; import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js"; import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js"; +import { RUNNER_REVISION_MISMATCH_CODE } from "../runner-revision-mismatch.js"; import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js"; import { recoveryRunResultColumn } from "./run-result-read.js"; import { readLegacyDispositionSource } from "./legacy-disposition-read.js"; @@ -561,6 +562,7 @@ const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set([ REUSABLE_SANDBOX_RESUME_FAILED_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE, PLUGIN_WORKER_UNAVAILABLE_CODE, + RUNNER_REVISION_MISMATCH_CODE, "low_trust_isolation_unavailable", "low_trust_requires_isolated_workspace", "low_trust_boundary_mismatch", diff --git a/server/src/services/recovery/stranded-notice.test.ts b/server/src/services/recovery/stranded-notice.test.ts index 71d3c4c759d..d00131f7d9a 100644 --- a/server/src/services/recovery/stranded-notice.test.ts +++ b/server/src/services/recovery/stranded-notice.test.ts @@ -321,3 +321,25 @@ it("names the unavailable AI account instead of suggesting secret bindings", () expect(notice.body).toContain("Reconnect the account"); expect(notice.body).not.toContain("secret/env"); }); + +it("names a found project authentication conflict instead of suggesting secret bindings", () => { + const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "ai_connection_incompatible", actionUrl: "/agents/agent-1/runtime" }); + expect(notice.title).toBe("AI connection conflicts with project settings"); + expect(notice.body).toContain("apiKeyHelper"); + expect(notice.nextAction).toContain("choose a different AI connection"); + expect(notice.body).not.toContain("secret/env"); +}); + +it("names a remote Runner that was still not updated after its retries", () => { + const notice = buildStrandedRecoveryEscalationNotice({ + seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }), + recoveryActionId: "runner-recovery", + recoveryOwner: null, + sourceRun: { id: "failed-run", status: "failed", errorCode: "runner_revision_mismatch" }, + }); + expect(notice.presentation.title).toBe("Remote Runner is out of date"); + expect(notice.body).toContain("Runner was still not updated"); + expect(notice.body).not.toContain("workspace"); + expect(notice.body).not.toContain("authentication"); + expect(JSON.stringify(notice.metadata)).toContain("remote Runner received the update"); +}); diff --git a/server/src/services/recovery/stranded-notice.ts b/server/src/services/recovery/stranded-notice.ts index c1fbd1877be..3a302bf3feb 100644 --- a/server/src/services/recovery/stranded-notice.ts +++ b/server/src/services/recovery/stranded-notice.ts @@ -64,6 +64,16 @@ const WORKSPACE_SCAN_NOTICES: Record = { + runner_revision_mismatch: { + title: "Remote Runner is out of date", + body: "The remote Runner does not match this Paperclip version, which is expected for a few minutes while a release updates the Runner. " + + "Paperclip retried before the agent started, but the Runner was still not updated.", + nextAction: "Check that the remote Runner received the update for this release, then retry the task.", + tone: "danger", + }, +}; + export function buildImmediateExecutionPathRecoveryNoticeSeed(input: { status: "todo" | "in_progress"; }): StrandedRecoveryNoticeSeed { @@ -162,6 +172,16 @@ export function buildConfigurationIncompleteRecoveryNoticeSeed( tone: "danger", }; } + if (readNonEmptyStringField(configurationIncomplete, "reason") === "ai_connection_incompatible") { + return { + title: "AI connection conflicts with project settings", + body: "Paperclip stopped before the agent started because this agent's managed AI connection conflicts with authentication settings " + + "in the project or the agent's arguments, such as an API key, token or apiKeyHelper. The task is blocked until the conflict is removed.", + nextAction: "Remove the API key, token or apiKeyHelper from the project's Claude or Codex settings or the agent's arguments, " + + "or choose a different AI connection for this agent, then retry the task.", + tone: "danger", + }; + } if (readNonEmptyStringField(configurationIncomplete, "reason") === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON) { const pluginKey = readNonEmptyStringField(configurationIncomplete, "pluginKey") ?? "the sandbox provider plugin"; const pluginStatus = readNonEmptyStringField(configurationIncomplete, "pluginStatus") ?? "not ready"; @@ -229,7 +249,7 @@ export function buildStrandedRecoveryEscalationNotice(input: { ...workspaceScan, body: `Paperclip could not prepare the workspace before the agent started. Automatic recovery could not continue. ${workspaceScan.nextAction}`, tone: "danger" as const, - } : input.seed; + } : SETUP_FAILURE_NOTICES[input.sourceRun?.errorCode ?? ""] ?? input.seed; const fallbackBody = input.fallbackBody?.trim(); const body = seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY); const title = diff --git a/server/src/services/runner-revision-mismatch.ts b/server/src/services/runner-revision-mismatch.ts new file mode 100644 index 00000000000..0ae2e0da07a --- /dev/null +++ b/server/src/services/runner-revision-mismatch.ts @@ -0,0 +1,19 @@ +import type { AdapterEnvironmentTestResult } from "@paperclipai/adapter-utils"; +import { readEnvironmentTaskErrorCode } from "@paperclipai/plugin-sdk"; + +export const RUNNER_REVISION_MISMATCH_CODE = "runner_revision_mismatch"; + +export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait"; + +export const RUNNER_REVISION_MISMATCH_RETRY = { + retryReason: RUNNER_REVISION_WAIT_RETRY_REASON, wakeReason: "runner_revision_retry", maxAttempts: 10, delayMs: 60_000, +} as const; + +export function isRunnerRevisionMismatch(error: unknown): boolean { + return readEnvironmentTaskErrorCode(error) === RUNNER_REVISION_MISMATCH_CODE; +} + +export function runnerRevisionMismatchTestResult(adapterType: string, testedAt = new Date().toISOString()): AdapterEnvironmentTestResult { + return { adapterType, status: "fail", testedAt, checks: [{ code: RUNNER_REVISION_MISMATCH_CODE, level: "error", retryable: true, + message: "The remote Runner does not match this Paperclip version yet. Paperclip will check again after the Runner update." }] }; +}