fix(server): retry a run when its remote Runner is out of date (#19326)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice

## Linked Issues or Issue Description

Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)

**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.

**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.

**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.

**Paperclip version or commit**
master at 377e98f1a2.

**Deployment mode**
Authenticated deployment with a plugin environment that provides a
remote Runner.

## What Changed

- Plugin SDK: add `PluginEnvironmentTaskError` and
`ENVIRONMENT_TASK_ERROR_CODES` with one code,
`runner_revision_mismatch`. The worker RPC host sends it as structured
`environmentTask` error data with the schema
`paperclip/environment-task-error-code/v1`
(`environmentTaskErrorCodeData`, read with
`readEnvironmentTaskErrorCode`). This uses the same envelope method as
the other environment RPC errors. Core reads only the code, never the
message. The source refusal diagnostic from #19302 keeps its own schema
and API unchanged, and it takes precedence on a source `submit`.
- `executeEnvironmentTask` still replaces each worker error with a
generic error. After the #19302 diagnostic check, it now keeps only that
closed code.
- Run setup: a Runner mismatch before provider dispatch gets the error
code `runner_revision_mismatch`. Setup schedules a bounded retry with
its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s
apart). The reason is in the same resource-wait lane as workspace and AI
connection waits, so a wait does not spend the failure retry budget. The
guards are the same as for `plugin_worker_unavailable`. Recovery and
legacy reconciliation treat the code like the other bootstrap-only
codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and
budget for a mismatch run when a caller gives no reason, and the
recovery sweeper reports exhaustion from that budget
(`failedRunRetryBudgetSpent`). An answered interaction that hits a
mismatch also waits for the Runner, and its successor keeps the resolved
interaction continuation identity.
- Move the managed project auth classification from `heartbeat.ts` to
`classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a
found `ai_connection_incompatible` conflict becomes a configuration
failure. All other errors keep their own code.
- Recovery notices: `ai_connection_incompatible` now has its own notice
instead of the "secret/env bindings" text. An exhausted Runner wait gets
the notice "Remote Runner is out of date".
- Agent verification: the Runner probe and the managed project auth
precheck report a mismatch as a retryable check.
`assertHarnessTestPassed` maps it to the new lifecycle code
`runner_revision_mismatch`. That code never counts toward the
three-failure park. The agent checks again every 60 s and becomes ready
when the Runner matches.
- Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`.

## Verification

- `cd packages/plugins/sdk && npx vitest run
tests/environment-task-error-code.test.ts
tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the
code crosses the real `environmentTask` RPC. A plain error with the same
message gets no code. Other methods get no envelope. The #19302
diagnostic tests still pass.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped
mismatch gets `runner_revision_mismatch` and the runner retry budget,
with no configuration failure. A started, untrusted (message only) or
native run keeps `setup_failed` with no retry. A real
`ai_connection_incompatible` stays `configuration_incomplete` with no
retry. Other errors, such as an unreachable execution host, are not
reported as an authentication conflict.
- `npx vitest run src/services/execution-recovery-attempt.test.ts`:
Runner waits do not change `failureRetries`, and earlier failures do not
shorten the wait.
- `npx vitest run src/services/heartbeat/retries.test.ts
src/modules/run-dispatch` (embedded Postgres): a recovery caller without
options gets the Runner wait budget and its exhaustion. An answered
interaction waits for the Runner, keeps its continuation identity and
does not spend failure retries.
- `npx vitest run src/__tests__/environment-task-runtime.test.ts`:
sanitization keeps only the code. No worker text or cause survives.
- `npx vitest run
src/modules/agent-lifecycle/__tests__/verification-retry.test.ts`
(embedded Postgres): a `verifying` agent keeps checking after five
mismatches. It never gets `retryRequired`. It becomes ready when the
next check passes.
- `npx vitest run
src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts
src/__tests__/agent-harness-verification-remote.test.ts
src/services/recovery/stranded-notice.test.ts
src/services/recovery/service.pause-durability.test.ts
src/services/legacy-execution-recovery.test.ts`.
- `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`.
- One test in `src/__tests__/heartbeat-process-recovery.test.ts`
("redacts opaque environment-bound credentials from Sentry diagnostics")
fails on my machine. It fails the same way on the base commit without
this change.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them.

## Risks

- Low to medium. The change adds one new setup failure code and one
lifecycle code. Other failures keep their current behavior.
- A Runner mismatch must come from a plugin that throws
`PluginEnvironmentTaskError`. Until an environment plugin does that,
behavior is the same as today.
- The run retry budget is 10 minutes. If a Runner update takes longer,
the task goes to board recovery with the Runner notice.
- A remote Runner run without a managed AI connection reaches the Runner
check only after native dispatch. That configuration is not supported,
so this change does not handle it. Such a run keeps the native session
retry (3 attempts, 30 s apart).
- #19172 edits the same precheck call in `agent-environment-test.ts`.
Whichever PR merges second needs a small rebase. Keep the mismatch check
before its generic retryable check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, shell and test runs). Reasoning
mode: standard agentic coding.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed 2026-11-05 14:34:23 -08:00
1 parent 4ece2107b8
commit a6ab6a301c
32 files changed
+468 -33

No files matched your search

+21
View File
@@ -67,6 +67,27 @@ authorized project IDs, and optional launch limits. It can also contain a saved
task ID for session reuse and a typed Codex or Claude credential. Core does not
supply an executable command or a guest path.
## Error codes
Core does not read the text of a task error. To report a failure that Core can
act on, throw `PluginEnvironmentTaskError` with a code from
`ENVIRONMENT_TASK_ERROR_CODES`. The SDK sends only the code as structured RPC
error data.
| Code | When to use it | Core behavior |
| --- | --- | --- |
| `runner_revision_mismatch` | The installed Runner does not match the revision that the plugin launches. Throw it from `describe` or `submit` before you prepare or start a task. | A run that has not started provider work is retried for up to 10 minutes. Agent verification checks again every minute. The task is not blocked during this wait. |
```ts
import { PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
if (installed.revision !== expectedRevision) {
throw new PluginEnvironmentTaskError("runner_revision_mismatch");
}
```
Any other error stays an ordinary setup failure. The source refusal diagnostic above is separate: it uses its own envelope and does not authorize a retry.
## Launch preparation
A provider can return `launchPreparationVersion: 1` from `describe`.
@@ -33,3 +33,48 @@ export function readEnvironmentTaskErrorDiagnostic(error: unknown): PluginEnviro
const diagnostic = field(data, "diagnostic");
return sanitize(field(diagnostic, "httpStatus"), field(diagnostic, "errorCode"), field(diagnostic, "reason"));
}
export const ENVIRONMENT_TASK_ERROR_CODES = ["runner_revision_mismatch"] as const;
export type PluginEnvironmentTaskErrorCode = typeof ENVIRONMENT_TASK_ERROR_CODES[number];
const CODE_SCHEMA = "paperclip/environment-task-error-code/v1";
function ownField(value: unknown, name: string): unknown {
if (!value || typeof value !== "object") return undefined;
try { return Object.getOwnPropertyDescriptor(value, name)?.value; }
catch { return undefined; }
}
function validateCode(value: unknown): PluginEnvironmentTaskErrorCode | null {
return ENVIRONMENT_TASK_ERROR_CODES.includes(value as PluginEnvironmentTaskErrorCode)
? value as PluginEnvironmentTaskErrorCode : null;
}
export class PluginEnvironmentTaskError extends Error {
readonly errorCode: PluginEnvironmentTaskErrorCode;
constructor(errorCode: PluginEnvironmentTaskErrorCode) {
const safe = validateCode(errorCode);
if (!safe) throw new TypeError("Unknown environment task error code");
super("The installed Runner does not match the expected revision");
this.name = "PluginEnvironmentTaskError";
this.errorCode = safe;
}
}
export function environmentTaskErrorCodeData(error: unknown): { schema: string; errorCode: PluginEnvironmentTaskErrorCode } | undefined {
if (!(error instanceof PluginEnvironmentTaskError)) return undefined;
const errorCode = validateCode(ownField(error, "errorCode"));
return errorCode ? { schema: CODE_SCHEMA, errorCode } : undefined;
}
export function readEnvironmentTaskErrorCode(error: unknown): PluginEnvironmentTaskErrorCode | null {
for (let current = error, depth = 0; current && depth < 5; current = ownField(current, "cause"), depth++) {
const data = ownField(current, "data");
if (ownField(data, "schema") === CODE_SCHEMA) {
const code = validateCode(ownField(data, "errorCode"));
if (code) return code;
}
}
return null;
}
+2
View File
@@ -464,6 +464,8 @@ export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readE
export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js";
export { PluginEnvironmentRunnerIngressError, environmentRunnerIngressErrorData, readEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js";
export type { PluginEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js";
export { ENVIRONMENT_TASK_ERROR_CODES, PluginEnvironmentTaskError, environmentTaskErrorCodeData, readEnvironmentTaskErrorCode } from "./environment-task-error.js";
export type { PluginEnvironmentTaskErrorCode } from "./environment-task-error.js";
export { PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "./environment-stop-error.js";
export type { PluginEnvironmentStopFailure, PluginEnvironmentStopFailureReason } from "./environment-stop-error.js";
export { environmentResumeErrorData, readEnvironmentResumeErrorDiagnostic } from "./environment-resume-error.js";
+3 -2
View File
@@ -9,7 +9,7 @@ import { environmentSyncErrorData, withEnvironmentSyncErrorCapture } from "./env
import { environmentRunnerIngressErrorData } from "./environment-runner-ingress-error.js";
import { environmentStopErrorData } from "./environment-stop-error.js";
import { environmentResumeErrorData } from "./environment-resume-error.js";
import { environmentTaskErrorData } from "./environment-task-error.js";
import { environmentTaskErrorCodeData, environmentTaskErrorData } from "./environment-task-error.js";
import { createInvocationTracer } from "./invocation-tracer.js";
import { createPluginIdleDrain } from "./idle-drain.js";
/**
@@ -1654,7 +1654,8 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
: method === "environmentSyncOut" ? environmentSyncErrorData(err)
: method === "environmentRunnerIngressEndpoint" ? environmentRunnerIngressErrorData(err)
: method === "environmentStopLease" ? environmentStopErrorData(err)
: method === "environmentResumeLease" ? environmentResumeErrorData(err) : taskError));
: method === "environmentResumeLease" ? environmentResumeErrorData(err)
: method === "environmentTask" ? taskError ?? environmentTaskErrorCodeData(err) : undefined));
} finally {
done?.();
}
@@ -0,0 +1,69 @@
import { createInterface } from "node:readline";
import { PassThrough } from "node:stream";
import { describe, expect, it } from "vitest";
import { definePlugin } from "../src/define-plugin.js";
import { environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode } from "../src/environment-task-error.js";
import { createRequest, isJsonRpcResponse, JsonRpcCallError, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js";
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
async function invoke(method: string, error: unknown) {
const stdin = new PassThrough();
const stdout = new PassThrough();
const lines = createInterface({ input: stdout });
const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({
async setup() {},
async onEnvironmentTask() { throw error; },
async onEnvironmentProbe() { throw error; },
}) });
try {
const response = new Promise<JsonRpcResponse>((resolve) => {
lines.on("line", (line) => {
const message = parseMessage(line);
if (isJsonRpcResponse(message)) resolve(message);
});
});
stdin.write(serializeMessage(createRequest(method, { taskId: "task-1", operation: { kind: "describe" } }, "task-test")));
return await response;
} finally {
worker.stop(); lines.close(); stdin.destroy(); stdout.destroy();
}
}
describe("environment task error codes", () => {
it("does not read the source refusal diagnostic envelope as a code", () => {
expect(readEnvironmentTaskErrorCode({ data: { schema: "paperclip/environment-task-error/v1", errorCode: "runner_revision_mismatch" } })).toBeNull();
});
it("carries runner_revision_mismatch across the real environmentTask RPC", async () => {
const response = await invoke("environmentTask", new PluginEnvironmentTaskError("runner_revision_mismatch"));
if (!("error" in response) || !response.error) throw new Error("Expected RPC error");
expect(response.error.data).toEqual({ schema: "paperclip/environment-task-error-code/v1", errorCode: "runner_revision_mismatch" });
const callError = new JsonRpcCallError(response.error);
expect(callError.message).toBe("The installed Runner does not match the expected revision");
expect(readEnvironmentTaskErrorCode(callError)).toBe("runner_revision_mismatch");
expect(readEnvironmentTaskErrorCode(new Error("wrapped", { cause: new Error("again", { cause: callError }) })))
.toBe("runner_revision_mismatch");
});
it("never derives a code from message text", async () => {
const response = await invoke("environmentTask", new Error("The installed Runner does not match the expected revision"));
expect("error" in response && response.error).not.toHaveProperty("data");
if (!("error" in response) || !response.error) throw new Error("Expected RPC error");
expect(readEnvironmentTaskErrorCode(new JsonRpcCallError(response.error))).toBeNull();
});
it("only attaches the envelope to environmentTask calls", async () => {
const response = await invoke("environmentProbe", new PluginEnvironmentTaskError("runner_revision_mismatch"));
expect("error" in response && response.error).not.toHaveProperty("data");
});
it("refuses codes outside the closed set", () => {
expect(() => new PluginEnvironmentTaskError("private" as never)).toThrow(TypeError);
expect(environmentTaskErrorCodeData(Object.assign(new Error("forged"), { errorCode: "runner_revision_mismatch" }))).toBeUndefined();
});
it.each([null, {}, { data: { schema: "other", errorCode: "runner_revision_mismatch" } },
{ data: { schema: "paperclip/environment-task-error-code/v1", errorCode: "private" } }])(
"rejects malformed envelopes %j", (value) => {
expect(readEnvironmentTaskErrorCode(value)).toBeNull();
});
});
@@ -65,3 +65,16 @@ describe("remote harness verification recovery", () => {
expect(() => assertHarnessTestPassed(recovered)).not.toThrow();
});
});
describe("Runner mismatch during verification", () => {
const context = { stage: "host" as const, phase: "verifying" as const, verificationFailures: 100 };
it("waits for the Runner update instead of parking verification", async () => {
const { runnerRevisionMismatchTestResult } = await import("../services/runner-revision-mismatch.js");
let error: unknown;
try { assertHarnessTestPassed(runnerRevisionMismatchTestResult("paperclip_runner")); } catch (caught) { error = caught; }
expect(error).toMatchObject({ code: "runner_revision_mismatch" });
expect(lifecycleVerificationNeedsRetry(context, error)).toBe(false);
expect(lifecycleFailureMessage(context, error)).toContain("will check again automatically");
expect(lifecycleFailureMessage(context, error)).not.toContain("Automatic verification is paused");
});
});
@@ -1,7 +1,8 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { PgDialect } from "drizzle-orm/pg-core";
import { executionWorkspaces, environmentLeases, type Db } from "@paperclipai/db";
import { environmentTaskOperationSchema, parseEnvironmentTaskResult, JsonRpcCallError, readEnvironmentTaskErrorDiagnostic } from "@paperclipai/plugin-sdk";
import { environmentTaskErrorCodeData, environmentTaskOperationSchema, parseEnvironmentTaskResult, JsonRpcCallError, PluginEnvironmentTaskError,
readEnvironmentTaskErrorCode, readEnvironmentTaskErrorDiagnostic } from "@paperclipai/plugin-sdk";
import { executeEnvironmentTask, settleDispatchedEnvironmentTask } from "../services/environment-task-runtime.js";
import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js";
@@ -512,6 +513,18 @@ describe("plugin-provided Runner execution", () => {
expect(error.message).toBe("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
expect(readEnvironmentTaskErrorDiagnostic(error)).toBeNull();
});
it("keeps only the plugin's stable task error code through worker error sanitization", async () => {
const workers = worker();
workers.call.mockRejectedValue(Object.assign(new JsonRpcCallError({ code: -32002, message: "private-credential",
data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) }), { extra: "private-credential" }));
const error = await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit }).catch(caught => caught);
expect(error.message).toBe("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
expect(readEnvironmentTaskErrorCode(error)).toBe("runner_revision_mismatch");
expect(JSON.stringify({ ...error, message: error.message })).not.toContain("private-credential");
expect(error.cause).toBeUndefined();
workers.call.mockRejectedValue(new Error("The installed Runner does not match the expected revision"));
expect(readEnvironmentTaskErrorCode(await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit }).catch(caught => caught))).toBeNull();
});
it("accepts PRP identity characters and length limits", () => {
for (const field of ["runnerId", "leaseId", "runId", "sessionId", "turnId", "itemId"]) {
for (const value of ["session:part.1", "a".repeat(160)]) {
@@ -133,6 +133,25 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
expect((await current(agent.id)).lifecycleOperation).toMatchObject({ verificationFailures: 3, retryRequired: true });
});
it("keeps checking an outdated remote Runner past the failure budget and becomes ready once it matches", async () => {
const { agent, work, runHost } = await failed("runner_revision_mismatch");
for (let check = 0; check < 5; check++) {
const waiting = await current(agent.id);
expect(waiting).toMatchObject({ lifecycleState: "verifying", status: "paused" });
expect(waiting.lifecycleOperation?.retryRequired).toBeUndefined();
expect(waiting.lifecycleOperation?.verificationFailures).toBeUndefined();
expect(waiting.lifecycleOperation?.retryAt).toEqual(expect.any(String));
expect(waiting.lifecycleError).toContain("Runner is being updated");
expect(waiting.lifecycleError).not.toContain("Automatic verification is paused");
await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
await work.process(agent.id);
}
runHost.mockImplementation(async () => "complete");
await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
await work.process(agent.id);
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle", lifecycleError: null });
});
it("recovers from a generic failed test before the automatic retry budget is exhausted", async () => {
const { agent, work, runHost } = await failed("harness_probe_timed_out");
runHost.mockRejectedValueOnce(Object.assign(new Error("temporary test failure"), { code: "harness_test_failed" }));
@@ -9,6 +9,7 @@ const reasons = {
harness_cwd_invalid: "The working directory is unavailable. Check the working directory in Configuration, then retry.",
harness_probe_timed_out: "The execution test timed out. Paperclip will retry automatically. If this continues, run the agent test in Configuration.",
harness_probe_unexpected_output: "The execution test did not return the expected response. Run the agent test in Configuration to inspect the failed checks.",
runner_revision_mismatch: "The remote Runner is being updated to match this Paperclip version. Paperclip will check again automatically.",
test_environment_unavailable: "The test environment is unavailable. Check the agent's environment in Configuration. Paperclip will retry automatically.",
required_plugin_unavailable: "A required plugin is unavailable. Check the plugin's status in Settings. Paperclip will retry automatically.",
invalid_lifecycle_result: "A required plugin returned an invalid result. Check the plugin's status in Settings. Paperclip will retry automatically.",
@@ -16,6 +16,13 @@ describe("wake context", () => {
expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionKind: "request_confirmation" })).toBe(false);
expect(isResolvedInteractionContinuationWakeContext({ ...context, connectionIntentResolution: undefined })).toBe(false);
});
it("keeps a resolved interaction continuation through a Runner update wait", () => {
const context = { interactionId: "interaction-1", interactionStatus: "answered", mutation: "interaction", retryReason: "runner_revision_wait", wakeReason: "runner_revision_retry" };
expect(isResolvedInteractionContinuationWakeContext(context)).toBe(true);
expect(isResolvedInteractionContinuationWakeContext({ ...context, mutation: undefined })).toBe(false);
expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionStatus: "pending" })).toBe(false);
expect(isResolvedInteractionContinuationWakeContext({ ...context, retryReason: "transient_failure" })).toBe(false);
});
it("requires the subscription-specific non-assignee receipt for a subscription wait", () => {
expect(isNonAssigneeWorkspaceBusyRetry("ai_connection_busy", {
aiConnectionBusyDeferredWhileAssignee: false,
@@ -19,6 +19,7 @@ export const MAX_TURN_CONTINUATION_RETRY_REASON = "max_turns_continuation";
export const WORKSPACE_BUSY_RETRY_REASON = "workspace_busy";
export const AI_CONNECTION_POOL_WAIT_RETRY_REASON = "ai_connection_pool_wait";
export const AI_CONNECTION_BUSY_RETRY_REASON = "ai_connection_busy";
export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait";
export const INTERACTION_CONTINUATION_INFRA_RETRY_REASON = "interaction_continuation_infra_retry";
export const INTERACTION_CONTINUATION_INFRA_WAKE_REASON = "interaction_continuation_infra_retry";
export const WAKE_COMMENT_IDS_KEY = "wakeCommentIds";
@@ -182,6 +183,7 @@ export function isResolvedInteractionContinuationWakeContext(contextSnapshot: un
const retryReason = readNonEmptyString(context.retryReason);
return (
(mutation === "interaction" && wakeReason === "issue_commented") ||
(mutation === "interaction" && retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) ||
wakeReason === INTERACTION_CONTINUATION_INFRA_WAKE_REASON ||
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON
);
+1
View File
@@ -14,6 +14,7 @@ export {
WORKSPACE_BUSY_RETRY_REASON,
AI_CONNECTION_BUSY_RETRY_REASON,
AI_CONNECTION_POOL_WAIT_RETRY_REASON,
RUNNER_REVISION_WAIT_RETRY_REASON,
INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
INTERACTION_CONTINUATION_INFRA_WAKE_REASON,
WAKE_COMMENT_IDS_KEY,
@@ -16,6 +16,7 @@ import { environmentService } from "./environments.js";
import { environmentRuntimeService } from "./environment-runtime.js";
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
import { prepareManagedAiRuntime, assertManagedAiProjectAuth } from "./ai-connection-runtime.js";
import { isRunnerRevisionMismatch, runnerRevisionMismatchTestResult } from "./runner-revision-mismatch.js";
import { aiConnectionService } from "./ai-connections.js";
import { validateAiApiKey } from "./ai-api-key-test.js";
import { probeCopilotExecutionTarget } from "./copilot-connection-probe.js";
@@ -573,7 +574,11 @@ export function agentEnvironmentTestService(db: Db, pluginWorkerManager?: Plugin
return { adapterType, status: "fail" as const, testedAt: new Date().toISOString(), checks: [{ code: error instanceof HttpError ? String(asRecord(error.details)?.code ?? "COPILOT_REQUEST_FAILED") : "COPILOT_REQUEST_FAILED", level: "error" as const, message: error instanceof HttpError ? error.message : "Copilot metadata verification failed." }] };
}
}
await assertManagedAiProjectAuth(context.config, binding.provider, context.executionTarget);
try { await assertManagedAiProjectAuth(context.config, binding.provider, context.executionTarget); }
catch (error) {
if (!isRunnerRevisionMismatch(error)) throw error;
return runnerRevisionMismatchTestResult(adapterType);
}
const target = context.executionTarget;
const result = adapterType === "paperclip_runner" && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask
? await probePluginRunnerEnvironment(context, { identity: managed.identity, accountIdentity: managed.sessionIdentity,
@@ -13,6 +13,7 @@ import { secretService } from "./secrets.js";
import { instanceSettingsService } from "./instance-settings.js";
import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js";
import { museRunnerBroker } from "./muse-runner-broker.js";
import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js";
import { dotRunnerBroker } from "./dot-runner-broker.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
import type { LifecycleAgent } from "../modules/agent-lifecycle/index.js";
@@ -26,7 +27,8 @@ export function assertHarnessTestPassed(result: AdapterEnvironmentTestResult) {
const incomplete = !verified && (result.status === "warn" || result.checks.some(check => check.code.includes("hello_probe")));
if (result.status === "fail" || authenticationFailed || incomplete) {
const has = (suffix: string) => result.checks.some(check => check.code.endsWith(suffix));
const code = result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable"
const code = result.checks.some(check => check.code === RUNNER_REVISION_MISMATCH_CODE) ? RUNNER_REVISION_MISMATCH_CODE
: result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable"
: authenticationFailed ? "harness_auth_failed"
: has("_hello_probe_model_unavailable") || has("_model_invalid") ? "harness_model_unavailable"
: has("_command_unresolvable") ? "harness_command_unavailable"
@@ -10,6 +10,7 @@ import { and, eq, inArray, sql } from "drizzle-orm";
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES,
environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode,
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
import { pluginRegistryService } from "./plugin-registry.js";
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
@@ -461,7 +462,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
data: environmentTaskErrorData({ status: diagnostic.httpStatus, code: diagnostic.errorCode, reason: diagnostic.reason }),
});
// Worker errors and schema diagnostics can contain request credentials.
throw new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
const code = readEnvironmentTaskErrorCode(error);
throw Object.assign(new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"),
code ? { data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError(code)) } : {});
}
}
@@ -1,5 +1,7 @@
import { describe, expect, it } from "vitest";
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
import { accountingForScheduledRetry, executionFailureRetryCount, executionRetryAttemptCount } from "./execution-recovery-attempt.js";
import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js";
import { RUNNER_REVISION_WAIT_RETRY_REASON as DISPATCH_RUNNER_REVISION_WAIT_RETRY_REASON } from "../modules/run-dispatch/index.js";
describe("failure attempts across resource waits", () => {
it("preserves prior failures through subscription waits without trusting unrelated context", () => {
@@ -27,7 +29,7 @@ describe("failure attempts across resource waits", () => {
});
describe("persisted independent accounting", () => {
it.each(["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait"])("%s cannot erase prior infrastructure debits or spend more", scheduledRetryReason => {
it.each(["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait"])("%s cannot erase prior infrastructure debits or spend more", scheduledRetryReason => {
expect(executionFailureRetryCount({ scheduledRetryReason, scheduledRetryAttempt: 20,
contextSnapshot: { executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 1 } },
})).toBe(2);
@@ -43,3 +45,25 @@ describe("persisted independent accounting", () => {
})).toBe(2);
});
});
describe("Runner update waits", () => {
it("uses the same retry reason as run dispatch", () => {
expect(RUNNER_REVISION_WAIT_RETRY_REASON).toBe(DISPATCH_RUNNER_REVISION_WAIT_RETRY_REASON);
});
const ledger = (failureRetries: number) => ({ executionRetryAccounting: { version: 1, failureRetries, maxTurnContinuations: 0 } });
it("never spend the failure retry budget", () => {
const waited = { scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 9, contextSnapshot: ledger(0) };
expect(executionRetryAttemptCount(waited, "runner_revision_wait")).toBe(9);
expect(accountingForScheduledRetry(waited, "runner_revision_wait", 10)).toMatchObject({ failureRetries: 0 });
expect(executionFailureRetryCount(waited)).toBe(0);
expect(executionRetryAttemptCount(waited, "transient_failure")).toBe(0);
});
it("keep failures recorded before the wait and are not shortened by them", () => {
const failed = { scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 1, contextSnapshot: ledger(1) };
expect(executionRetryAttemptCount(failed, "runner_revision_wait")).toBe(0);
const waited = { scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 3,
contextSnapshot: { ...ledger(1), executionRetryAccounting: accountingForScheduledRetry(failed, "runner_revision_wait", 3) } };
expect(executionFailureRetryCount(waited)).toBe(1);
expect(executionRetryAttemptCount(waited, "transient_failure")).toBe(1);
});
});
@@ -1,3 +1,5 @@
import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js";
type RetryRun = {
scheduledRetryAttempt?: number | null;
scheduledRetryReason?: string | null;
@@ -42,7 +44,7 @@ function historicalFailureCount(run: RetryRun): number {
export function executionRetryAccounting(run: RetryRun): ExecutionRetryAccounting {
const saved = savedAccounting(run);
const nonFailureLane = ["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait"].includes(run.scheduledRetryReason ?? "");
const nonFailureLane = ["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", RUNNER_REVISION_WAIT_RETRY_REASON].includes(run.scheduledRetryReason ?? "");
return {
version: 1,
failureRetries: Math.max(saved?.failureRetries ?? 0, saved && nonFailureLane ? 0 : historicalFailureCount(run)),
@@ -57,7 +59,7 @@ export function executionFailureRetryCount(run: RetryRun): number {
}
export function executionRetryAttemptCount(run: RetryRun, reason: string): number {
if (reason === "workspace_busy" || reason === "ai_connection_busy" || reason === "ai_connection_pool_wait") {
if (reason === "workspace_busy" || reason === "ai_connection_busy" || reason === "ai_connection_pool_wait" || reason === RUNNER_REVISION_WAIT_RETRY_REASON) {
return run.scheduledRetryReason === reason ? count(run.scheduledRetryAttempt) ?? 0 : 0;
}
const accounting = executionRetryAccounting(run);
@@ -67,6 +69,7 @@ export function executionRetryAttemptCount(run: RetryRun, reason: string): numbe
export function accountingForScheduledRetry(run: RetryRun, reason: string, attempt: number): ExecutionRetryAccounting {
const accounting = executionRetryAccounting(run);
if (reason === "max_turns_continuation") accounting.maxTurnContinuations = attempt;
else if (reason !== "workspace_busy" && reason !== "ai_connection_busy" && reason !== "ai_connection_pool_wait") accounting.failureRetries = attempt;
else if (reason !== "workspace_busy" && reason !== "ai_connection_busy" && reason !== "ai_connection_pool_wait"
&& reason !== RUNNER_REVISION_WAIT_RETRY_REASON) accounting.failureRetries = attempt;
return accounting;
}
+4 -11
View File
@@ -98,7 +98,7 @@ import {
createHeartbeatRecovery,
} from "./heartbeat/recovery.js";
import {
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS,
failedRunRetryBudgetSpent,
isWorkspaceBusyDeferral,
createHeartbeatRetries,
} from "./heartbeat/retries.js";
@@ -147,6 +147,7 @@ export {
import {
ConfigurationIncompleteFailure,
classifyManagedProjectAuthFailure,
deriveTaskKey,
type WakeupOptions,
mergeCoalescedContextSnapshot,
@@ -289,7 +290,6 @@ import { retryChatControlAdmission } from "./chat-control-admission-retry.js";
import {
prepareManagedAiRuntime,
assertManagedAiProjectAuth,
isAiConnectionIncompatible,
} from "./ai-connection-runtime.js";
import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared";
import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js";
@@ -307,7 +307,6 @@ import {
adapterExecutionControls,
createAdapterExecutionControl,
} from "./adapter-execution-control.js";
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
import { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status-payload.js";
export { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status-payload.js";
import { StaleExecutionContinuationError } from "./execution-continuation.js";
@@ -1002,9 +1001,7 @@ export function heartbeatService(
// Mirrors scheduleBoundedRetryForRun's transient budget check: a failed
// or interrupted run that has already consumed every bounded transient
// attempt cannot be retried again through this lane.
transientRetryBudgetSpent: (run) =>
executionFailureRetryCount(run) >=
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS,
transientRetryBudgetSpent: failedRunRetryBudgetSpent,
});
const runDispatch = createRunDispatch(db);
const {
@@ -2825,11 +2822,7 @@ export function heartbeatService(
});
if (managedAiRuntime && aiBinding) {
try { await assertManagedAiProjectAuth({ ...resolvedConfig, cwd: executionWorkspace.cwd }, aiBinding.provider, executionTarget); }
catch (error) {
// Only a found conflict is the board's to fix; an unfinished check stays an ordinary failure.
if (!isAiConnectionIncompatible(error)) throw error;
throw new ConfigurationIncompleteFailure(error.message, { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } });
}
catch (error) { throw classifyManagedProjectAuthFailure(error, agent.id); }
}
const remoteExecution = realizationResult.remoteExecution;
if (nativeChatWorkspaceScope && materializeNativeChatWorktree) {
+34 -1
View File
@@ -6,7 +6,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest
import { writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils";
import { HttpError } from "../../errors.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { createRunDispatch, type PostCommitEffect } from "../../modules/run-dispatch/index.js";
import { createRunDispatch, isResolvedInteractionContinuationWakeContext, type PostCommitEffect } from "../../modules/run-dispatch/index.js";
import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js";
import * as legacy from "../heartbeat.js";
import { createHeartbeatRunState } from "./run-state.js";
@@ -112,6 +112,22 @@ describe("heartbeat retry module callbacks", () => {
expect(database.access).not.toHaveBeenCalled();
});
it("gives every caller the Runner wait budget for a Runner mismatch and reports its exhaustion", async () => {
const database = guardedDatabase();
const deps = callbacks();
const run = { ...unitRun(), status: "failed", errorCode: "runner_revision_mismatch", scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 10,
contextSnapshot: { issueId: "issue", executionRetryAccounting: { version: 1, failureRetries: 0, maxTurnContinuations: 0 } } } as Run;
const retry = await extracted.createHeartbeatRetries(database.db, deps).scheduleBoundedRetryForRun(run, { adapterType: "codex_local" } as Agent);
expect(retry).toMatchObject({ outcome: "retry_exhausted", attempt: 11, maxAttempts: 10 });
expect(deps.appendRunEvent).toHaveBeenCalledWith(run, expect.objectContaining({
retryExhaustion: { retryReason: "runner_revision_wait", scheduledRetryAttempt: 10, maxAttempts: 10 },
}));
expect(extracted.failedRunRetryBudgetSpent(run)).toBe(true);
expect(extracted.failedRunRetryBudgetSpent({ ...run, scheduledRetryAttempt: 9 })).toBe(false);
expect(extracted.failedRunRetryBudgetSpent({ ...run, errorCode: "setup_failed" })).toBe(false);
expect(database.access).not.toHaveBeenCalled();
});
it("leaves successor work to the committed case handoff without retry escalation", async () => {
const database = guardedDatabase();
const deps = callbacks();
@@ -273,6 +289,23 @@ describePostgres("heartbeat retry module database wiring", () => {
}
});
it("waits for a Runner update without spending failure retries and keeps an answered interaction", async () => {
const { run, agent } = await fixture();
const snapshot = { ...run.contextSnapshot, interactionId: "interaction-1", interactionStatus: "answered", interactionKind: "ask_user_questions",
mutation: "interaction", wakeReason: "issue_commented", executionRetryAccounting: { version: 1, failureRetries: 1, maxTurnContinuations: 0 } };
const [failed] = await db.update(heartbeatRuns).set({ errorCode: "runner_revision_mismatch", contextSnapshot: snapshot })
.where(eq(heartbeatRuns.id, run.id)).returning();
const result = await boundRetries().retries.scheduleBoundedRetryForRun(failed!, agent, { now });
expect(result.outcome).toBe("scheduled");
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id));
expect(successor).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 1 });
expect(successor!.contextSnapshot).toMatchObject({ mutation: "interaction", interactionId: "interaction-1",
executionRetryAccounting: { failureRetries: 1 } });
expect(isResolvedInteractionContinuationWakeContext(successor!.contextSnapshot)).toBe(true);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, successor!.wakeupRequestId!));
expect(wake!.payload).toMatchObject({ mutation: "interaction", interactionId: "interaction-1", retryReason: "runner_revision_wait" });
});
it("resolves current skills on a retry while retaining the failed attempt snapshot", async () => {
const {run,agent}=await fixture();
const snapshot={...run.contextSnapshot,skillVersions:{"company/review":"v1"},skillSelectionSnapshot:true,shadowedSkills:[{key:"old",shadowedBy:"company/review"}]};
+17 -4
View File
@@ -24,6 +24,7 @@ import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-sched
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
import { RUNNER_REVISION_MISMATCH_CODE, RUNNER_REVISION_MISMATCH_RETRY, RUNNER_REVISION_WAIT_RETRY_REASON } from "../runner-revision-mismatch.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
import { randomUUID } from "node:crypto";
import {
@@ -136,12 +137,19 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated
|| code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE;
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE
|| code === RUNNER_REVISION_MISMATCH_CODE;
}
export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length;
export function failedRunRetryBudgetSpent(run: Parameters<typeof executionFailureRetryCount>[0] & { errorCode?: string | null }): boolean {
return run.errorCode === RUNNER_REVISION_MISMATCH_CODE
? executionRetryAttemptCount(run, RUNNER_REVISION_WAIT_RETRY_REASON) >= RUNNER_REVISION_MISMATCH_RETRY.maxAttempts
: executionFailureRetryCount(run) >= BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS;
}
const INTERACTION_CONTINUATION_INFRA_MAX_ATTEMPTS = 2;
const MAX_TURN_CONTINUATION_LIVE_RUN_STATUSES = [
@@ -343,7 +351,8 @@ function isRetryableInteractionContinuationInfrastructureFailure(
run.errorCode === WORKSPACE_VALIDATION_FAILURE_CODE ||
run.errorCode === "process_lost" ||
run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE ||
run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE ||
run.errorCode === RUNNER_REVISION_MISMATCH_CODE
) {
return true;
}
@@ -419,6 +428,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
delayMs?: number;
},
) {
if (!opts?.retryReason && run.errorCode === RUNNER_REVISION_MISMATCH_CODE) opts = { ...RUNNER_REVISION_MISMATCH_RETRY, ...opts };
if (await hasCaseHandoffForRun(run)) return { outcome: "not_scheduled" as const,
reason: "The durable case handoff owns successor work.", issueId: readNonEmptyString(run.contextSnapshot?.issueId) };
if (["openai_dot", "muse"].includes(String(parseObject(agent.adapterConfig).provider))
@@ -615,8 +625,10 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
}
const taskKey = deriveTaskKeyWithHeartbeatFallback(contextSnapshot, null);
const sessionBefore = await resolveSessionBeforeForWakeup(agent, taskKey);
const interactionRunnerWait = retryReason === RUNNER_REVISION_WAIT_RETRY_REASON
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot);
const interactionContinuationPayload =
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON || interactionRunnerWait
? {
mutation: "interaction",
interactionId: readNonEmptyString(contextSnapshot.interactionId),
@@ -646,6 +658,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
retryOfRunId: run.id,
wakeReason,
retryReason,
...(interactionRunnerWait ? { mutation: "interaction" } : {}),
...(retryReason === WORKSPACE_BUSY_RETRY_REASON
? {
failureRetriesBeforeWorkspaceWait:
@@ -1001,7 +1014,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
const scheduledRunId = randomUUID();
if (contextSnapshot.explicitUserContinuation) {
const continuation = issueId && retryReason === "transient_failure" ? await admitExplicitContinuationRetry({
const continuation = issueId && (retryReason === "transient_failure" || retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) ? await admitExplicitContinuationRetry({
db: tx as unknown as Db, companyId: run.companyId, issueId, agentId: run.agentId,
parentRunId: run.id, successorRunId: scheduledRunId, now,
}) : null;
@@ -30,8 +30,11 @@ import { RemoteWorkspaceOperationError, RemoteWorkspaceOperationTimeoutError, Re
import { EnvironmentRunError } from "../environment-run-orchestrator.js";
import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
import { PluginWorkerUnavailableError } from "../plugin-worker-unavailable-error.js";
import { environmentTaskErrorCodeData, JsonRpcCallError, PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
import { RUNNER_REVISION_MISMATCH_RETRY } from "../runner-revision-mismatch.js";
import { unprocessable } from "../../errors.js";
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js";
import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure } from "./run-preparation.js";
import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure, classifyManagedProjectAuthFailure } from "./run-preparation.js";
import { createHeartbeatRunState, getAdapterSessionCodec } from "./run-state.js";
import { buildEffectiveRunSessionConfigMetadata } from "./workspaces.js";
import { createHeartbeatRunCompletion, type CompleteHeartbeatRunInput, type FailHeartbeatRunInput, type HeartbeatRunCompletionDependencies } from "./run-completion.js";
@@ -966,6 +969,61 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
});
const runnerMismatch = () => new JsonRpcCallError({ code: -32002, message: "The installed Runner does not match the expected revision",
data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) });
it.each(["fresh", "wrapped", "started", "untrusted", "retained"] as const)("waits for the Runner update after a %s Runner mismatch", async scenario => {
const f = await fixture({ runtimeMode: scenario === "retained" ? "native" : "legacy" });
const refusal = runnerMismatch();
const error = scenario === "untrusted" ? new Error(refusal.message)
: scenario === "wrapped" ? new Error("Plugin Runner task failed", { cause: refusal })
: classifyManagedProjectAuthFailure(refusal, f.agent.id);
f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
await f.completion.failRunSetup(f.setup(error, scenario === "started"));
const failed = await read(f.run.id);
const retried = scenario === "fresh" || scenario === "wrapped";
expect(failed).toMatchObject({ status: "failed", errorCode: retried ? "runner_revision_mismatch" : "setup_failed" });
expect(failed.resultJson).not.toHaveProperty("configurationIncomplete");
if (retried) {
expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent, RUNNER_REVISION_MISMATCH_RETRY);
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
});
it("waits for the Runner update on an answered interaction instead of its two-attempt continuation retry", async () => {
const f = await fixture({ contextSnapshot: { interactionId: "interaction-1", interactionStatus: "answered", interactionKind: "ask_user_questions",
mutation: "interaction", wakeReason: "issue_commented" } });
f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
await f.completion.failRunSetup(f.setup(runnerMismatch()));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "runner_revision_mismatch" });
expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent, RUNNER_REVISION_MISMATCH_RETRY);
expect(f.deps.scheduleInteractionContinuationInfrastructureRetryIfEligible).not.toHaveBeenCalled();
});
it("keeps a found project authentication conflict a configuration gap that blocks for the board", async () => {
const f = await fixture();
const conflict = unprocessable("This agent's managed AI connection conflicts with project authentication settings in .claude/settings.json.",
{ code: "ai_connection_incompatible" });
await f.completion.failRunSetup(f.setup(classifyManagedProjectAuthFailure(conflict, f.agent.id)));
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "configuration_incomplete",
resultJson: { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${f.agent.id}/runtime` } } });
expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
});
it.each([
["an unavailable Runner identity", new Error("Runner identity unavailable")],
["an unreachable execution host", Object.assign(new Error("connect ECONNREFUSED"), { code: "ECONNREFUSED" })],
["another 422", unprocessable("Select a saved AI account", { code: "ai_connection_busy" })],
])("does not report %s as a project authentication conflict", async (_name, error) => {
const f = await fixture();
const classified = classifyManagedProjectAuthFailure(error, f.agent.id);
expect(classified).toBe(error);
await f.completion.failRunSetup(f.setup(classified));
const failed = await read(f.run.id);
expect(failed.errorCode).not.toBe("configuration_incomplete");
expect(failed.resultJson).not.toHaveProperty("configurationIncomplete");
});
it("routes a lease acquisition refused by a stopped provider worker through bounded bootstrap recovery", async () => {
const f = await fixture();
const refusal = new PluginWorkerUnavailableError({ pluginId: "plugin-1", workerStatus: null,
@@ -13,6 +13,7 @@ import { RemoteWorkspacePreparationTimeoutError, REMOTE_WORKSPACE_RESTORE_FAILED
import { escalateRemoteWorkspaceRestoreFailure } from "../remote-workspace-restore-recovery.js";
import { findReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
import { findPluginWorkerUnavailableError, PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
import { isRunnerRevisionMismatch, RUNNER_REVISION_MISMATCH_CODE, RUNNER_REVISION_MISMATCH_RETRY } from "../runner-revision-mismatch.js";
import { type RunSessionOutcome, resolveNextSessionState, normalizeUsageTotals, resolveCacheAdjustedCostUsd, resolveLedgerBiller, resolveLedgerCostStatus, normalizeLedgerBillingType } from "./run-state.js";
import { readGitConnectionFailure } from "../git-connection-failure.js";
import { CONFIGURATION_INCOMPLETE_FAILURE_CODE, ConfigurationIncompleteFailure } from "./run-preparation.js";
@@ -1376,12 +1377,15 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
// bootstrap budget; the worker is restarted or replaced meanwhile.
const pluginWorkerUnavailableFailure = !providerDispatchStarted && run.runtimeMode === "legacy"
&& run.nativeSessionId === null && run.runnerInstanceId === null ? findPluginWorkerUnavailableError(outerErr) : null;
const runnerRevisionMismatch = !providerDispatchStarted && run.runtimeMode === "legacy"
&& run.nativeSessionId === null && run.runnerInstanceId === null && isRunnerRevisionMismatch(outerErr);
const setupFailureErrorCode =
(controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
remoteWorkspacePreparationTimeout?.code ??
reusableSandboxResumeFailure?.code ??
remoteWorkspaceRestoreFailure?.code ??
pluginWorkerUnavailableFailure?.code ??
(runnerRevisionMismatch ? RUNNER_REVISION_MISMATCH_CODE : null) ??
workspaceGitScanFailure?.code ??
workspaceValidationSetupFailure?.code ??
configurationIncompleteSetupFailure?.code ??
@@ -1544,7 +1548,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
const interactionOwnsRetry = livenessRun.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
&& isResolvedInteractionContinuationWakeContext(livenessRun.contextSnapshot);
await (isTransientWorkspacePreparationCode(livenessRun.errorCode) && !interactionOwnsRetry
? scheduleBoundedRetryForRun(livenessRun, failedAgent)
? livenessRun.errorCode === RUNNER_REVISION_MISMATCH_CODE
? scheduleBoundedRetryForRun(livenessRun, failedAgent, RUNNER_REVISION_MISMATCH_RETRY)
: scheduleBoundedRetryForRun(livenessRun, failedAgent)
: scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent)
).catch((retryError) => {
logger.warn(
@@ -69,6 +69,7 @@ import {
toolProfiles,
} from "@paperclipai/db";
import { HttpError } from "../../errors.js";
import { isAiConnectionIncompatible } from "../ai-connection-runtime.js";
import type { ReviewedChatBindingDenialReason } from "../reviewed-chat-binding-diagnostics.js";
import { filterResolvedGitHubConnectionsForRun } from "../git-credentials.js";
import type {
@@ -162,6 +163,13 @@ export class ConfigurationIncompleteFailure extends Error {
}
}
export function classifyManagedProjectAuthFailure(error: unknown, agentId: string): unknown {
if (!isAiConnectionIncompatible(error)) return error;
return new ConfigurationIncompleteFailure(error.message, {
configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agentId}/runtime` },
});
}
const ISSUE_RESPONSIBLE_USER_WAKE_REASONS = new Set([
"issue_assigned",
"issue_checked_out",
@@ -10,7 +10,7 @@ const stopped = {
},
};
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable", "runner_revision_mismatch"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
const run = { runtimeMode: "legacy", status: "failed", errorCode, scheduledRetryAttempt: 2,
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
expect(legacyExecutionNeedsReconciliation(run)).toBe(false);
@@ -9,6 +9,7 @@ import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wai
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "./plugin-worker-unavailable-error.js";
import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js";
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js";
import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js";
@@ -79,7 +80,8 @@ export function legacyExecutionNeedsReconciliation(
// that the bootstrap evidence proves never started. Keep unknown outcomes held.
if ((run.errorCode === "workspace_git_scan_timeout" || run.errorCode === "workspace_git_scan_saturated"
|| run.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|| run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE) &&
|| run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
|| run.errorCode === RUNNER_REVISION_MISMATCH_CODE) &&
evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false) return false;
if (executionFailureRetryCount(run) >= 2) return true;
return !(
@@ -0,0 +1,26 @@
import { describe, expect, it } from "vitest";
import { environmentTaskErrorCodeData, JsonRpcCallError, PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
import { probePluginRunnerEnvironment } from "./plugin-runner-probe.js";
import { assertHarnessTestPassed } from "../agent-harness-verification.js";
const probe = (runnerTask: () => Promise<unknown>) => probePluginRunnerEnvironment({
companyId: "fixture-company", adapterType: "paperclip_runner", config: { provider: "codex", model: "fixture-model" },
executionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "probe-lease",
runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session" }, runnerTask },
} as Parameters<typeof probePluginRunnerEnvironment>[0]);
describe("plugin Runner probe with an outdated remote Runner", () => {
it("reports a retryable Runner mismatch from the plugin's stable code", async () => {
const refusal = new JsonRpcCallError({ code: -32002, message: "The installed Runner does not match the expected revision",
data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) });
const result = await probe(async () => { throw refusal; });
expect(result).toMatchObject({ status: "fail", checks: [{ code: "runner_revision_mismatch", level: "error", retryable: true }] });
expect(() => assertHarnessTestPassed(result)).toThrow(expect.objectContaining({ code: "runner_revision_mismatch" }));
});
it("keeps other plugin failures a non-retryable failed probe, even with the same message", async () => {
const result = await probe(async () => { throw new Error("The installed Runner does not match the expected revision"); });
expect(result).toMatchObject({ status: "fail", checks: [{ code: "runner_hello_probe_failed", level: "error" }] });
expect(result.checks[0]).not.toHaveProperty("retryable");
});
});
@@ -11,6 +11,7 @@ import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js";
import { createPluginRunnerTaskOwner } from "./plugin-runner-task.js";
import { resolvePaperclipRunnerProviderProfile } from "./provider-profile.js";
import { resolveAcpxQualification } from "./acpx-qualification.js";
import { isRunnerRevisionMismatch, runnerRevisionMismatchTestResult } from "../runner-revision-mismatch.js";
/** Core verifies a real Runner; providers only implement their task API. */
export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTestContext,
@@ -70,6 +71,7 @@ export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTe
checks: [{ code: "runner_hello_probe_passed", level: "info", message: `Runner verified ${result.providerDriver} with model ${result.effectiveModel}.` }] };
} catch (error) {
if (error instanceof NativeEnvironmentProbeCleanupError) throw error;
if (isRunnerRevisionMismatch(error)) return runnerRevisionMismatchTestResult("paperclip_runner", testedAt);
// Provider failures may contain credentials or private addressing.
return { adapterType: "paperclip_runner", status: "fail", testedAt,
checks: [{ code: "runner_hello_probe_failed", level: "error", message: "The selected plugin environment did not complete the Runner hello probe and confirmed cleanup. Check its runtime, account, and model access." }] };
@@ -5,7 +5,7 @@ const run = (errorCode: string | null) =>
({ errorCode } as unknown as Parameters<typeof classifyContinuationFailure>[0]);
describe("pause durability: continuation retry classification", () => {
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not grant %s another recovery budget", (code) => {
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable", "runner_revision_mismatch"])("does not grant %s another recovery budget", (code) => {
expect(classifyContinuationFailure(run(code))).toMatchObject({ kind: "non_retryable", maxAttempts: 0 });
});
it("agent_paused is retryable so work resumes (Option A: Resume Continues Work)", () => {
+2
View File
@@ -10,6 +10,7 @@ import { boundRunIssueId } from "../run-task-binding.js";
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
import { RUNNER_REVISION_MISMATCH_CODE } from "../runner-revision-mismatch.js";
import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js";
import { recoveryRunResultColumn } from "./run-result-read.js";
import { readLegacyDispositionSource } from "./legacy-disposition-read.js";
@@ -561,6 +562,7 @@ const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set<string>([
REUSABLE_SANDBOX_RESUME_FAILED_CODE,
REMOTE_WORKSPACE_RESTORE_FAILED_CODE,
PLUGIN_WORKER_UNAVAILABLE_CODE,
RUNNER_REVISION_MISMATCH_CODE,
"low_trust_isolation_unavailable",
"low_trust_requires_isolated_workspace",
"low_trust_boundary_mismatch",
@@ -321,3 +321,25 @@ it("names the unavailable AI account instead of suggesting secret bindings", ()
expect(notice.body).toContain("Reconnect the account");
expect(notice.body).not.toContain("secret/env");
});
it("names a found project authentication conflict instead of suggesting secret bindings", () => {
const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "ai_connection_incompatible", actionUrl: "/agents/agent-1/runtime" });
expect(notice.title).toBe("AI connection conflicts with project settings");
expect(notice.body).toContain("apiKeyHelper");
expect(notice.nextAction).toContain("choose a different AI connection");
expect(notice.body).not.toContain("secret/env");
});
it("names a remote Runner that was still not updated after its retries", () => {
const notice = buildStrandedRecoveryEscalationNotice({
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }),
recoveryActionId: "runner-recovery",
recoveryOwner: null,
sourceRun: { id: "failed-run", status: "failed", errorCode: "runner_revision_mismatch" },
});
expect(notice.presentation.title).toBe("Remote Runner is out of date");
expect(notice.body).toContain("Runner was still not updated");
expect(notice.body).not.toContain("workspace");
expect(notice.body).not.toContain("authentication");
expect(JSON.stringify(notice.metadata)).toContain("remote Runner received the update");
});
@@ -64,6 +64,16 @@ const WORKSPACE_SCAN_NOTICES: Record<string, { title: string; nextAction: string
workspace_git_scan_cancelled: { title: "Workspace scan was cancelled", nextAction: "Inspect why workspace preparation was cancelled before retrying the task." },
};
const SETUP_FAILURE_NOTICES: Record<string, StrandedRecoveryNoticeSeed> = {
runner_revision_mismatch: {
title: "Remote Runner is out of date",
body: "The remote Runner does not match this Paperclip version, which is expected for a few minutes while a release updates the Runner. "
+ "Paperclip retried before the agent started, but the Runner was still not updated.",
nextAction: "Check that the remote Runner received the update for this release, then retry the task.",
tone: "danger",
},
};
export function buildImmediateExecutionPathRecoveryNoticeSeed(input: {
status: "todo" | "in_progress";
}): StrandedRecoveryNoticeSeed {
@@ -162,6 +172,16 @@ export function buildConfigurationIncompleteRecoveryNoticeSeed(
tone: "danger",
};
}
if (readNonEmptyStringField(configurationIncomplete, "reason") === "ai_connection_incompatible") {
return {
title: "AI connection conflicts with project settings",
body: "Paperclip stopped before the agent started because this agent's managed AI connection conflicts with authentication settings "
+ "in the project or the agent's arguments, such as an API key, token or apiKeyHelper. The task is blocked until the conflict is removed.",
nextAction: "Remove the API key, token or apiKeyHelper from the project's Claude or Codex settings or the agent's arguments, "
+ "or choose a different AI connection for this agent, then retry the task.",
tone: "danger",
};
}
if (readNonEmptyStringField(configurationIncomplete, "reason") === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON) {
const pluginKey = readNonEmptyStringField(configurationIncomplete, "pluginKey") ?? "the sandbox provider plugin";
const pluginStatus = readNonEmptyStringField(configurationIncomplete, "pluginStatus") ?? "not ready";
@@ -229,7 +249,7 @@ export function buildStrandedRecoveryEscalationNotice(input: {
...workspaceScan,
body: `Paperclip could not prepare the workspace before the agent started. Automatic recovery could not continue. ${workspaceScan.nextAction}`,
tone: "danger" as const,
} : input.seed;
} : SETUP_FAILURE_NOTICES[input.sourceRun?.errorCode ?? ""] ?? input.seed;
const fallbackBody = input.fallbackBody?.trim();
const body = seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY);
const title =
@@ -0,0 +1,19 @@
import type { AdapterEnvironmentTestResult } from "@paperclipai/adapter-utils";
import { readEnvironmentTaskErrorCode } from "@paperclipai/plugin-sdk";
export const RUNNER_REVISION_MISMATCH_CODE = "runner_revision_mismatch";
export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait";
export const RUNNER_REVISION_MISMATCH_RETRY = {
retryReason: RUNNER_REVISION_WAIT_RETRY_REASON, wakeReason: "runner_revision_retry", maxAttempts: 10, delayMs: 60_000,
} as const;
export function isRunnerRevisionMismatch(error: unknown): boolean {
return readEnvironmentTaskErrorCode(error) === RUNNER_REVISION_MISMATCH_CODE;
}
export function runnerRevisionMismatchTestResult(adapterType: string, testedAt = new Date().toISOString()): AdapterEnvironmentTestResult {
return { adapterType, status: "fail", testedAt, checks: [{ code: RUNNER_REVISION_MISMATCH_CODE, level: "error", retryable: true,
message: "The remote Runner does not match this Paperclip version yet. Paperclip will check again after the Runner update." }] };
}