mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:12:25 +01:00
fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice
## Linked Issues or Issue Description
Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)
**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.
**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.
**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.
**Paperclip version or commit**
master at 377e98f1a2.
**Deployment mode**
Authenticated deployment with a plugin environment that provides a
remote Runner.
## What Changed
- Plugin SDK: add `PluginEnvironmentTaskError` and
`ENVIRONMENT_TASK_ERROR_CODES` with one code,
`runner_revision_mismatch`. The worker RPC host sends it as structured
`environmentTask` error data with the schema
`paperclip/environment-task-error-code/v1`
(`environmentTaskErrorCodeData`, read with
`readEnvironmentTaskErrorCode`). This uses the same envelope method as
the other environment RPC errors. Core reads only the code, never the
message. The source refusal diagnostic from #19302 keeps its own schema
and API unchanged, and it takes precedence on a source `submit`.
- `executeEnvironmentTask` still replaces each worker error with a
generic error. After the #19302 diagnostic check, it now keeps only that
closed code.
- Run setup: a Runner mismatch before provider dispatch gets the error
code `runner_revision_mismatch`. Setup schedules a bounded retry with
its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s
apart). The reason is in the same resource-wait lane as workspace and AI
connection waits, so a wait does not spend the failure retry budget. The
guards are the same as for `plugin_worker_unavailable`. Recovery and
legacy reconciliation treat the code like the other bootstrap-only
codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and
budget for a mismatch run when a caller gives no reason, and the
recovery sweeper reports exhaustion from that budget
(`failedRunRetryBudgetSpent`). An answered interaction that hits a
mismatch also waits for the Runner, and its successor keeps the resolved
interaction continuation identity.
- Move the managed project auth classification from `heartbeat.ts` to
`classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a
found `ai_connection_incompatible` conflict becomes a configuration
failure. All other errors keep their own code.
- Recovery notices: `ai_connection_incompatible` now has its own notice
instead of the "secret/env bindings" text. An exhausted Runner wait gets
the notice "Remote Runner is out of date".
- Agent verification: the Runner probe and the managed project auth
precheck report a mismatch as a retryable check.
`assertHarnessTestPassed` maps it to the new lifecycle code
`runner_revision_mismatch`. That code never counts toward the
three-failure park. The agent checks again every 60 s and becomes ready
when the Runner matches.
- Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`.
## Verification
- `cd packages/plugins/sdk && npx vitest run
tests/environment-task-error-code.test.ts
tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the
code crosses the real `environmentTask` RPC. A plain error with the same
message gets no code. Other methods get no envelope. The #19302
diagnostic tests still pass.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped
mismatch gets `runner_revision_mismatch` and the runner retry budget,
with no configuration failure. A started, untrusted (message only) or
native run keeps `setup_failed` with no retry. A real
`ai_connection_incompatible` stays `configuration_incomplete` with no
retry. Other errors, such as an unreachable execution host, are not
reported as an authentication conflict.
- `npx vitest run src/services/execution-recovery-attempt.test.ts`:
Runner waits do not change `failureRetries`, and earlier failures do not
shorten the wait.
- `npx vitest run src/services/heartbeat/retries.test.ts
src/modules/run-dispatch` (embedded Postgres): a recovery caller without
options gets the Runner wait budget and its exhaustion. An answered
interaction waits for the Runner, keeps its continuation identity and
does not spend failure retries.
- `npx vitest run src/__tests__/environment-task-runtime.test.ts`:
sanitization keeps only the code. No worker text or cause survives.
- `npx vitest run
src/modules/agent-lifecycle/__tests__/verification-retry.test.ts`
(embedded Postgres): a `verifying` agent keeps checking after five
mismatches. It never gets `retryRequired`. It becomes ready when the
next check passes.
- `npx vitest run
src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts
src/__tests__/agent-harness-verification-remote.test.ts
src/services/recovery/stranded-notice.test.ts
src/services/recovery/service.pause-durability.test.ts
src/services/legacy-execution-recovery.test.ts`.
- `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`.
- One test in `src/__tests__/heartbeat-process-recovery.test.ts`
("redacts opaque environment-bound credentials from Sentry diagnostics")
fails on my machine. It fails the same way on the base commit without
this change.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them.
## Risks
- Low to medium. The change adds one new setup failure code and one
lifecycle code. Other failures keep their current behavior.
- A Runner mismatch must come from a plugin that throws
`PluginEnvironmentTaskError`. Until an environment plugin does that,
behavior is the same as today.
- The run retry budget is 10 minutes. If a Runner update takes longer,
the task goes to board recovery with the Runner notice.
- A remote Runner run without a managed AI connection reaches the Runner
check only after native dispatch. That configuration is not supported,
so this change does not handle it. Such a run keeps the native session
retry (3 attempts, 30 s apart).
- #19172 edits the same precheck call in `agent-environment-test.ts`.
Whichever PR merges second needs a small rebase. Keep the mismatch check
before its generic retryable check.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, shell and test runs). Reasoning
mode: standard agentic coding.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
4ece2107b8
commit
a6ab6a301c
32 files changed
+468
-33
No files matched your search
@@ -67,6 +67,27 @@ authorized project IDs, and optional launch limits. It can also contain a saved
|
||||
task ID for session reuse and a typed Codex or Claude credential. Core does not
|
||||
supply an executable command or a guest path.
|
||||
|
||||
## Error codes
|
||||
|
||||
Core does not read the text of a task error. To report a failure that Core can
|
||||
act on, throw `PluginEnvironmentTaskError` with a code from
|
||||
`ENVIRONMENT_TASK_ERROR_CODES`. The SDK sends only the code as structured RPC
|
||||
error data.
|
||||
|
||||
| Code | When to use it | Core behavior |
|
||||
| --- | --- | --- |
|
||||
| `runner_revision_mismatch` | The installed Runner does not match the revision that the plugin launches. Throw it from `describe` or `submit` before you prepare or start a task. | A run that has not started provider work is retried for up to 10 minutes. Agent verification checks again every minute. The task is not blocked during this wait. |
|
||||
|
||||
```ts
|
||||
import { PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
|
||||
|
||||
if (installed.revision !== expectedRevision) {
|
||||
throw new PluginEnvironmentTaskError("runner_revision_mismatch");
|
||||
}
|
||||
```
|
||||
|
||||
Any other error stays an ordinary setup failure. The source refusal diagnostic above is separate: it uses its own envelope and does not authorize a retry.
|
||||
|
||||
## Launch preparation
|
||||
|
||||
A provider can return `launchPreparationVersion: 1` from `describe`.
|
||||
|
||||
@@ -33,3 +33,48 @@ export function readEnvironmentTaskErrorDiagnostic(error: unknown): PluginEnviro
|
||||
const diagnostic = field(data, "diagnostic");
|
||||
return sanitize(field(diagnostic, "httpStatus"), field(diagnostic, "errorCode"), field(diagnostic, "reason"));
|
||||
}
|
||||
|
||||
export const ENVIRONMENT_TASK_ERROR_CODES = ["runner_revision_mismatch"] as const;
|
||||
export type PluginEnvironmentTaskErrorCode = typeof ENVIRONMENT_TASK_ERROR_CODES[number];
|
||||
|
||||
const CODE_SCHEMA = "paperclip/environment-task-error-code/v1";
|
||||
|
||||
function ownField(value: unknown, name: string): unknown {
|
||||
if (!value || typeof value !== "object") return undefined;
|
||||
try { return Object.getOwnPropertyDescriptor(value, name)?.value; }
|
||||
catch { return undefined; }
|
||||
}
|
||||
|
||||
function validateCode(value: unknown): PluginEnvironmentTaskErrorCode | null {
|
||||
return ENVIRONMENT_TASK_ERROR_CODES.includes(value as PluginEnvironmentTaskErrorCode)
|
||||
? value as PluginEnvironmentTaskErrorCode : null;
|
||||
}
|
||||
|
||||
export class PluginEnvironmentTaskError extends Error {
|
||||
readonly errorCode: PluginEnvironmentTaskErrorCode;
|
||||
|
||||
constructor(errorCode: PluginEnvironmentTaskErrorCode) {
|
||||
const safe = validateCode(errorCode);
|
||||
if (!safe) throw new TypeError("Unknown environment task error code");
|
||||
super("The installed Runner does not match the expected revision");
|
||||
this.name = "PluginEnvironmentTaskError";
|
||||
this.errorCode = safe;
|
||||
}
|
||||
}
|
||||
|
||||
export function environmentTaskErrorCodeData(error: unknown): { schema: string; errorCode: PluginEnvironmentTaskErrorCode } | undefined {
|
||||
if (!(error instanceof PluginEnvironmentTaskError)) return undefined;
|
||||
const errorCode = validateCode(ownField(error, "errorCode"));
|
||||
return errorCode ? { schema: CODE_SCHEMA, errorCode } : undefined;
|
||||
}
|
||||
|
||||
export function readEnvironmentTaskErrorCode(error: unknown): PluginEnvironmentTaskErrorCode | null {
|
||||
for (let current = error, depth = 0; current && depth < 5; current = ownField(current, "cause"), depth++) {
|
||||
const data = ownField(current, "data");
|
||||
if (ownField(data, "schema") === CODE_SCHEMA) {
|
||||
const code = validateCode(ownField(data, "errorCode"));
|
||||
if (code) return code;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -464,6 +464,8 @@ export { preserveEnvironmentSyncErrorDiagnostic, environmentSyncErrorData, readE
|
||||
export type { PluginEnvironmentSyncErrorDiagnostic } from "./environment-sync-error.js";
|
||||
export { PluginEnvironmentRunnerIngressError, environmentRunnerIngressErrorData, readEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js";
|
||||
export type { PluginEnvironmentRunnerIngressFailure } from "./environment-runner-ingress-error.js";
|
||||
export { ENVIRONMENT_TASK_ERROR_CODES, PluginEnvironmentTaskError, environmentTaskErrorCodeData, readEnvironmentTaskErrorCode } from "./environment-task-error.js";
|
||||
export type { PluginEnvironmentTaskErrorCode } from "./environment-task-error.js";
|
||||
export { PluginEnvironmentStopError, environmentStopErrorData, readEnvironmentStopFailure } from "./environment-stop-error.js";
|
||||
export type { PluginEnvironmentStopFailure, PluginEnvironmentStopFailureReason } from "./environment-stop-error.js";
|
||||
export { environmentResumeErrorData, readEnvironmentResumeErrorDiagnostic } from "./environment-resume-error.js";
|
||||
|
||||
@@ -9,7 +9,7 @@ import { environmentSyncErrorData, withEnvironmentSyncErrorCapture } from "./env
|
||||
import { environmentRunnerIngressErrorData } from "./environment-runner-ingress-error.js";
|
||||
import { environmentStopErrorData } from "./environment-stop-error.js";
|
||||
import { environmentResumeErrorData } from "./environment-resume-error.js";
|
||||
import { environmentTaskErrorData } from "./environment-task-error.js";
|
||||
import { environmentTaskErrorCodeData, environmentTaskErrorData } from "./environment-task-error.js";
|
||||
import { createInvocationTracer } from "./invocation-tracer.js";
|
||||
import { createPluginIdleDrain } from "./idle-drain.js";
|
||||
/**
|
||||
@@ -1654,7 +1654,8 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
|
||||
: method === "environmentSyncOut" ? environmentSyncErrorData(err)
|
||||
: method === "environmentRunnerIngressEndpoint" ? environmentRunnerIngressErrorData(err)
|
||||
: method === "environmentStopLease" ? environmentStopErrorData(err)
|
||||
: method === "environmentResumeLease" ? environmentResumeErrorData(err) : taskError));
|
||||
: method === "environmentResumeLease" ? environmentResumeErrorData(err)
|
||||
: method === "environmentTask" ? taskError ?? environmentTaskErrorCodeData(err) : undefined));
|
||||
} finally {
|
||||
done?.();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
import { createInterface } from "node:readline";
|
||||
import { PassThrough } from "node:stream";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { definePlugin } from "../src/define-plugin.js";
|
||||
import { environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode } from "../src/environment-task-error.js";
|
||||
import { createRequest, isJsonRpcResponse, JsonRpcCallError, parseMessage, serializeMessage, type JsonRpcResponse } from "../src/protocol.js";
|
||||
import { startWorkerRpcHost } from "../src/worker-rpc-host.js";
|
||||
|
||||
async function invoke(method: string, error: unknown) {
|
||||
const stdin = new PassThrough();
|
||||
const stdout = new PassThrough();
|
||||
const lines = createInterface({ input: stdout });
|
||||
const worker = startWorkerRpcHost({ stdin, stdout, plugin: definePlugin({
|
||||
async setup() {},
|
||||
async onEnvironmentTask() { throw error; },
|
||||
async onEnvironmentProbe() { throw error; },
|
||||
}) });
|
||||
try {
|
||||
const response = new Promise<JsonRpcResponse>((resolve) => {
|
||||
lines.on("line", (line) => {
|
||||
const message = parseMessage(line);
|
||||
if (isJsonRpcResponse(message)) resolve(message);
|
||||
});
|
||||
});
|
||||
stdin.write(serializeMessage(createRequest(method, { taskId: "task-1", operation: { kind: "describe" } }, "task-test")));
|
||||
return await response;
|
||||
} finally {
|
||||
worker.stop(); lines.close(); stdin.destroy(); stdout.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
describe("environment task error codes", () => {
|
||||
it("does not read the source refusal diagnostic envelope as a code", () => {
|
||||
expect(readEnvironmentTaskErrorCode({ data: { schema: "paperclip/environment-task-error/v1", errorCode: "runner_revision_mismatch" } })).toBeNull();
|
||||
});
|
||||
it("carries runner_revision_mismatch across the real environmentTask RPC", async () => {
|
||||
const response = await invoke("environmentTask", new PluginEnvironmentTaskError("runner_revision_mismatch"));
|
||||
if (!("error" in response) || !response.error) throw new Error("Expected RPC error");
|
||||
expect(response.error.data).toEqual({ schema: "paperclip/environment-task-error-code/v1", errorCode: "runner_revision_mismatch" });
|
||||
const callError = new JsonRpcCallError(response.error);
|
||||
expect(callError.message).toBe("The installed Runner does not match the expected revision");
|
||||
expect(readEnvironmentTaskErrorCode(callError)).toBe("runner_revision_mismatch");
|
||||
expect(readEnvironmentTaskErrorCode(new Error("wrapped", { cause: new Error("again", { cause: callError }) })))
|
||||
.toBe("runner_revision_mismatch");
|
||||
});
|
||||
|
||||
it("never derives a code from message text", async () => {
|
||||
const response = await invoke("environmentTask", new Error("The installed Runner does not match the expected revision"));
|
||||
expect("error" in response && response.error).not.toHaveProperty("data");
|
||||
if (!("error" in response) || !response.error) throw new Error("Expected RPC error");
|
||||
expect(readEnvironmentTaskErrorCode(new JsonRpcCallError(response.error))).toBeNull();
|
||||
});
|
||||
|
||||
it("only attaches the envelope to environmentTask calls", async () => {
|
||||
const response = await invoke("environmentProbe", new PluginEnvironmentTaskError("runner_revision_mismatch"));
|
||||
expect("error" in response && response.error).not.toHaveProperty("data");
|
||||
});
|
||||
|
||||
it("refuses codes outside the closed set", () => {
|
||||
expect(() => new PluginEnvironmentTaskError("private" as never)).toThrow(TypeError);
|
||||
expect(environmentTaskErrorCodeData(Object.assign(new Error("forged"), { errorCode: "runner_revision_mismatch" }))).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each([null, {}, { data: { schema: "other", errorCode: "runner_revision_mismatch" } },
|
||||
{ data: { schema: "paperclip/environment-task-error-code/v1", errorCode: "private" } }])(
|
||||
"rejects malformed envelopes %j", (value) => {
|
||||
expect(readEnvironmentTaskErrorCode(value)).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -65,3 +65,16 @@ describe("remote harness verification recovery", () => {
|
||||
expect(() => assertHarnessTestPassed(recovered)).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("Runner mismatch during verification", () => {
|
||||
const context = { stage: "host" as const, phase: "verifying" as const, verificationFailures: 100 };
|
||||
it("waits for the Runner update instead of parking verification", async () => {
|
||||
const { runnerRevisionMismatchTestResult } = await import("../services/runner-revision-mismatch.js");
|
||||
let error: unknown;
|
||||
try { assertHarnessTestPassed(runnerRevisionMismatchTestResult("paperclip_runner")); } catch (caught) { error = caught; }
|
||||
expect(error).toMatchObject({ code: "runner_revision_mismatch" });
|
||||
expect(lifecycleVerificationNeedsRetry(context, error)).toBe(false);
|
||||
expect(lifecycleFailureMessage(context, error)).toContain("will check again automatically");
|
||||
expect(lifecycleFailureMessage(context, error)).not.toContain("Automatic verification is paused");
|
||||
});
|
||||
});
|
||||
@@ -1,7 +1,8 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { PgDialect } from "drizzle-orm/pg-core";
|
||||
import { executionWorkspaces, environmentLeases, type Db } from "@paperclipai/db";
|
||||
import { environmentTaskOperationSchema, parseEnvironmentTaskResult, JsonRpcCallError, readEnvironmentTaskErrorDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import { environmentTaskErrorCodeData, environmentTaskOperationSchema, parseEnvironmentTaskResult, JsonRpcCallError, PluginEnvironmentTaskError,
|
||||
readEnvironmentTaskErrorCode, readEnvironmentTaskErrorDiagnostic } from "@paperclipai/plugin-sdk";
|
||||
import { executeEnvironmentTask, settleDispatchedEnvironmentTask } from "../services/environment-task-runtime.js";
|
||||
import { NativeExecutionContext } from "../services/native-runtime/native-execution-context.js";
|
||||
|
||||
@@ -512,6 +513,18 @@ describe("plugin-provided Runner execution", () => {
|
||||
expect(error.message).toBe("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
|
||||
expect(readEnvironmentTaskErrorDiagnostic(error)).toBeNull();
|
||||
});
|
||||
it("keeps only the plugin's stable task error code through worker error sanitization", async () => {
|
||||
const workers = worker();
|
||||
workers.call.mockRejectedValue(Object.assign(new JsonRpcCallError({ code: -32002, message: "private-credential",
|
||||
data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) }), { extra: "private-credential" }));
|
||||
const error = await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit }).catch(caught => caught);
|
||||
expect(error.message).toBe("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
|
||||
expect(readEnvironmentTaskErrorCode(error)).toBe("runner_revision_mismatch");
|
||||
expect(JSON.stringify({ ...error, message: error.message })).not.toContain("private-credential");
|
||||
expect(error.cause).toBeUndefined();
|
||||
workers.call.mockRejectedValue(new Error("The installed Runner does not match the expected revision"));
|
||||
expect(readEnvironmentTaskErrorCode(await executeEnvironmentTask(database().db, workers as never, { companyId: "company", leaseId, operation: submit }).catch(caught => caught))).toBeNull();
|
||||
});
|
||||
it("accepts PRP identity characters and length limits", () => {
|
||||
for (const field of ["runnerId", "leaseId", "runId", "sessionId", "turnId", "itemId"]) {
|
||||
for (const value of ["session:part.1", "a".repeat(160)]) {
|
||||
|
||||
@@ -133,6 +133,25 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
expect((await current(agent.id)).lifecycleOperation).toMatchObject({ verificationFailures: 3, retryRequired: true });
|
||||
});
|
||||
|
||||
it("keeps checking an outdated remote Runner past the failure budget and becomes ready once it matches", async () => {
|
||||
const { agent, work, runHost } = await failed("runner_revision_mismatch");
|
||||
for (let check = 0; check < 5; check++) {
|
||||
const waiting = await current(agent.id);
|
||||
expect(waiting).toMatchObject({ lifecycleState: "verifying", status: "paused" });
|
||||
expect(waiting.lifecycleOperation?.retryRequired).toBeUndefined();
|
||||
expect(waiting.lifecycleOperation?.verificationFailures).toBeUndefined();
|
||||
expect(waiting.lifecycleOperation?.retryAt).toEqual(expect.any(String));
|
||||
expect(waiting.lifecycleError).toContain("Runner is being updated");
|
||||
expect(waiting.lifecycleError).not.toContain("Automatic verification is paused");
|
||||
await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
|
||||
await work.process(agent.id);
|
||||
}
|
||||
runHost.mockImplementation(async () => "complete");
|
||||
await db.update(agents).set({ lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt'` }).where(eq(agents.id, agent.id));
|
||||
await work.process(agent.id);
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle", lifecycleError: null });
|
||||
});
|
||||
|
||||
it("recovers from a generic failed test before the automatic retry budget is exhausted", async () => {
|
||||
const { agent, work, runHost } = await failed("harness_probe_timed_out");
|
||||
runHost.mockRejectedValueOnce(Object.assign(new Error("temporary test failure"), { code: "harness_test_failed" }));
|
||||
|
||||
@@ -9,6 +9,7 @@ const reasons = {
|
||||
harness_cwd_invalid: "The working directory is unavailable. Check the working directory in Configuration, then retry.",
|
||||
harness_probe_timed_out: "The execution test timed out. Paperclip will retry automatically. If this continues, run the agent test in Configuration.",
|
||||
harness_probe_unexpected_output: "The execution test did not return the expected response. Run the agent test in Configuration to inspect the failed checks.",
|
||||
runner_revision_mismatch: "The remote Runner is being updated to match this Paperclip version. Paperclip will check again automatically.",
|
||||
test_environment_unavailable: "The test environment is unavailable. Check the agent's environment in Configuration. Paperclip will retry automatically.",
|
||||
required_plugin_unavailable: "A required plugin is unavailable. Check the plugin's status in Settings. Paperclip will retry automatically.",
|
||||
invalid_lifecycle_result: "A required plugin returned an invalid result. Check the plugin's status in Settings. Paperclip will retry automatically.",
|
||||
|
||||
@@ -16,6 +16,13 @@ describe("wake context", () => {
|
||||
expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionKind: "request_confirmation" })).toBe(false);
|
||||
expect(isResolvedInteractionContinuationWakeContext({ ...context, connectionIntentResolution: undefined })).toBe(false);
|
||||
});
|
||||
it("keeps a resolved interaction continuation through a Runner update wait", () => {
|
||||
const context = { interactionId: "interaction-1", interactionStatus: "answered", mutation: "interaction", retryReason: "runner_revision_wait", wakeReason: "runner_revision_retry" };
|
||||
expect(isResolvedInteractionContinuationWakeContext(context)).toBe(true);
|
||||
expect(isResolvedInteractionContinuationWakeContext({ ...context, mutation: undefined })).toBe(false);
|
||||
expect(isResolvedInteractionContinuationWakeContext({ ...context, interactionStatus: "pending" })).toBe(false);
|
||||
expect(isResolvedInteractionContinuationWakeContext({ ...context, retryReason: "transient_failure" })).toBe(false);
|
||||
});
|
||||
it("requires the subscription-specific non-assignee receipt for a subscription wait", () => {
|
||||
expect(isNonAssigneeWorkspaceBusyRetry("ai_connection_busy", {
|
||||
aiConnectionBusyDeferredWhileAssignee: false,
|
||||
|
||||
@@ -19,6 +19,7 @@ export const MAX_TURN_CONTINUATION_RETRY_REASON = "max_turns_continuation";
|
||||
export const WORKSPACE_BUSY_RETRY_REASON = "workspace_busy";
|
||||
export const AI_CONNECTION_POOL_WAIT_RETRY_REASON = "ai_connection_pool_wait";
|
||||
export const AI_CONNECTION_BUSY_RETRY_REASON = "ai_connection_busy";
|
||||
export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait";
|
||||
export const INTERACTION_CONTINUATION_INFRA_RETRY_REASON = "interaction_continuation_infra_retry";
|
||||
export const INTERACTION_CONTINUATION_INFRA_WAKE_REASON = "interaction_continuation_infra_retry";
|
||||
export const WAKE_COMMENT_IDS_KEY = "wakeCommentIds";
|
||||
@@ -182,6 +183,7 @@ export function isResolvedInteractionContinuationWakeContext(contextSnapshot: un
|
||||
const retryReason = readNonEmptyString(context.retryReason);
|
||||
return (
|
||||
(mutation === "interaction" && wakeReason === "issue_commented") ||
|
||||
(mutation === "interaction" && retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) ||
|
||||
wakeReason === INTERACTION_CONTINUATION_INFRA_WAKE_REASON ||
|
||||
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON
|
||||
);
|
||||
|
||||
@@ -14,6 +14,7 @@ export {
|
||||
WORKSPACE_BUSY_RETRY_REASON,
|
||||
AI_CONNECTION_BUSY_RETRY_REASON,
|
||||
AI_CONNECTION_POOL_WAIT_RETRY_REASON,
|
||||
RUNNER_REVISION_WAIT_RETRY_REASON,
|
||||
INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
|
||||
INTERACTION_CONTINUATION_INFRA_WAKE_REASON,
|
||||
WAKE_COMMENT_IDS_KEY,
|
||||
|
||||
@@ -16,6 +16,7 @@ import { environmentService } from "./environments.js";
|
||||
import { environmentRuntimeService } from "./environment-runtime.js";
|
||||
import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
|
||||
import { prepareManagedAiRuntime, assertManagedAiProjectAuth } from "./ai-connection-runtime.js";
|
||||
import { isRunnerRevisionMismatch, runnerRevisionMismatchTestResult } from "./runner-revision-mismatch.js";
|
||||
import { aiConnectionService } from "./ai-connections.js";
|
||||
import { validateAiApiKey } from "./ai-api-key-test.js";
|
||||
import { probeCopilotExecutionTarget } from "./copilot-connection-probe.js";
|
||||
@@ -573,7 +574,11 @@ export function agentEnvironmentTestService(db: Db, pluginWorkerManager?: Plugin
|
||||
return { adapterType, status: "fail" as const, testedAt: new Date().toISOString(), checks: [{ code: error instanceof HttpError ? String(asRecord(error.details)?.code ?? "COPILOT_REQUEST_FAILED") : "COPILOT_REQUEST_FAILED", level: "error" as const, message: error instanceof HttpError ? error.message : "Copilot metadata verification failed." }] };
|
||||
}
|
||||
}
|
||||
await assertManagedAiProjectAuth(context.config, binding.provider, context.executionTarget);
|
||||
try { await assertManagedAiProjectAuth(context.config, binding.provider, context.executionTarget); }
|
||||
catch (error) {
|
||||
if (!isRunnerRevisionMismatch(error)) throw error;
|
||||
return runnerRevisionMismatchTestResult(adapterType);
|
||||
}
|
||||
const target = context.executionTarget;
|
||||
const result = adapterType === "paperclip_runner" && target?.kind === "remote" && target.transport === "sandbox" && target.runnerTask
|
||||
? await probePluginRunnerEnvironment(context, { identity: managed.identity, accountIdentity: managed.sessionIdentity,
|
||||
|
||||
@@ -13,6 +13,7 @@ import { secretService } from "./secrets.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js";
|
||||
import { museRunnerBroker } from "./muse-runner-broker.js";
|
||||
import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js";
|
||||
import { dotRunnerBroker } from "./dot-runner-broker.js";
|
||||
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
import type { LifecycleAgent } from "../modules/agent-lifecycle/index.js";
|
||||
@@ -26,7 +27,8 @@ export function assertHarnessTestPassed(result: AdapterEnvironmentTestResult) {
|
||||
const incomplete = !verified && (result.status === "warn" || result.checks.some(check => check.code.includes("hello_probe")));
|
||||
if (result.status === "fail" || authenticationFailed || incomplete) {
|
||||
const has = (suffix: string) => result.checks.some(check => check.code.endsWith(suffix));
|
||||
const code = result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable"
|
||||
const code = result.checks.some(check => check.code === RUNNER_REVISION_MISMATCH_CODE) ? RUNNER_REVISION_MISMATCH_CODE
|
||||
: result.checks.some(check => check.level === "error" && check.retryable === true) ? "test_environment_unavailable"
|
||||
: authenticationFailed ? "harness_auth_failed"
|
||||
: has("_hello_probe_model_unavailable") || has("_model_invalid") ? "harness_model_unavailable"
|
||||
: has("_command_unresolvable") ? "harness_command_unavailable"
|
||||
|
||||
@@ -10,6 +10,7 @@ import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { environmentLeases, executionWorkspaces, environments, heartbeatRuns, issues, projects, type Db } from "@paperclipai/db";
|
||||
import { freshWorkspaceSchema, environmentTaskOperationSchema, parseEnvironmentTaskResult,
|
||||
environmentTaskErrorData, readEnvironmentTaskErrorDiagnostic, JsonRpcCallError, PLUGIN_RPC_ERROR_CODES,
|
||||
environmentTaskErrorCodeData, PluginEnvironmentTaskError, readEnvironmentTaskErrorCode,
|
||||
type HostToWorkerMethodName, type HostToWorkerMethods, type PluginEnvironmentTaskOperation } from "@paperclipai/plugin-sdk";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
@@ -461,7 +462,9 @@ export async function executeEnvironmentTask(db: Db, workers: PluginWorkerManage
|
||||
data: environmentTaskErrorData({ status: diagnostic.httpStatus, code: diagnostic.errorCode, reason: diagnostic.reason }),
|
||||
});
|
||||
// Worker errors and schema diagnostics can contain request credentials.
|
||||
throw new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying");
|
||||
const code = readEnvironmentTaskErrorCode(error);
|
||||
throw Object.assign(new Error("Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying"),
|
||||
code ? { data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError(code)) } : {});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
|
||||
import { accountingForScheduledRetry, executionFailureRetryCount, executionRetryAttemptCount } from "./execution-recovery-attempt.js";
|
||||
import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js";
|
||||
import { RUNNER_REVISION_WAIT_RETRY_REASON as DISPATCH_RUNNER_REVISION_WAIT_RETRY_REASON } from "../modules/run-dispatch/index.js";
|
||||
|
||||
describe("failure attempts across resource waits", () => {
|
||||
it("preserves prior failures through subscription waits without trusting unrelated context", () => {
|
||||
@@ -27,7 +29,7 @@ describe("failure attempts across resource waits", () => {
|
||||
});
|
||||
|
||||
describe("persisted independent accounting", () => {
|
||||
it.each(["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait"])("%s cannot erase prior infrastructure debits or spend more", scheduledRetryReason => {
|
||||
it.each(["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait"])("%s cannot erase prior infrastructure debits or spend more", scheduledRetryReason => {
|
||||
expect(executionFailureRetryCount({ scheduledRetryReason, scheduledRetryAttempt: 20,
|
||||
contextSnapshot: { executionRetryAccounting: { version: 1, failureRetries: 2, maxTurnContinuations: 1 } },
|
||||
})).toBe(2);
|
||||
@@ -43,3 +45,25 @@ describe("persisted independent accounting", () => {
|
||||
})).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Runner update waits", () => {
|
||||
it("uses the same retry reason as run dispatch", () => {
|
||||
expect(RUNNER_REVISION_WAIT_RETRY_REASON).toBe(DISPATCH_RUNNER_REVISION_WAIT_RETRY_REASON);
|
||||
});
|
||||
const ledger = (failureRetries: number) => ({ executionRetryAccounting: { version: 1, failureRetries, maxTurnContinuations: 0 } });
|
||||
it("never spend the failure retry budget", () => {
|
||||
const waited = { scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 9, contextSnapshot: ledger(0) };
|
||||
expect(executionRetryAttemptCount(waited, "runner_revision_wait")).toBe(9);
|
||||
expect(accountingForScheduledRetry(waited, "runner_revision_wait", 10)).toMatchObject({ failureRetries: 0 });
|
||||
expect(executionFailureRetryCount(waited)).toBe(0);
|
||||
expect(executionRetryAttemptCount(waited, "transient_failure")).toBe(0);
|
||||
});
|
||||
it("keep failures recorded before the wait and are not shortened by them", () => {
|
||||
const failed = { scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 1, contextSnapshot: ledger(1) };
|
||||
expect(executionRetryAttemptCount(failed, "runner_revision_wait")).toBe(0);
|
||||
const waited = { scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 3,
|
||||
contextSnapshot: { ...ledger(1), executionRetryAccounting: accountingForScheduledRetry(failed, "runner_revision_wait", 3) } };
|
||||
expect(executionFailureRetryCount(waited)).toBe(1);
|
||||
expect(executionRetryAttemptCount(waited, "transient_failure")).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import { RUNNER_REVISION_WAIT_RETRY_REASON } from "./runner-revision-mismatch.js";
|
||||
|
||||
type RetryRun = {
|
||||
scheduledRetryAttempt?: number | null;
|
||||
scheduledRetryReason?: string | null;
|
||||
@@ -42,7 +44,7 @@ function historicalFailureCount(run: RetryRun): number {
|
||||
|
||||
export function executionRetryAccounting(run: RetryRun): ExecutionRetryAccounting {
|
||||
const saved = savedAccounting(run);
|
||||
const nonFailureLane = ["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait"].includes(run.scheduledRetryReason ?? "");
|
||||
const nonFailureLane = ["max_turns_continuation", "issue_disposition_repair", "workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", RUNNER_REVISION_WAIT_RETRY_REASON].includes(run.scheduledRetryReason ?? "");
|
||||
return {
|
||||
version: 1,
|
||||
failureRetries: Math.max(saved?.failureRetries ?? 0, saved && nonFailureLane ? 0 : historicalFailureCount(run)),
|
||||
@@ -57,7 +59,7 @@ export function executionFailureRetryCount(run: RetryRun): number {
|
||||
}
|
||||
|
||||
export function executionRetryAttemptCount(run: RetryRun, reason: string): number {
|
||||
if (reason === "workspace_busy" || reason === "ai_connection_busy" || reason === "ai_connection_pool_wait") {
|
||||
if (reason === "workspace_busy" || reason === "ai_connection_busy" || reason === "ai_connection_pool_wait" || reason === RUNNER_REVISION_WAIT_RETRY_REASON) {
|
||||
return run.scheduledRetryReason === reason ? count(run.scheduledRetryAttempt) ?? 0 : 0;
|
||||
}
|
||||
const accounting = executionRetryAccounting(run);
|
||||
@@ -67,6 +69,7 @@ export function executionRetryAttemptCount(run: RetryRun, reason: string): numbe
|
||||
export function accountingForScheduledRetry(run: RetryRun, reason: string, attempt: number): ExecutionRetryAccounting {
|
||||
const accounting = executionRetryAccounting(run);
|
||||
if (reason === "max_turns_continuation") accounting.maxTurnContinuations = attempt;
|
||||
else if (reason !== "workspace_busy" && reason !== "ai_connection_busy" && reason !== "ai_connection_pool_wait") accounting.failureRetries = attempt;
|
||||
else if (reason !== "workspace_busy" && reason !== "ai_connection_busy" && reason !== "ai_connection_pool_wait"
|
||||
&& reason !== RUNNER_REVISION_WAIT_RETRY_REASON) accounting.failureRetries = attempt;
|
||||
return accounting;
|
||||
}
|
||||
@@ -98,7 +98,7 @@ import {
|
||||
createHeartbeatRecovery,
|
||||
} from "./heartbeat/recovery.js";
|
||||
import {
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS,
|
||||
failedRunRetryBudgetSpent,
|
||||
isWorkspaceBusyDeferral,
|
||||
createHeartbeatRetries,
|
||||
} from "./heartbeat/retries.js";
|
||||
@@ -147,6 +147,7 @@ export {
|
||||
|
||||
import {
|
||||
ConfigurationIncompleteFailure,
|
||||
classifyManagedProjectAuthFailure,
|
||||
deriveTaskKey,
|
||||
type WakeupOptions,
|
||||
mergeCoalescedContextSnapshot,
|
||||
@@ -289,7 +290,6 @@ import { retryChatControlAdmission } from "./chat-control-admission-retry.js";
|
||||
import {
|
||||
prepareManagedAiRuntime,
|
||||
assertManagedAiProjectAuth,
|
||||
isAiConnectionIncompatible,
|
||||
} from "./ai-connection-runtime.js";
|
||||
import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared";
|
||||
import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js";
|
||||
@@ -307,7 +307,6 @@ import {
|
||||
adapterExecutionControls,
|
||||
createAdapterExecutionControl,
|
||||
} from "./adapter-execution-control.js";
|
||||
import { executionFailureRetryCount } from "./execution-recovery-attempt.js";
|
||||
import { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status-payload.js";
|
||||
export { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status-payload.js";
|
||||
import { StaleExecutionContinuationError } from "./execution-continuation.js";
|
||||
@@ -1002,9 +1001,7 @@ export function heartbeatService(
|
||||
// Mirrors scheduleBoundedRetryForRun's transient budget check: a failed
|
||||
// or interrupted run that has already consumed every bounded transient
|
||||
// attempt cannot be retried again through this lane.
|
||||
transientRetryBudgetSpent: (run) =>
|
||||
executionFailureRetryCount(run) >=
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS,
|
||||
transientRetryBudgetSpent: failedRunRetryBudgetSpent,
|
||||
});
|
||||
const runDispatch = createRunDispatch(db);
|
||||
const {
|
||||
@@ -2825,11 +2822,7 @@ export function heartbeatService(
|
||||
});
|
||||
if (managedAiRuntime && aiBinding) {
|
||||
try { await assertManagedAiProjectAuth({ ...resolvedConfig, cwd: executionWorkspace.cwd }, aiBinding.provider, executionTarget); }
|
||||
catch (error) {
|
||||
// Only a found conflict is the board's to fix; an unfinished check stays an ordinary failure.
|
||||
if (!isAiConnectionIncompatible(error)) throw error;
|
||||
throw new ConfigurationIncompleteFailure(error.message, { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } });
|
||||
}
|
||||
catch (error) { throw classifyManagedProjectAuthFailure(error, agent.id); }
|
||||
}
|
||||
const remoteExecution = realizationResult.remoteExecution;
|
||||
if (nativeChatWorkspaceScope && materializeNativeChatWorktree) {
|
||||
|
||||
@@ -6,7 +6,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest
|
||||
import { writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { HttpError } from "../../errors.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { createRunDispatch, type PostCommitEffect } from "../../modules/run-dispatch/index.js";
|
||||
import { createRunDispatch, isResolvedInteractionContinuationWakeContext, type PostCommitEffect } from "../../modules/run-dispatch/index.js";
|
||||
import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js";
|
||||
import * as legacy from "../heartbeat.js";
|
||||
import { createHeartbeatRunState } from "./run-state.js";
|
||||
@@ -112,6 +112,22 @@ describe("heartbeat retry module callbacks", () => {
|
||||
expect(database.access).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("gives every caller the Runner wait budget for a Runner mismatch and reports its exhaustion", async () => {
|
||||
const database = guardedDatabase();
|
||||
const deps = callbacks();
|
||||
const run = { ...unitRun(), status: "failed", errorCode: "runner_revision_mismatch", scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 10,
|
||||
contextSnapshot: { issueId: "issue", executionRetryAccounting: { version: 1, failureRetries: 0, maxTurnContinuations: 0 } } } as Run;
|
||||
const retry = await extracted.createHeartbeatRetries(database.db, deps).scheduleBoundedRetryForRun(run, { adapterType: "codex_local" } as Agent);
|
||||
expect(retry).toMatchObject({ outcome: "retry_exhausted", attempt: 11, maxAttempts: 10 });
|
||||
expect(deps.appendRunEvent).toHaveBeenCalledWith(run, expect.objectContaining({
|
||||
retryExhaustion: { retryReason: "runner_revision_wait", scheduledRetryAttempt: 10, maxAttempts: 10 },
|
||||
}));
|
||||
expect(extracted.failedRunRetryBudgetSpent(run)).toBe(true);
|
||||
expect(extracted.failedRunRetryBudgetSpent({ ...run, scheduledRetryAttempt: 9 })).toBe(false);
|
||||
expect(extracted.failedRunRetryBudgetSpent({ ...run, errorCode: "setup_failed" })).toBe(false);
|
||||
expect(database.access).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("leaves successor work to the committed case handoff without retry escalation", async () => {
|
||||
const database = guardedDatabase();
|
||||
const deps = callbacks();
|
||||
@@ -273,6 +289,23 @@ describePostgres("heartbeat retry module database wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("waits for a Runner update without spending failure retries and keeps an answered interaction", async () => {
|
||||
const { run, agent } = await fixture();
|
||||
const snapshot = { ...run.contextSnapshot, interactionId: "interaction-1", interactionStatus: "answered", interactionKind: "ask_user_questions",
|
||||
mutation: "interaction", wakeReason: "issue_commented", executionRetryAccounting: { version: 1, failureRetries: 1, maxTurnContinuations: 0 } };
|
||||
const [failed] = await db.update(heartbeatRuns).set({ errorCode: "runner_revision_mismatch", contextSnapshot: snapshot })
|
||||
.where(eq(heartbeatRuns.id, run.id)).returning();
|
||||
const result = await boundRetries().retries.scheduleBoundedRetryForRun(failed!, agent, { now });
|
||||
expect(result.outcome).toBe("scheduled");
|
||||
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run.id));
|
||||
expect(successor).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "runner_revision_wait", scheduledRetryAttempt: 1 });
|
||||
expect(successor!.contextSnapshot).toMatchObject({ mutation: "interaction", interactionId: "interaction-1",
|
||||
executionRetryAccounting: { failureRetries: 1 } });
|
||||
expect(isResolvedInteractionContinuationWakeContext(successor!.contextSnapshot)).toBe(true);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, successor!.wakeupRequestId!));
|
||||
expect(wake!.payload).toMatchObject({ mutation: "interaction", interactionId: "interaction-1", retryReason: "runner_revision_wait" });
|
||||
});
|
||||
|
||||
it("resolves current skills on a retry while retaining the failed attempt snapshot", async () => {
|
||||
const {run,agent}=await fixture();
|
||||
const snapshot={...run.contextSnapshot,skillVersions:{"company/review":"v1"},skillSelectionSnapshot:true,shadowedSkills:[{key:"old",shadowedBy:"company/review"}]};
|
||||
|
||||
@@ -24,6 +24,7 @@ import { WORKSPACE_GIT_SCAN_ERROR_CODES } from "../workspace-git-operation-sched
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
|
||||
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
|
||||
import { RUNNER_REVISION_MISMATCH_CODE, RUNNER_REVISION_MISMATCH_RETRY, RUNNER_REVISION_WAIT_RETRY_REASON } from "../runner-revision-mismatch.js";
|
||||
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import {
|
||||
@@ -136,12 +137,19 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
|
||||
export function isTransientWorkspacePreparationCode(code: string | null | undefined): boolean {
|
||||
return code === WORKSPACE_GIT_SCAN_ERROR_CODES.timeout || code === WORKSPACE_GIT_SCAN_ERROR_CODES.saturated
|
||||
|| code === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || code === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|
||||
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE;
|
||||
|| code === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || code === PLUGIN_WORKER_UNAVAILABLE_CODE
|
||||
|| code === RUNNER_REVISION_MISMATCH_CODE;
|
||||
}
|
||||
|
||||
export const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS =
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length;
|
||||
|
||||
export function failedRunRetryBudgetSpent(run: Parameters<typeof executionFailureRetryCount>[0] & { errorCode?: string | null }): boolean {
|
||||
return run.errorCode === RUNNER_REVISION_MISMATCH_CODE
|
||||
? executionRetryAttemptCount(run, RUNNER_REVISION_WAIT_RETRY_REASON) >= RUNNER_REVISION_MISMATCH_RETRY.maxAttempts
|
||||
: executionFailureRetryCount(run) >= BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS;
|
||||
}
|
||||
|
||||
const INTERACTION_CONTINUATION_INFRA_MAX_ATTEMPTS = 2;
|
||||
|
||||
const MAX_TURN_CONTINUATION_LIVE_RUN_STATUSES = [
|
||||
@@ -343,7 +351,8 @@ function isRetryableInteractionContinuationInfrastructureFailure(
|
||||
run.errorCode === WORKSPACE_VALIDATION_FAILURE_CODE ||
|
||||
run.errorCode === "process_lost" ||
|
||||
run.errorCode === CONTROLLER_LEASE_LOST_FAILURE_CODE ||
|
||||
run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
|
||||
run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE ||
|
||||
run.errorCode === RUNNER_REVISION_MISMATCH_CODE
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
@@ -419,6 +428,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
delayMs?: number;
|
||||
},
|
||||
) {
|
||||
if (!opts?.retryReason && run.errorCode === RUNNER_REVISION_MISMATCH_CODE) opts = { ...RUNNER_REVISION_MISMATCH_RETRY, ...opts };
|
||||
if (await hasCaseHandoffForRun(run)) return { outcome: "not_scheduled" as const,
|
||||
reason: "The durable case handoff owns successor work.", issueId: readNonEmptyString(run.contextSnapshot?.issueId) };
|
||||
if (["openai_dot", "muse"].includes(String(parseObject(agent.adapterConfig).provider))
|
||||
@@ -615,8 +625,10 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
}
|
||||
const taskKey = deriveTaskKeyWithHeartbeatFallback(contextSnapshot, null);
|
||||
const sessionBefore = await resolveSessionBeforeForWakeup(agent, taskKey);
|
||||
const interactionRunnerWait = retryReason === RUNNER_REVISION_WAIT_RETRY_REASON
|
||||
&& isResolvedInteractionContinuationWakeContext(run.contextSnapshot);
|
||||
const interactionContinuationPayload =
|
||||
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON
|
||||
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON || interactionRunnerWait
|
||||
? {
|
||||
mutation: "interaction",
|
||||
interactionId: readNonEmptyString(contextSnapshot.interactionId),
|
||||
@@ -646,6 +658,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
retryOfRunId: run.id,
|
||||
wakeReason,
|
||||
retryReason,
|
||||
...(interactionRunnerWait ? { mutation: "interaction" } : {}),
|
||||
...(retryReason === WORKSPACE_BUSY_RETRY_REASON
|
||||
? {
|
||||
failureRetriesBeforeWorkspaceWait:
|
||||
@@ -1001,7 +1014,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
|
||||
|
||||
const scheduledRunId = randomUUID();
|
||||
if (contextSnapshot.explicitUserContinuation) {
|
||||
const continuation = issueId && retryReason === "transient_failure" ? await admitExplicitContinuationRetry({
|
||||
const continuation = issueId && (retryReason === "transient_failure" || retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) ? await admitExplicitContinuationRetry({
|
||||
db: tx as unknown as Db, companyId: run.companyId, issueId, agentId: run.agentId,
|
||||
parentRunId: run.id, successorRunId: scheduledRunId, now,
|
||||
}) : null;
|
||||
|
||||
@@ -30,8 +30,11 @@ import { RemoteWorkspaceOperationError, RemoteWorkspaceOperationTimeoutError, Re
|
||||
import { EnvironmentRunError } from "../environment-run-orchestrator.js";
|
||||
import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
|
||||
import { PluginWorkerUnavailableError } from "../plugin-worker-unavailable-error.js";
|
||||
import { environmentTaskErrorCodeData, JsonRpcCallError, PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
|
||||
import { RUNNER_REVISION_MISMATCH_RETRY } from "../runner-revision-mismatch.js";
|
||||
import { unprocessable } from "../../errors.js";
|
||||
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js";
|
||||
import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure } from "./run-preparation.js";
|
||||
import { createHeartbeatRunPreparation, ConfigurationIncompleteFailure, classifyManagedProjectAuthFailure } from "./run-preparation.js";
|
||||
import { createHeartbeatRunState, getAdapterSessionCodec } from "./run-state.js";
|
||||
import { buildEffectiveRunSessionConfigMetadata } from "./workspaces.js";
|
||||
import { createHeartbeatRunCompletion, type CompleteHeartbeatRunInput, type FailHeartbeatRunInput, type HeartbeatRunCompletionDependencies } from "./run-completion.js";
|
||||
@@ -966,6 +969,61 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
|
||||
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
const runnerMismatch = () => new JsonRpcCallError({ code: -32002, message: "The installed Runner does not match the expected revision",
|
||||
data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) });
|
||||
|
||||
it.each(["fresh", "wrapped", "started", "untrusted", "retained"] as const)("waits for the Runner update after a %s Runner mismatch", async scenario => {
|
||||
const f = await fixture({ runtimeMode: scenario === "retained" ? "native" : "legacy" });
|
||||
const refusal = runnerMismatch();
|
||||
const error = scenario === "untrusted" ? new Error(refusal.message)
|
||||
: scenario === "wrapped" ? new Error("Plugin Runner task failed", { cause: refusal })
|
||||
: classifyManagedProjectAuthFailure(refusal, f.agent.id);
|
||||
f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
|
||||
await f.completion.failRunSetup(f.setup(error, scenario === "started"));
|
||||
const failed = await read(f.run.id);
|
||||
const retried = scenario === "fresh" || scenario === "wrapped";
|
||||
expect(failed).toMatchObject({ status: "failed", errorCode: retried ? "runner_revision_mismatch" : "setup_failed" });
|
||||
expect(failed.resultJson).not.toHaveProperty("configurationIncomplete");
|
||||
if (retried) {
|
||||
expect(failed.resultJson).toMatchObject({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } });
|
||||
expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent, RUNNER_REVISION_MISMATCH_RETRY);
|
||||
} else expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("waits for the Runner update on an answered interaction instead of its two-attempt continuation retry", async () => {
|
||||
const f = await fixture({ contextSnapshot: { interactionId: "interaction-1", interactionStatus: "answered", interactionKind: "ask_user_questions",
|
||||
mutation: "interaction", wakeReason: "issue_commented" } });
|
||||
f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce({ outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
|
||||
await f.completion.failRunSetup(f.setup(runnerMismatch()));
|
||||
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "runner_revision_mismatch" });
|
||||
expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledWith(expect.objectContaining({ id: f.run.id }), f.agent, RUNNER_REVISION_MISMATCH_RETRY);
|
||||
expect(f.deps.scheduleInteractionContinuationInfrastructureRetryIfEligible).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps a found project authentication conflict a configuration gap that blocks for the board", async () => {
|
||||
const f = await fixture();
|
||||
const conflict = unprocessable("This agent's managed AI connection conflicts with project authentication settings in .claude/settings.json.",
|
||||
{ code: "ai_connection_incompatible" });
|
||||
await f.completion.failRunSetup(f.setup(classifyManagedProjectAuthFailure(conflict, f.agent.id)));
|
||||
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "configuration_incomplete",
|
||||
resultJson: { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${f.agent.id}/runtime` } } });
|
||||
expect(f.deps.scheduleBoundedRetryForRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["an unavailable Runner identity", new Error("Runner identity unavailable")],
|
||||
["an unreachable execution host", Object.assign(new Error("connect ECONNREFUSED"), { code: "ECONNREFUSED" })],
|
||||
["another 422", unprocessable("Select a saved AI account", { code: "ai_connection_busy" })],
|
||||
])("does not report %s as a project authentication conflict", async (_name, error) => {
|
||||
const f = await fixture();
|
||||
const classified = classifyManagedProjectAuthFailure(error, f.agent.id);
|
||||
expect(classified).toBe(error);
|
||||
await f.completion.failRunSetup(f.setup(classified));
|
||||
const failed = await read(f.run.id);
|
||||
expect(failed.errorCode).not.toBe("configuration_incomplete");
|
||||
expect(failed.resultJson).not.toHaveProperty("configurationIncomplete");
|
||||
});
|
||||
|
||||
it("routes a lease acquisition refused by a stopped provider worker through bounded bootstrap recovery", async () => {
|
||||
const f = await fixture();
|
||||
const refusal = new PluginWorkerUnavailableError({ pluginId: "plugin-1", workerStatus: null,
|
||||
|
||||
@@ -13,6 +13,7 @@ import { RemoteWorkspacePreparationTimeoutError, REMOTE_WORKSPACE_RESTORE_FAILED
|
||||
import { escalateRemoteWorkspaceRestoreFailure } from "../remote-workspace-restore-recovery.js";
|
||||
import { findReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
|
||||
import { findPluginWorkerUnavailableError, PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
|
||||
import { isRunnerRevisionMismatch, RUNNER_REVISION_MISMATCH_CODE, RUNNER_REVISION_MISMATCH_RETRY } from "../runner-revision-mismatch.js";
|
||||
import { type RunSessionOutcome, resolveNextSessionState, normalizeUsageTotals, resolveCacheAdjustedCostUsd, resolveLedgerBiller, resolveLedgerCostStatus, normalizeLedgerBillingType } from "./run-state.js";
|
||||
import { readGitConnectionFailure } from "../git-connection-failure.js";
|
||||
import { CONFIGURATION_INCOMPLETE_FAILURE_CODE, ConfigurationIncompleteFailure } from "./run-preparation.js";
|
||||
@@ -1376,12 +1377,15 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
// bootstrap budget; the worker is restarted or replaced meanwhile.
|
||||
const pluginWorkerUnavailableFailure = !providerDispatchStarted && run.runtimeMode === "legacy"
|
||||
&& run.nativeSessionId === null && run.runnerInstanceId === null ? findPluginWorkerUnavailableError(outerErr) : null;
|
||||
const runnerRevisionMismatch = !providerDispatchStarted && run.runtimeMode === "legacy"
|
||||
&& run.nativeSessionId === null && run.runnerInstanceId === null && isRunnerRevisionMismatch(outerErr);
|
||||
const setupFailureErrorCode =
|
||||
(controllerLeaseLost ? CONTROLLER_LEASE_LOST_FAILURE_CODE : null) ??
|
||||
remoteWorkspacePreparationTimeout?.code ??
|
||||
reusableSandboxResumeFailure?.code ??
|
||||
remoteWorkspaceRestoreFailure?.code ??
|
||||
pluginWorkerUnavailableFailure?.code ??
|
||||
(runnerRevisionMismatch ? RUNNER_REVISION_MISMATCH_CODE : null) ??
|
||||
workspaceGitScanFailure?.code ??
|
||||
workspaceValidationSetupFailure?.code ??
|
||||
configurationIncompleteSetupFailure?.code ??
|
||||
@@ -1544,7 +1548,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
const interactionOwnsRetry = livenessRun.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
|
||||
&& isResolvedInteractionContinuationWakeContext(livenessRun.contextSnapshot);
|
||||
await (isTransientWorkspacePreparationCode(livenessRun.errorCode) && !interactionOwnsRetry
|
||||
? scheduleBoundedRetryForRun(livenessRun, failedAgent)
|
||||
? livenessRun.errorCode === RUNNER_REVISION_MISMATCH_CODE
|
||||
? scheduleBoundedRetryForRun(livenessRun, failedAgent, RUNNER_REVISION_MISMATCH_RETRY)
|
||||
: scheduleBoundedRetryForRun(livenessRun, failedAgent)
|
||||
: scheduleInteractionContinuationInfrastructureRetryIfEligible(livenessRun, failedAgent)
|
||||
).catch((retryError) => {
|
||||
logger.warn(
|
||||
|
||||
@@ -69,6 +69,7 @@ import {
|
||||
toolProfiles,
|
||||
} from "@paperclipai/db";
|
||||
import { HttpError } from "../../errors.js";
|
||||
import { isAiConnectionIncompatible } from "../ai-connection-runtime.js";
|
||||
import type { ReviewedChatBindingDenialReason } from "../reviewed-chat-binding-diagnostics.js";
|
||||
import { filterResolvedGitHubConnectionsForRun } from "../git-credentials.js";
|
||||
import type {
|
||||
@@ -162,6 +163,13 @@ export class ConfigurationIncompleteFailure extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
export function classifyManagedProjectAuthFailure(error: unknown, agentId: string): unknown {
|
||||
if (!isAiConnectionIncompatible(error)) return error;
|
||||
return new ConfigurationIncompleteFailure(error.message, {
|
||||
configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agentId}/runtime` },
|
||||
});
|
||||
}
|
||||
|
||||
const ISSUE_RESPONSIBLE_USER_WAKE_REASONS = new Set([
|
||||
"issue_assigned",
|
||||
"issue_checked_out",
|
||||
|
||||
@@ -10,7 +10,7 @@ const stopped = {
|
||||
},
|
||||
};
|
||||
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable", "runner_revision_mismatch"])("does not invent unknown provider actions after exhausted %s bootstrap retries", (errorCode) => {
|
||||
const run = { runtimeMode: "legacy", status: "failed", errorCode, scheduledRetryAttempt: 2,
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
|
||||
expect(legacyExecutionNeedsReconciliation(run)).toBe(false);
|
||||
|
||||
@@ -9,6 +9,7 @@ import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wai
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "./remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
|
||||
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "./plugin-worker-unavailable-error.js";
|
||||
import { RUNNER_REVISION_MISMATCH_CODE } from "./runner-revision-mismatch.js";
|
||||
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
|
||||
import { normalizeMaxTurnStopReason } from "./heartbeat-stop-metadata.js";
|
||||
import { claimedAdapterType, hasConversationContinuationPolicy } from "./conversation-continuation.js";
|
||||
@@ -79,7 +80,8 @@ export function legacyExecutionNeedsReconciliation(
|
||||
// that the bootstrap evidence proves never started. Keep unknown outcomes held.
|
||||
if ((run.errorCode === "workspace_git_scan_timeout" || run.errorCode === "workspace_git_scan_saturated"
|
||||
|| run.errorCode === REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE || run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE
|
||||
|| run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE) &&
|
||||
|| run.errorCode === REMOTE_WORKSPACE_RESTORE_FAILED_CODE || run.errorCode === PLUGIN_WORKER_UNAVAILABLE_CODE
|
||||
|| run.errorCode === RUNNER_REVISION_MISMATCH_CODE) &&
|
||||
evidence?.kind === "bootstrap" && evidence.providerWorkStarted === false) return false;
|
||||
if (executionFailureRetryCount(run) >= 2) return true;
|
||||
return !(
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { environmentTaskErrorCodeData, JsonRpcCallError, PluginEnvironmentTaskError } from "@paperclipai/plugin-sdk";
|
||||
import { probePluginRunnerEnvironment } from "./plugin-runner-probe.js";
|
||||
import { assertHarnessTestPassed } from "../agent-harness-verification.js";
|
||||
|
||||
const probe = (runnerTask: () => Promise<unknown>) => probePluginRunnerEnvironment({
|
||||
companyId: "fixture-company", adapterType: "paperclip_runner", config: { provider: "codex", model: "fixture-model" },
|
||||
executionTarget: { kind: "remote", transport: "sandbox", remoteCwd: ".", leaseId: "probe-lease",
|
||||
runnerTaskProbe: { runId: "probe-run", sessionId: "probe-session" }, runnerTask },
|
||||
} as Parameters<typeof probePluginRunnerEnvironment>[0]);
|
||||
|
||||
describe("plugin Runner probe with an outdated remote Runner", () => {
|
||||
it("reports a retryable Runner mismatch from the plugin's stable code", async () => {
|
||||
const refusal = new JsonRpcCallError({ code: -32002, message: "The installed Runner does not match the expected revision",
|
||||
data: environmentTaskErrorCodeData(new PluginEnvironmentTaskError("runner_revision_mismatch")) });
|
||||
const result = await probe(async () => { throw refusal; });
|
||||
expect(result).toMatchObject({ status: "fail", checks: [{ code: "runner_revision_mismatch", level: "error", retryable: true }] });
|
||||
expect(() => assertHarnessTestPassed(result)).toThrow(expect.objectContaining({ code: "runner_revision_mismatch" }));
|
||||
});
|
||||
|
||||
it("keeps other plugin failures a non-retryable failed probe, even with the same message", async () => {
|
||||
const result = await probe(async () => { throw new Error("The installed Runner does not match the expected revision"); });
|
||||
expect(result).toMatchObject({ status: "fail", checks: [{ code: "runner_hello_probe_failed", level: "error" }] });
|
||||
expect(result.checks[0]).not.toHaveProperty("retryable");
|
||||
});
|
||||
});
|
||||
@@ -11,6 +11,7 @@ import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js";
|
||||
import { createPluginRunnerTaskOwner } from "./plugin-runner-task.js";
|
||||
import { resolvePaperclipRunnerProviderProfile } from "./provider-profile.js";
|
||||
import { resolveAcpxQualification } from "./acpx-qualification.js";
|
||||
import { isRunnerRevisionMismatch, runnerRevisionMismatchTestResult } from "../runner-revision-mismatch.js";
|
||||
|
||||
/** Core verifies a real Runner; providers only implement their task API. */
|
||||
export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTestContext,
|
||||
@@ -70,6 +71,7 @@ export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTe
|
||||
checks: [{ code: "runner_hello_probe_passed", level: "info", message: `Runner verified ${result.providerDriver} with model ${result.effectiveModel}.` }] };
|
||||
} catch (error) {
|
||||
if (error instanceof NativeEnvironmentProbeCleanupError) throw error;
|
||||
if (isRunnerRevisionMismatch(error)) return runnerRevisionMismatchTestResult("paperclip_runner", testedAt);
|
||||
// Provider failures may contain credentials or private addressing.
|
||||
return { adapterType: "paperclip_runner", status: "fail", testedAt,
|
||||
checks: [{ code: "runner_hello_probe_failed", level: "error", message: "The selected plugin environment did not complete the Runner hello probe and confirmed cleanup. Check its runtime, account, and model access." }] };
|
||||
|
||||
@@ -5,7 +5,7 @@ const run = (errorCode: string | null) =>
|
||||
({ errorCode } as unknown as Parameters<typeof classifyContinuationFailure>[0]);
|
||||
|
||||
describe("pause durability: continuation retry classification", () => {
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable"])("does not grant %s another recovery budget", (code) => {
|
||||
it.each(["workspace_git_scan_timeout", "workspace_git_scan_saturated", "workspace_git_scan_failed", "workspace_git_scan_output_limit", "workspace_git_scan_cancelled", "remote_workspace_prepare_timeout", "reusable_sandbox_resume_failed", "remote_workspace_restore_failed", "plugin_worker_unavailable", "runner_revision_mismatch"])("does not grant %s another recovery budget", (code) => {
|
||||
expect(classifyContinuationFailure(run(code))).toMatchObject({ kind: "non_retryable", maxAttempts: 0 });
|
||||
});
|
||||
it("agent_paused is retryable so work resumes (Option A: Resume Continues Work)", () => {
|
||||
|
||||
@@ -10,6 +10,7 @@ import { boundRunIssueId } from "../run-task-binding.js";
|
||||
import { REMOTE_WORKSPACE_PREPARATION_TIMEOUT_CODE, REMOTE_WORKSPACE_RESTORE_FAILED_CODE } from "../remote-workspace-transport.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "../reusable-sandbox-resume-error.js";
|
||||
import { PLUGIN_WORKER_UNAVAILABLE_CODE } from "../plugin-worker-unavailable-error.js";
|
||||
import { RUNNER_REVISION_MISMATCH_CODE } from "../runner-revision-mismatch.js";
|
||||
import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js";
|
||||
import { recoveryRunResultColumn } from "./run-result-read.js";
|
||||
import { readLegacyDispositionSource } from "./legacy-disposition-read.js";
|
||||
@@ -561,6 +562,7 @@ const NON_RETRYABLE_CONTINUATION_ERROR_CODES = new Set<string>([
|
||||
REUSABLE_SANDBOX_RESUME_FAILED_CODE,
|
||||
REMOTE_WORKSPACE_RESTORE_FAILED_CODE,
|
||||
PLUGIN_WORKER_UNAVAILABLE_CODE,
|
||||
RUNNER_REVISION_MISMATCH_CODE,
|
||||
"low_trust_isolation_unavailable",
|
||||
"low_trust_requires_isolated_workspace",
|
||||
"low_trust_boundary_mismatch",
|
||||
|
||||
@@ -321,3 +321,25 @@ it("names the unavailable AI account instead of suggesting secret bindings", ()
|
||||
expect(notice.body).toContain("Reconnect the account");
|
||||
expect(notice.body).not.toContain("secret/env");
|
||||
});
|
||||
|
||||
it("names a found project authentication conflict instead of suggesting secret bindings", () => {
|
||||
const notice = buildConfigurationIncompleteRecoveryNoticeSeed({ reason: "ai_connection_incompatible", actionUrl: "/agents/agent-1/runtime" });
|
||||
expect(notice.title).toBe("AI connection conflicts with project settings");
|
||||
expect(notice.body).toContain("apiKeyHelper");
|
||||
expect(notice.nextAction).toContain("choose a different AI connection");
|
||||
expect(notice.body).not.toContain("secret/env");
|
||||
});
|
||||
|
||||
it("names a remote Runner that was still not updated after its retries", () => {
|
||||
const notice = buildStrandedRecoveryEscalationNotice({
|
||||
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }),
|
||||
recoveryActionId: "runner-recovery",
|
||||
recoveryOwner: null,
|
||||
sourceRun: { id: "failed-run", status: "failed", errorCode: "runner_revision_mismatch" },
|
||||
});
|
||||
expect(notice.presentation.title).toBe("Remote Runner is out of date");
|
||||
expect(notice.body).toContain("Runner was still not updated");
|
||||
expect(notice.body).not.toContain("workspace");
|
||||
expect(notice.body).not.toContain("authentication");
|
||||
expect(JSON.stringify(notice.metadata)).toContain("remote Runner received the update");
|
||||
});
|
||||
@@ -64,6 +64,16 @@ const WORKSPACE_SCAN_NOTICES: Record<string, { title: string; nextAction: string
|
||||
workspace_git_scan_cancelled: { title: "Workspace scan was cancelled", nextAction: "Inspect why workspace preparation was cancelled before retrying the task." },
|
||||
};
|
||||
|
||||
const SETUP_FAILURE_NOTICES: Record<string, StrandedRecoveryNoticeSeed> = {
|
||||
runner_revision_mismatch: {
|
||||
title: "Remote Runner is out of date",
|
||||
body: "The remote Runner does not match this Paperclip version, which is expected for a few minutes while a release updates the Runner. "
|
||||
+ "Paperclip retried before the agent started, but the Runner was still not updated.",
|
||||
nextAction: "Check that the remote Runner received the update for this release, then retry the task.",
|
||||
tone: "danger",
|
||||
},
|
||||
};
|
||||
|
||||
export function buildImmediateExecutionPathRecoveryNoticeSeed(input: {
|
||||
status: "todo" | "in_progress";
|
||||
}): StrandedRecoveryNoticeSeed {
|
||||
@@ -162,6 +172,16 @@ export function buildConfigurationIncompleteRecoveryNoticeSeed(
|
||||
tone: "danger",
|
||||
};
|
||||
}
|
||||
if (readNonEmptyStringField(configurationIncomplete, "reason") === "ai_connection_incompatible") {
|
||||
return {
|
||||
title: "AI connection conflicts with project settings",
|
||||
body: "Paperclip stopped before the agent started because this agent's managed AI connection conflicts with authentication settings "
|
||||
+ "in the project or the agent's arguments, such as an API key, token or apiKeyHelper. The task is blocked until the conflict is removed.",
|
||||
nextAction: "Remove the API key, token or apiKeyHelper from the project's Claude or Codex settings or the agent's arguments, "
|
||||
+ "or choose a different AI connection for this agent, then retry the task.",
|
||||
tone: "danger",
|
||||
};
|
||||
}
|
||||
if (readNonEmptyStringField(configurationIncomplete, "reason") === SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON) {
|
||||
const pluginKey = readNonEmptyStringField(configurationIncomplete, "pluginKey") ?? "the sandbox provider plugin";
|
||||
const pluginStatus = readNonEmptyStringField(configurationIncomplete, "pluginStatus") ?? "not ready";
|
||||
@@ -229,7 +249,7 @@ export function buildStrandedRecoveryEscalationNotice(input: {
|
||||
...workspaceScan,
|
||||
body: `Paperclip could not prepare the workspace before the agent started. Automatic recovery could not continue. ${workspaceScan.nextAction}`,
|
||||
tone: "danger" as const,
|
||||
} : input.seed;
|
||||
} : SETUP_FAILURE_NOTICES[input.sourceRun?.errorCode ?? ""] ?? input.seed;
|
||||
const fallbackBody = input.fallbackBody?.trim();
|
||||
const body = seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY);
|
||||
const title =
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
import type { AdapterEnvironmentTestResult } from "@paperclipai/adapter-utils";
|
||||
import { readEnvironmentTaskErrorCode } from "@paperclipai/plugin-sdk";
|
||||
|
||||
export const RUNNER_REVISION_MISMATCH_CODE = "runner_revision_mismatch";
|
||||
|
||||
export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait";
|
||||
|
||||
export const RUNNER_REVISION_MISMATCH_RETRY = {
|
||||
retryReason: RUNNER_REVISION_WAIT_RETRY_REASON, wakeReason: "runner_revision_retry", maxAttempts: 10, delayMs: 60_000,
|
||||
} as const;
|
||||
|
||||
export function isRunnerRevisionMismatch(error: unknown): boolean {
|
||||
return readEnvironmentTaskErrorCode(error) === RUNNER_REVISION_MISMATCH_CODE;
|
||||
}
|
||||
|
||||
export function runnerRevisionMismatchTestResult(adapterType: string, testedAt = new Date().toISOString()): AdapterEnvironmentTestResult {
|
||||
return { adapterType, status: "fail", testedAt, checks: [{ code: RUNNER_REVISION_MISMATCH_CODE, level: "error", retryable: true,
|
||||
message: "The remote Runner does not match this Paperclip version yet. Paperclip will check again after the Runner update." }] };
|
||||
}
|
||||
Reference in new issue
Block a user