mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(native): stop a session.open model rejection instead of a doomed bootstrap retry (#19190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner runs start a provider session with the `session.open` command before any prompt > - A definite `session.open` rejection is treated as a pre-session failure and gets a bounded same-run bootstrap retry > - The runner also reports a closed classification. Two classifications say that the provider cannot run the selected model: `requested_model_unsupported` and `grok_model_mismatch` > - Paperclip ignored that classification. It scheduled a retry that must fail the same way. On a remote sandbox with a retained runner PID, the due retry entered restart recovery. Restart recovery has no checkpoint for a bootstrap retry, so it failed closed as `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence` > - The task then stayed `blocked`. The recovery actions settled as no-replay holds, so no active action told anyone to change the model, and the parent task stalled too > - This pull request classifies those two definite rejections as the existing `native_provider_model_rejected` failure > - The benefit is a terminal failure on the first attempt, an accurate error, and an active board recovery action that names the next step: choose a supported model ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a Grok agent run failed at `session.open`. The runner log said `ACPX sidecar command session.open was rejected (retryable=false, classification=grok_model_mismatch)`. Paperclip recorded `recoveryMode: "bootstrap_retry"` and scheduled a same-run retry 30 seconds later. The run had a remote sandbox and a recorded sandbox PID, so the due retry went to restart recovery. Restart recovery found no checkpoint and no provider events and failed closed with `ambiguous_remote_recovery_evidence`. The run ended as `native_restart_recovery_blocked`. The task stayed `blocked` with no active recovery action for more than one hour. Its parent task stayed blocked on it. **What did you expect to happen?** The selected model cannot start, so a retry cannot succeed. The run should fail once with the model-rejection error. The task should show the existing "choose a model" action, owned by the board, until someone changes the model and retries. **Steps to reproduce** 1. Start a native run whose `session.open` command fails with `classification=grok_model_mismatch` (or `requested_model_unsupported`). 2. Read the `native_run_finalizations` row. Before this change it is `retryable_failure` with `recoveryMode: "bootstrap_retry"` and a `nextAttemptAt`. 3. On a remote sandbox run with a recorded PID, the due retry enters restart recovery and fails as `native_restart_recovery_blocked`. **Paperclip version or commit** `master` at `f5f5184f23`. Related: #18910 routes due remote retries with a retained PID to restart recovery. #18908 continues verified failed native preparation. Open PRs #19177, #19180, #19182 and #19184 change no-replay hold settlement and workspace hold release. They do not change this classification. This PR does not duplicate them. ## What Changed - `nativeSessionFailureSourceCode` (`server/src/services/native-runtime/native-session-executor.ts`) returns `native_provider_model_rejected` for a definite `session.open` failure or rejection whose runner classification is `requested_model_unsupported` or `grok_model_mismatch`. Other definite `session.open` rejections still return `provider_session_open_rejected` and keep the bounded startup retry. An indeterminate `session.open` outcome stays `native_session_interrupted`. - `native_provider_model_rejected` is already a permanent failure. The run stops on the first attempt with no `nextAttemptAt`, so it never enters the restart-recovery retry path. - When no provider session exists, the coordinator `nextAction` and the board recovery action `nextAction` say: choose a supported model for this agent, then retry the task. - The existing product paths for `native_provider_model_rejected` apply: the task presentation shows `choose_model`, and the recovery service treats it as `configuration_incomplete`. - `doc/execution-semantics.md` describes the rule under "Native provider model capacity". ## Verification Red before the fix (new test, unchanged production code): ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 2 failed | 21 passed # failureCode: native_session_interrupted, phase: retryable_failure, recoveryMode: bootstrap_retry, # nextAction "Retry provider bootstrap on this same run ..." - the same record seen on staging ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 23 passed cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts \ src/services/native-runtime/native-startup-retry.integration.test.ts src/services/task-interruption-presentation.test.ts # Test Files 3 passed, Tests 1186 passed pnpm --filter @paperclipai/server typecheck # exit 0 ``` The new integration cases use embedded PostgreSQL and the real executor failure path. For both classifications they check: - coordinator `terminal_failure`, `failureCode: native_provider_model_rejected`, `nextAttemptAt: null`, attempt 1 - run `failed` with `errorCode: native_provider_model_rejected` - task `blocked`, execution lock released - an `active` board recovery action with cause `native_provider_model_rejected`, return owner set to the agent, and the choose-model next action - `settleUnrecoverableExecutions` leaves the action `active`, so it does not become a silent no-replay hold - `claimNativeSessionResumptions` claims nothing, so there is no second executor The unit test also checks that `session_ensure_failed` keeps its bounded retry and that an indeterminate model rejection stays unclassified. A live Grok run with a mismatched model was not run. The failure record in the test matches the staging record field by field. ## Risks - Low risk. The change adds two closed runner classifications to an existing permanent failure code. Other `session.open` rejections keep their current retry. - A model mismatch that a retry could fix would now need an explicit retry. The runner already marks these rejections `retryable=false`. - This PR does not change restart recovery. A remote bootstrap retry with a retained sandbox PID for other `session.open` causes can still fail closed as `ambiguous_remote_recovery_evidence`. That needs a separate change with remote stop evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads of the staging API). It did the investigation, code change, tests and PR text. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
46a50a3bdd
commit
f1516e8009
4 files changed
+80
-1
No files matched your search
@@ -959,6 +959,14 @@ workspace finalization, and environment cleanup to settle. Model incompatibility
|
||||
usage-limit exhaustion, unknown failures, and unbound diagnostic text do not
|
||||
qualify as capacity failures.
|
||||
|
||||
A definite `session.open` rejection whose runner classification says the
|
||||
provider cannot run the selected model (`requested_model_unsupported`,
|
||||
`grok_model_mismatch`) is a model rejection, not a startup interruption. It
|
||||
stops on its first attempt as `native_provider_model_rejected`, with no
|
||||
same-run bootstrap retry, and leaves an active board recovery action that asks
|
||||
for a supported model. Other `session.open` rejections keep the bounded
|
||||
startup retry. An indeterminate `session.open` outcome stays unclassified.
|
||||
|
||||
### Workspace scan failures before provider startup
|
||||
|
||||
Workspace source preparation keeps its dedicated advisory-lock connection active
|
||||
|
||||
@@ -17979,6 +17979,15 @@ it("classifies only a definite session.open outcome as a bounded pre-session fai
|
||||
.toBe("native_session_interrupted");
|
||||
expect(nativeSessionFailureSourceCode(new Error(`PRP command turn.start failed: ${detail}`)))
|
||||
.toBe("native_session_interrupted");
|
||||
// A definite model rejection needs a model choice; an indeterminate one stays unknown.
|
||||
const modelDetail = detail.replace("classification=session_ensure_failed", "classification=grok_model_mismatch");
|
||||
const modelCode = nativeSessionFailureSourceCode(new Error(`PRP command session.open failed: ${modelDetail}`));
|
||||
expect(modelCode).toBe("native_provider_model_rejected");
|
||||
expect(nativeSessionFailureDisposition(1, new Date(), modelCode)).toEqual({
|
||||
phase: "terminal_failure", failureCode: "native_provider_model_rejected", nextAttemptAt: null,
|
||||
});
|
||||
expect(nativeSessionFailureSourceCode(new Error(`PRP command session.open indeterminate: ${modelDetail}`)))
|
||||
.toBe("native_session_interrupted");
|
||||
});
|
||||
describe("HTTP Stop during remote runner staging", () => {
|
||||
it("does not dispatch after Stop and bounded bootstrap cleanup", async () => {
|
||||
|
||||
@@ -7305,6 +7305,12 @@ export function nativeSessionFailureSourceCode(
|
||||
// before any prompt, so provider work cannot have started. An indeterminate
|
||||
// outcome stays generic.
|
||||
if (/\bPRP command session\.open (?:failed|rejected): /.test(message)) {
|
||||
// The sidecar's closed classification says the provider cannot run the
|
||||
// selected model. A same-run bootstrap retry would open the same session
|
||||
// and fail the same way, so this needs a model choice, not a retry.
|
||||
if (/classification=(?:requested_model_unsupported|grok_model_mismatch)\)/.test(message)) {
|
||||
return "native_provider_model_rejected";
|
||||
}
|
||||
return "provider_session_open_rejected";
|
||||
}
|
||||
return "native_session_interrupted";
|
||||
@@ -7313,6 +7319,9 @@ export function nativeSessionFailureSourceCode(
|
||||
const NATIVE_PROVIDER_STATE_INCOMPATIBLE_ACTION =
|
||||
"The current runner cannot restore this task's saved provider session, so retrying this run cannot succeed. Retry the task to continue in a fresh provider session with the same workspace. The earlier provider conversation is not carried over.";
|
||||
|
||||
const NATIVE_STARTUP_MODEL_REJECTED_ACTION =
|
||||
"The provider could not start the selected model before any provider work began. Choose a supported model for this agent, then retry the task. Automatic retries are stopped.";
|
||||
|
||||
const NATIVE_CLEANUP_OPERATOR_RECOVERY_MESSAGE =
|
||||
"Send a new message to continue after Paperclip verifies that the previous provider and its tools have stopped. If cleanup cannot be verified, inspect the run and its environment. Clearing a task session does not resolve this quarantine. Automatic retries are stopped.";
|
||||
|
||||
@@ -10746,6 +10755,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
? "The computer provider rejected access. Restore the environment's credentials or permissions before retrying. Automatic retries are stopped."
|
||||
: sourceFailureCode === "native_provider_terminal_failed"
|
||||
? "The provider session is permanently unusable. Verify stopped execution, completed actions, and task context before starting a linked continuation."
|
||||
: sourceFailureCode === "native_provider_model_rejected" && !recoveryEvidence.providerSessionEstablished
|
||||
? NATIVE_STARTUP_MODEL_REJECTED_ACTION
|
||||
: sourceFailureCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE
|
||||
? NATIVE_PROVIDER_STATE_INCOMPATIBLE_ACTION
|
||||
: recoveryEvidence.recoveryMode === "ambiguous_state"
|
||||
@@ -10997,6 +11008,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
? "The computer provider rejected access. Restore the environment's credentials or permissions before retrying. Automatic retries are stopped."
|
||||
: sourceFailureCode === "native_provider_terminal_failed"
|
||||
? "Verify that the failed provider stopped and reconcile its action outcomes. A linked continuation can proceed only after these checks succeed."
|
||||
: sourceFailureCode === "native_provider_model_rejected" && !recoveryEvidence.providerSessionEstablished
|
||||
? NATIVE_STARTUP_MODEL_REJECTED_ACTION
|
||||
: sourceFailureCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE
|
||||
? NATIVE_PROVIDER_STATE_INCOMPATIBLE_ACTION
|
||||
: recoveryEvidence.recoveryMode === "ambiguous_state"
|
||||
|
||||
@@ -38,6 +38,7 @@ import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
|
||||
import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentRunnerIngressError } from "@paperclipai/plugin-sdk";
|
||||
import { accountRunCost } from "../run-cost-accounting.js";
|
||||
import { settleUnrecoverableExecutions } from "../execution-recovery-resolution.js";
|
||||
|
||||
type SessionOpenStatus = "failed" | "rejected" | "indeterminate";
|
||||
|
||||
@@ -45,6 +46,7 @@ type SessionOpenStatus = "failed" | "rejected" | "indeterminate";
|
||||
function sessionOpenFailure(options: {
|
||||
status?: SessionOpenStatus;
|
||||
providerFailure?: Record<string, unknown>;
|
||||
classification?: string;
|
||||
} = {}) {
|
||||
const status = options.status ?? "failed";
|
||||
const result = {
|
||||
@@ -52,7 +54,7 @@ function sessionOpenFailure(options: {
|
||||
code: "command_execution_failed",
|
||||
message: options.providerFailure
|
||||
? "Provider request failed"
|
||||
: "failed to start ACPX provider: ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)",
|
||||
: `failed to start ACPX provider: ACPX sidecar command session.open was rejected (retryable=false, classification=${options.classification ?? "session_ensure_failed"})`,
|
||||
...(options.providerFailure ? { providerFailure: options.providerFailure } : {}),
|
||||
},
|
||||
status,
|
||||
@@ -476,6 +478,53 @@ describe("native provider startup retry", () => {
|
||||
expect(state.issue.status).toBe("in_progress");
|
||||
});
|
||||
|
||||
it.each(["grok_model_mismatch", "requested_model_unsupported"])(
|
||||
"stops at a session.open %s rejection with a choose-model action instead of a doomed retry",
|
||||
async (classification) => {
|
||||
const seeded = await seedRun(await createAgent());
|
||||
await failStartup(seeded, sessionOpenFailure({ classification }));
|
||||
|
||||
const state = await read(seeded);
|
||||
// No automatic same-run bootstrap retry: the selected model cannot start.
|
||||
expect(state.coordinator).toMatchObject({
|
||||
phase: "terminal_failure",
|
||||
recoveryState: "blocked",
|
||||
failureCode: "native_provider_model_rejected",
|
||||
attempt: 1,
|
||||
nextAttemptAt: null,
|
||||
failureDetail: expect.objectContaining({
|
||||
originalFailureCode: "native_provider_model_rejected",
|
||||
providerSessionEstablished: false,
|
||||
providerEventsExist: false,
|
||||
checkpointExists: false,
|
||||
}),
|
||||
});
|
||||
expect(String(state.coordinator.failureDetail?.nextAction)).toContain("Choose a supported model");
|
||||
expect(state.run).toMatchObject({ status: "failed", errorCode: "native_provider_model_rejected" });
|
||||
expect(state.issue.status).toBe("blocked");
|
||||
expect(state.issue.executionRunId).toBeNull();
|
||||
// A named, durable board action owns the next step.
|
||||
expect(state.action).toMatchObject({
|
||||
status: "active",
|
||||
ownerType: "board",
|
||||
cause: "native_provider_model_rejected",
|
||||
returnOwnerAgentId: seeded.agentId,
|
||||
wakePolicy: null,
|
||||
});
|
||||
expect(state.action!.nextAction).toContain("Choose a supported model");
|
||||
// The no-replay settler must not fold this owned action into a silent hold.
|
||||
await settleUnrecoverableExecutions(db);
|
||||
expect((await read(seeded)).action).toMatchObject({ id: state.action!.id, status: "active" });
|
||||
expect(await claimNativeSessionResumptions({
|
||||
db,
|
||||
runnerInstanceId: "startup-retry-test",
|
||||
runIds: [seeded.runId],
|
||||
now: new Date(Date.now() + 60 * 60_000),
|
||||
deferIneligible: false,
|
||||
})).toEqual([]);
|
||||
},
|
||||
);
|
||||
|
||||
it("blocks with a clear reason once the bounded startup budget is exhausted", async () => {
|
||||
const seeded = await seedRun(await createAgent());
|
||||
await failStartup(seeded, sessionOpenFailure());
|
||||
|
||||
Reference in new issue
Block a user