From fa3cd6f3302d1a7a647394d3359c869a6ca0f223 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 07:48:18 -0600 Subject: [PATCH] fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner starts Codex for native runs. On a remote sandbox such as Daytona, the sandbox is the OS boundary. > - A Codex planning run on Daytona failed in `session.open`. Codex's read-only sandbox helper could not set up loopback in the container (`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md` through that helper. > - The server treated that definite provider rejection as an interrupted transport. Its same-run bootstrap retry went through remote restart recovery and stopped on `ambiguous_remote_recovery_evidence`. The task showed "Recovering saved work…" and no Retry, although no provider work was done. > - This pull request gives such startup rejections a named, permanent outcome with a working Retry, and pins ask and standard runs to the external sandbox policy. > - The benefit is that a provider startup refusal ends as a clear failure with Retry, never as an ownership hold. ## Linked Issues or Issue Description **What happened?** A Codex planning run on a Daytona sandbox failed in `session.open` with `error creating thread: failed to load AGENTS.md instructions ... bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`. The server recorded `native_session_interrupted` with recovery mode `bootstrap_retry`. Thirty seconds later, remote restart recovery blocked the run with `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence`. No server restart happened. All default-mode Codex runs on the same image succeeded that day; only the planning run failed. **Expected behavior** A provider startup failure with no provider work must end as a clear failure with a working Retry, or get a bounded automatic retry that also ends with Retry. It must never become an ambiguous-ownership hold. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`. On master the deterministic case is `retryable_failure` / `native_session_interrupted`. 2. `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "provider itself rejected"`. On master the new code is not produced. **Paperclip version or commit** master at 84fc67317e. ## What Changed - Server (`native-session-executor.ts`): new permanent code `native_provider_startup_rejected`. - A definite `session.open` failure maps to it when its startup `providerFailure` is `requires_action` for `authentication`, `permission` or `invalid_request`. - On a remote execution target (`nativeRemoteStartupRejectionCode`), any definite `session.open` failure with a startup `providerFailure` maps to it. A remote same-run bootstrap after the runner launched cannot prove ownership and stops on `ambiguous_remote_recovery_evidence`. - The run ends as `terminal_failure` with a board owner and a plain next action. The board offers Retry, which starts one new run with a fresh provider session. - On a local target, unknown, transient, rate-limit, quota and model startup failures keep the bounded same-run retries and end as `native_session_retry_exhausted` with Retry. That code is unchanged because the chat-channel failed-run retry depends on it. - Runner: planning runs keep Codex's `paperclip-runner-workspace-read-only` profile, also in an external sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode (Conversational)") and does not block writes or commands, so it cannot replace that enforcement. - Ask mode: an ask task builds `executionMode: "default"`. Its launch has `readOnly: false`, the external sandbox profile, and no collaboration mode. Ask mode restricts only Paperclip mutation tools, not the Codex sandbox. - Tests: - `native-startup-rejected-retry.integration.test.ts` (new, embedded Postgres, Codex provider): a deterministic rejection ends at once with `canRetry: true`, and Retry starts exactly one new run. Unknown and transient local failures stay `retryable_failure`. A spent retry budget ends `terminal_failure` with `canRetry: true`. - Unit tests for the deterministic, local-retry and remote rules. - `runner-provider-launch.test.ts`: ask, standard and planning launches in an external sandbox. - The Rust external-sandbox test now fails if a non-planning turn sends a collaboration mode (`--forbid-plan-collaboration` switch on the fake app server). - Docs: `packages/paperclip-runner/docs/codex-driver.md`, `packages/paperclip-runner/docs/durable-recovery.md`, `doc/execution-semantics.md`. ## Verification - Red on master, green here: the commands in "Steps to reproduce". - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1364 pass. - `npx vitest run src/live/runner-provider-launch.test.ts` (packages/paperclip-runner): 22 pass. - `cargo test -p paperclip-runner-core --test codex_provider external_sandbox`: pass. `cargo fmt --check`: clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify` returned exit 2 (a global CI input changed). Required CI supplies the broad coverage. ## Risks - Codex planning on Daytona still fails at thread creation. With this change it ends at once with the provider message and Retry instead of an ownership hold. A follow-up must make Codex's read-only sandbox work in that container, or add a verified Paperclip read-only guard. This PR does not relax read-only enforcement. - A remote startup failure that could clear by itself now needs a Retry click instead of an automatic same-run retry. Before this change, that retry ended in an ownership hold, so no working automatic path is lost. - A false `native_provider_startup_rejected` costs automatic retries and shows Retry earlier. It never replays provider work, because `session.open` precedes every prompt. - Saved state: no persisted shape or launch argument changes. A run that an older release held on `ambiguous_remote_recovery_evidence` after the same rejection already offers Retry once its saved workspace is recovered. No migration is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 11 +- .../paperclip-runner/docs/codex-driver.md | 11 ++ .../paperclip-runner/docs/durable-recovery.md | 14 ++ .../src/bin/fake-codex-app-server.rs | 8 ++ .../runner-core/tests/codex_provider.rs | 6 +- .../src/live/runner-provider-launch.test.ts | 17 +++ .../native-session-executor.test.ts | 58 +++++++++ .../native-runtime/native-session-executor.ts | 32 ++++- ...startup-rejected-retry.integration.test.ts | 121 ++++++++++++++++++ 9 files changed, 273 insertions(+), 5 deletions(-) create mode 100644 server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 8916c01d62e..1812b0839ae 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1355,8 +1355,15 @@ provider cannot run the selected model (`requested_model_unsupported`, `grok_model_mismatch`) is a model rejection, not a startup interruption. It stops on its first attempt as `native_provider_model_rejected`, with no same-run bootstrap retry, and leaves an active board recovery action that asks -for a supported model. Other `session.open` rejections keep the bounded -startup retry. An indeterminate `session.open` outcome stays unclassified. +for a supported model. A definite rejection whose startup `providerFailure` +is `requires_action` for `authentication`, `permission` or `invalid_request` +stops on its first attempt as `native_provider_startup_rejected`. On a remote +execution target, any definite rejection with a startup `providerFailure` stops +the same way, because a remote same-run bootstrap cannot prove ownership. No +provider work began, so the board offers Retry, which starts a new run with a +fresh provider session. Other local `session.open` rejections keep the bounded +startup retry and end with Retry when it is spent. An +indeterminate `session.open` outcome stays unclassified. ### Workspace scan failures before provider startup diff --git a/packages/paperclip-runner/docs/codex-driver.md b/packages/paperclip-runner/docs/codex-driver.md index a3021d88e67..037c48349ed 100644 --- a/packages/paperclip-runner/docs/codex-driver.md +++ b/packages/paperclip-runner/docs/codex-driver.md @@ -82,6 +82,17 @@ valid, but when `PAPERCLIP_WORKSPACE_CWD` is present its canonical path must be equal to or below that assigned workspace so sibling and symlink escapes fail before provider startup. +When the execution target is itself the sandbox (a remote sandbox such as +Daytona sets `PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`), standard and ask runs use +the default execution mode. The Runner starts their Codex threads and turns +with the external sandbox policy and sends no collaboration mode. Ask mode +restricts Paperclip mutation tools, not the Codex sandbox. Planning runs keep +the `paperclip-runner-workspace-read-only` profile everywhere, because Codex's +plan collaboration mode is a prompt and does not block writes or commands. +Inside a container without network-namespace rights, that profile's helper can +fail thread creation (`bwrap: loopback: Failed RTM_NEWADDR`); the run then ends +as a provider startup failure with Retry (see `durable-recovery.md`). + The returned sandbox facts remain authoritative. Codex 0.132.0 may inject a provider-managed writable root such as `~/.codex/memories` after a first run, even with `features.memories=false` and an explicit Codex-home deny. That makes diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 5d021ed8172..60200f44237 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -151,6 +151,20 @@ which proves that the sidecar and the provider are gone. the saved task context. An older Runner sends the same text without the class; the controller then keeps its bounded same-run retries, because the same text also covers a session that can reopen after a restart. +- A definite `session.open` failure whose command result carries a startup + `providerFailure` with `retryability: requires_action` and category + `authentication`, `permission` or `invalid_request` records + `native_provider_startup_rejected`. No provider work began, so the + controller stops same-run bootstrap retries and the board offers Retry at once + with the provider message. On a remote execution target, every definite + `session.open` failure with a startup `providerFailure` records the same code, + because a remote same-run bootstrap after the runner launched cannot prove + ownership and would stop on `ambiguous_remote_recovery_evidence`. On a local + target, other startup failures (unknown, transient, rate limit, quota, model) + keep the bounded same-run startup retries and end as + `native_session_retry_exhausted` with Retry. A run an older release held on + `ambiguous_remote_recovery_evidence` after such a rejection offers Retry once + its saved workspace is recovered; no migration is needed. Until a proof settles it, the saved session stays `turn_starting`, so a Runner killed during the proof restores through the same check. The cut-off command is diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs index 2fe4ddc0c13..194f81ad1a4 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs @@ -803,6 +803,9 @@ fn run() -> Result<(), Box> { let require_external_sandbox = args .iter() .any(|value| value == "--require-external-sandbox"); + let forbid_plan_collaboration = args + .iter() + .any(|value| value == "--forbid-plan-collaboration"); let expected_canonical_task_context = argument(&args, "--expected-canonical-task-context") .map(|value| serde_json::from_str::(&value)) .transpose()?; @@ -1756,6 +1759,11 @@ fn run() -> Result<(), Box> { { return Err("turn/start omitted the external sandbox boundary".into()); } + if forbid_plan_collaboration + && message.pointer("/params/collaborationMode").is_some() + { + return Err("turn/start requested a collaboration mode outside planning".into()); + } if let Some(pid_file) = escaped_tool_pid_file.as_deref() { if turn_start_count == 0 { std::process::Command::new("perl") diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs index c0122536536..30fea94da43 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs @@ -632,8 +632,12 @@ fn ordinary_recovery_preserves_command_replay_and_collision_checks() { #[test] fn delegates_command_isolation_to_an_explicit_external_sandbox() { let directory = temporary_directory("external-sandbox"); - let mut config = provider_config(&directory, &["--require-external-sandbox"]); + let mut config = provider_config( + &directory, + &["--require-external-sandbox", "--forbid-plan-collaboration"], + ); config.externally_sandboxed = true; + config.read_only = false; let mut provider = CodexProvider::start(&config, None) .expect("start Codex with an externally owned sandbox boundary"); diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts index 62183f9a5e8..0c240356ac6 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { createRunnerProviderLaunch, preparedRunnerLaunchWorkingDirectory, runnerLaunchRequirementsDigest, runnerLaunchSettingsMatch, validatePreparedRunnerLaunch, runnerProviderLaunchRequirements, type RunnerProviderLaunchEnvironment, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js"; +import { createSecuredCodexThreadParams } from "../drivers/codex/codex-security-config.js"; const requirements: RunnerProviderLaunchRequirements = { schema: "paperclip.runner.launch-requirements.v1", @@ -27,6 +28,22 @@ describe("built-in provider launch preparation", () => { expect(frozen.settings).toEqual({}); }); + it.each([ + { workMode: "ask", executionMode: "default", readOnly: false, collaborationMode: "default" }, + { workMode: "standard", executionMode: "default", readOnly: false, collaborationMode: "default" }, + { workMode: "planning", executionMode: "plan", readOnly: true, collaborationMode: "plan" }, + ] as const)("launches a $workMode Codex run in an external sandbox without a read-only Codex sandbox claim", (mode) => { + const thread = createSecuredCodexThreadParams("/execution/workspace", mode.executionMode, true, false, + { PAPERCLIP_RUNNER_EXTERNAL_SANDBOX: "1" }); + const selected = runnerProviderLaunchRequirements({ provider: "codex", externallySandboxed: true }, + requirements.identity, { model: requirements.model, ...thread }, null, ""); + expect(selected.readOnly).toBe(mode.readOnly); + const launch = createRunnerProviderLaunch(selected, { ...environment, + options: { ...environment.options, externallySandboxed: true } }); + expect(launch).toMatchObject({ externallySandboxed: true, collaborationMode: mode.collaborationMode }); + expect("readOnly" in launch).toBe(mode.readOnly); + }); + it("preserves probe project-instruction suppression through a prepared launch", () => { const selected = runnerProviderLaunchRequirements({ provider: "codex", disableCodexProjectInstructions: true, disableCodexWebSearch: true }, requirements.identity, diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 09fff02dfce..654dba53d8d 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -435,6 +435,8 @@ import { nativeProviderUsageLimitFromEvent, nativeSessionFailureSourceCode, NATIVE_PROVIDER_SESSION_CLOSED, + NATIVE_PROVIDER_STARTUP_REJECTED, + nativeRemoteStartupRejectionCode, nativeSessionRecoveryProjection, nativeGovernedWaitResult, nativeConversationReplyResult, @@ -14060,6 +14062,62 @@ describe("native session bounded recovery", () => { }); }); + describe("a session.open the provider itself rejected before any provider work", () => { + const sessionOpenFailure = (providerFailure: Record) => { + const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2, + result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" }; + const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } }; + return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`), + "runnerCommandResult", { value: commandResult, enumerable: false }); + }; + const startupFailure = (fields: Record) => ({ schema: "paperclip.provider-failure.v1", provider: "openai", + phase: "startup", category: "unknown", retryability: "unknown", + message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted", + ...fields }); + + it.each(["authentication", "permission", "invalid_request"])("stops on the first %s rejection and hands the run to the board", (category) => { + const code = nativeSessionFailureSourceCode(sessionOpenFailure(startupFailure({ category, retryability: "requires_action" }))); + expect(code).toBe(NATIVE_PROVIDER_STARTUP_REJECTED); + const disposition = nativeSessionFailureDisposition(1, new Date(), code); + expect(disposition).toEqual({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null }); + expect(nativeSessionRecoveryProjection({ ...disposition, agentId: "agent" })).toMatchObject({ + issueStatus: "blocked", recoveryOwner: { kind: "board" }, + }); + }); + + it("keeps bounded same-run retries for a local startup failure without a definite permanent cause", () => { + for (const fields of [ + {}, + { message: "Query closed before response received" }, + { category: "overloaded", retryability: "transient", httpStatus: 503 }, + { category: "timeout", retryability: "transient", httpStatus: 504 }, + { category: "rate_limit", httpStatus: 429 }, + { category: "quota_exhausted", retryability: "requires_action", providerCode: "usageLimitExceeded" }, + { category: "model_unavailable", retryability: "requires_action" }, + { phase: "turn", category: "authentication", retryability: "requires_action" }, + ]) { + const failure = sessionOpenFailure(startupFailure(fields)); + const code = nativeSessionFailureSourceCode(failure); + expect(code).toBe("provider_session_open_rejected"); + expect(nativeRemoteStartupRejectionCode(code, failure, false)).toBe("provider_session_open_rejected"); + } + expect(nativeSessionFailureDisposition(2, new Date(), "provider_session_open_rejected")).toMatchObject({ phase: "retryable_failure" }); + expect(nativeSessionFailureDisposition(3, new Date(), "provider_session_open_rejected")).toEqual({ + phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null, + }); + }); + + it("stops a remote startup rejection at once because a remote same-run bootstrap cannot be verified", () => { + const failure = sessionOpenFailure(startupFailure({ message: "Query closed before response received" })); + expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(failure), failure, true)) + .toBe(NATIVE_PROVIDER_STARTUP_REJECTED); + const unreported = new Error(`PRP command session.open failed: {"result":{"code":"command_execution_failed","message":"Provider request failed"},"status":"failed"}`); + expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(unreported), unreported, true)) + .toBe("provider_session_open_rejected"); + expect(nativeRemoteStartupRejectionCode("native_session_interrupted", failure, true)).toBe("native_session_interrupted"); + }); + }); + it("stops same-run retries for an operator while a saved side-effecting call has no result", () => { const failure = new NativeProviderSideEffectPendingError( "Error: native_runner_source_semantic_reconciliation_required", diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 2029f7c7442..6c7b4b5685d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -8443,7 +8443,8 @@ export function nativeSessionFailureDisposition( sourceFailureCode === "native_session_cleanup_quarantined" || sourceFailureCode === "native_adopted_runner_authentication_timeout" || sourceFailureCode === "native_provider_usage_limit" || - sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED; + sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED || + sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED; const permissionDeclined = sourceFailureCode === "native_permission_declined"; const exhausted = permanentFailure || permissionDeclined || attempt >= 3; return { @@ -8496,6 +8497,26 @@ export const NATIVE_PROVIDER_SESSION_CLOSED_ACTION = "The provider session closed after a turn start that may have reached the provider, and this run cannot reopen it. Retry starts a new run with a fresh provider conversation and the saved task context."; const RUNNER_PROVIDER_SESSION_CLOSED = "provider_session_closed"; +export const NATIVE_PROVIDER_STARTUP_REJECTED = "native_provider_startup_rejected"; +export const NATIVE_PROVIDER_STARTUP_REJECTED_ACTION = + "The provider refused to start a session, so no provider work began. Read the provider message and fix its cause in the agent's environment or AI connection if needed, then Retry. Retry starts a new run with a fresh provider session."; + +function deterministicProviderStartupFailure(error: unknown): boolean { + const failure = providerFailureFromError(error); + return failure?.phase === "startup" && failure.retryability === "requires_action" + && ["authentication", "permission", "invalid_request"].includes(failure.category); +} + +export function nativeRemoteStartupRejectionCode( + sourceFailureCode: Code, + error: unknown, + remoteTarget: boolean, +): Code | typeof NATIVE_PROVIDER_STARTUP_REJECTED { + return remoteTarget && sourceFailureCode === "provider_session_open_rejected" + && providerFailureFromError(error)?.phase === "startup" + ? NATIVE_PROVIDER_STARTUP_REJECTED : sourceFailureCode; +} + /** Reads the Runner's closed class from an authenticated turn.start failure. */ function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean { if (!error || typeof error !== "object" || depth > 5) return false; @@ -8522,6 +8543,7 @@ export function nativeSessionFailureSourceCode( | "native_provider_approval_required" | "native_provider_usage_limit" | typeof NATIVE_PROVIDER_SESSION_CLOSED + | typeof NATIVE_PROVIDER_STARTUP_REJECTED | "native_session_cleanup_quarantined" | "native_adopted_runner_authentication_timeout" | "runner_remote_provider_artifact_incompatible" @@ -8663,6 +8685,7 @@ export function nativeSessionFailureSourceCode( if (/classification=(?:requested_model_unsupported|grok_model_mismatch)\)/.test(message)) { return "native_provider_model_rejected"; } + if (deterministicProviderStartupFailure(error)) return NATIVE_PROVIDER_STARTUP_REJECTED; return "provider_session_open_rejected"; } // The controller stopped waiting for session.open while runner startup was @@ -8811,6 +8834,7 @@ export async function nativeProviderRecoveryEvidence(input: { "provider_request_protocol_error", "native_runner_process_exited", "provider_session_open_rejected", + NATIVE_PROVIDER_STARTUP_REJECTED, "provider_session_open_timeout", "local_runtime_api_listener_unavailable", ]).has(input.sourceFailureCode); @@ -12297,7 +12321,7 @@ async function executePaperclipNativeSessionAttempt( ? classifiedFailureCode : providerUsageLimitObserved ? "native_provider_usage_limit" - : classifiedFailureCode; + : nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote"); const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote" && !input.restartRecovery && !retainedTransition && !remoteWarmRecovery ? { @@ -12456,6 +12480,8 @@ async function executePaperclipNativeSessionAttempt( ? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler." : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION + : sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED + ? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION : sourceFailureCode === "native_session_cleanup_quarantined" ? nativeCleanupRefusalAction(completedTurnCleanup) : sourceFailureCode === "runner_remote_provider_verification_unavailable" @@ -12731,6 +12757,8 @@ async function executePaperclipNativeSessionAttempt( ? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler." : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION + : sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED + ? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION : sourceFailureCode === "native_session_cleanup_quarantined" ? nativeCleanupRefusalAction(completedTurnCleanup) : sourceFailureCode === "runner_task_admission_rejected" diff --git a/server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts b/server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts new file mode 100644 index 00000000000..20586d0b943 --- /dev/null +++ b/server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts @@ -0,0 +1,121 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db"; +import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; + +const provider = vi.hoisted(() => ({ execute: vi.fn() })); +vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({ + ...(await importOriginal()), + executeNativeSession: provider.execute, +})); + +import { executePaperclipNativeSession, NATIVE_PROVIDER_STARTUP_REJECTED } from "./native-session-executor.js"; +import { buildNativeCompletionContract } from "./completion-contracts.js"; +import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; +import { heartbeatService } from "../heartbeat.js"; +import { requestTaskRetry } from "../task-retry.js"; +import { taskInterruptionPresentation } from "../task-interruption-presentation.js"; + +describe("native session.open the provider rejected before any provider work", () => { + let temporary: Awaited>; + let db: ReturnType; + const companyId = randomUUID(); + const agentId = randomUUID(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-startup-rejected-"); + db = createDb(temporary.connectionString); + await db.insert(companies).values({ id: companyId, name: "Startup rejected", issuePrefix: "NSR" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Codex agent", status: "active", adapterType: "paperclip_runner", + adapterConfig: { provider: "codex", model: "gpt-test" } }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" }); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + const rejection = (providerFailure: Record) => { + const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2, + result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" }; + const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } }; + return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`), + "runnerCommandResult", { value: commandResult, enumerable: false }); + }; + const startupFailure = (fields: Record = {}) => ({ schema: "paperclip.provider-failure.v1", provider: "openai", + phase: "startup", category: "unknown", retryability: "unknown", + message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted", + ...fields }); + + async function rejectSessionOpen(failure: Error, attempt = 0) { + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Plan the change", status: "in_progress", workMode: "standard", + assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + const runId = randomUUID(); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "assignment", triggerDetail: "system", + contextSnapshot: { issueId } }).returning(); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() }); + const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId)); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt }); + const execution: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", + binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId }, + provider: { kind: "codex", model: "gpt-test" }, + task: { identifier: issueId, title: issue.title, description: null, prompt: "Plan the change.", workMode: "standard" }, + workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + interactionResponses: [], credentialBindings: [], + }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId)); + provider.execute.mockReset().mockRejectedValue(failure); + await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { + descriptor: async () => { throw new Error("Unexpected provider call"); }, + openSession: async () => { throw new Error("Unexpected provider call"); }, + } }).catch(() => undefined); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + return { issueId, runId, coordinator }; + } + + it("fails the run on the first deterministic rejection with a plain explanation, and Retry starts a new run", async () => { + const { issueId, runId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ category: "authentication", + retryability: "requires_action", message: "error creating thread: the AI connection was revoked" }))); + expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null, + failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry", providerSessionEstablished: false, + providerFailure: expect.objectContaining({ phase: "startup" }), + nextAction: expect.stringContaining("Retry starts a new run") }) }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true, + providerFailure: expect.objectContaining({ phase: "startup" }) }); + const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session", + idempotencyKey: randomUUID(), observedRunId: runId }); + expect(receipt).toMatchObject({ accepted: true, stale: false }); + const heartbeat = heartbeatService(db); + await heartbeat.resumeTaskRetry(companyId, receipt.requestId!); + const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId))) + .filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId); + expect(retried).toHaveLength(1); + await heartbeat.drainActiveRunExecutions(); + }, 120_000); + + it.each([ + startupFailure(), + startupFailure({ message: "Query closed before response received" }), + startupFailure({ category: "overloaded", retryability: "transient", httpStatus: 503 }), + ])("retries an unknown or transient local startup failure on the same run: $message", async (failure) => { + const { coordinator } = await rejectSessionOpen(rejection(failure)); + expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted", + failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry" }) }); + }, 120_000); + + it("ends with Retry once the bounded startup retries are spent", async () => { + const { issueId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ message: "Query closed before response received" })), 2); + expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null, + failureDetail: expect.objectContaining({ nextAction: expect.stringContaining("retry the task") }) }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true }); + }, 120_000); +});