diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 8916c01d62e..1812b0839ae 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1355,8 +1355,15 @@ provider cannot run the selected model (`requested_model_unsupported`, `grok_model_mismatch`) is a model rejection, not a startup interruption. It stops on its first attempt as `native_provider_model_rejected`, with no same-run bootstrap retry, and leaves an active board recovery action that asks -for a supported model. Other `session.open` rejections keep the bounded -startup retry. An indeterminate `session.open` outcome stays unclassified. +for a supported model. A definite rejection whose startup `providerFailure` +is `requires_action` for `authentication`, `permission` or `invalid_request` +stops on its first attempt as `native_provider_startup_rejected`. On a remote +execution target, any definite rejection with a startup `providerFailure` stops +the same way, because a remote same-run bootstrap cannot prove ownership. No +provider work began, so the board offers Retry, which starts a new run with a +fresh provider session. Other local `session.open` rejections keep the bounded +startup retry and end with Retry when it is spent. An +indeterminate `session.open` outcome stays unclassified. ### Workspace scan failures before provider startup diff --git a/packages/paperclip-runner/docs/codex-driver.md b/packages/paperclip-runner/docs/codex-driver.md index a3021d88e67..037c48349ed 100644 --- a/packages/paperclip-runner/docs/codex-driver.md +++ b/packages/paperclip-runner/docs/codex-driver.md @@ -82,6 +82,17 @@ valid, but when `PAPERCLIP_WORKSPACE_CWD` is present its canonical path must be equal to or below that assigned workspace so sibling and symlink escapes fail before provider startup. +When the execution target is itself the sandbox (a remote sandbox such as +Daytona sets `PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`), standard and ask runs use +the default execution mode. The Runner starts their Codex threads and turns +with the external sandbox policy and sends no collaboration mode. Ask mode +restricts Paperclip mutation tools, not the Codex sandbox. Planning runs keep +the `paperclip-runner-workspace-read-only` profile everywhere, because Codex's +plan collaboration mode is a prompt and does not block writes or commands. +Inside a container without network-namespace rights, that profile's helper can +fail thread creation (`bwrap: loopback: Failed RTM_NEWADDR`); the run then ends +as a provider startup failure with Retry (see `durable-recovery.md`). + The returned sandbox facts remain authoritative. Codex 0.132.0 may inject a provider-managed writable root such as `~/.codex/memories` after a first run, even with `features.memories=false` and an explicit Codex-home deny. That makes diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 5d021ed8172..60200f44237 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -151,6 +151,20 @@ which proves that the sidecar and the provider are gone. the saved task context. An older Runner sends the same text without the class; the controller then keeps its bounded same-run retries, because the same text also covers a session that can reopen after a restart. +- A definite `session.open` failure whose command result carries a startup + `providerFailure` with `retryability: requires_action` and category + `authentication`, `permission` or `invalid_request` records + `native_provider_startup_rejected`. No provider work began, so the + controller stops same-run bootstrap retries and the board offers Retry at once + with the provider message. On a remote execution target, every definite + `session.open` failure with a startup `providerFailure` records the same code, + because a remote same-run bootstrap after the runner launched cannot prove + ownership and would stop on `ambiguous_remote_recovery_evidence`. On a local + target, other startup failures (unknown, transient, rate limit, quota, model) + keep the bounded same-run startup retries and end as + `native_session_retry_exhausted` with Retry. A run an older release held on + `ambiguous_remote_recovery_evidence` after such a rejection offers Retry once + its saved workspace is recovered; no migration is needed. Until a proof settles it, the saved session stays `turn_starting`, so a Runner killed during the proof restores through the same check. The cut-off command is diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs index 2fe4ddc0c13..194f81ad1a4 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs @@ -803,6 +803,9 @@ fn run() -> Result<(), Box> { let require_external_sandbox = args .iter() .any(|value| value == "--require-external-sandbox"); + let forbid_plan_collaboration = args + .iter() + .any(|value| value == "--forbid-plan-collaboration"); let expected_canonical_task_context = argument(&args, "--expected-canonical-task-context") .map(|value| serde_json::from_str::(&value)) .transpose()?; @@ -1756,6 +1759,11 @@ fn run() -> Result<(), Box> { { return Err("turn/start omitted the external sandbox boundary".into()); } + if forbid_plan_collaboration + && message.pointer("/params/collaborationMode").is_some() + { + return Err("turn/start requested a collaboration mode outside planning".into()); + } if let Some(pid_file) = escaped_tool_pid_file.as_deref() { if turn_start_count == 0 { std::process::Command::new("perl") diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs index c0122536536..30fea94da43 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs @@ -632,8 +632,12 @@ fn ordinary_recovery_preserves_command_replay_and_collision_checks() { #[test] fn delegates_command_isolation_to_an_explicit_external_sandbox() { let directory = temporary_directory("external-sandbox"); - let mut config = provider_config(&directory, &["--require-external-sandbox"]); + let mut config = provider_config( + &directory, + &["--require-external-sandbox", "--forbid-plan-collaboration"], + ); config.externally_sandboxed = true; + config.read_only = false; let mut provider = CodexProvider::start(&config, None) .expect("start Codex with an externally owned sandbox boundary"); diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts index 62183f9a5e8..0c240356ac6 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { createRunnerProviderLaunch, preparedRunnerLaunchWorkingDirectory, runnerLaunchRequirementsDigest, runnerLaunchSettingsMatch, validatePreparedRunnerLaunch, runnerProviderLaunchRequirements, type RunnerProviderLaunchEnvironment, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js"; +import { createSecuredCodexThreadParams } from "../drivers/codex/codex-security-config.js"; const requirements: RunnerProviderLaunchRequirements = { schema: "paperclip.runner.launch-requirements.v1", @@ -27,6 +28,22 @@ describe("built-in provider launch preparation", () => { expect(frozen.settings).toEqual({}); }); + it.each([ + { workMode: "ask", executionMode: "default", readOnly: false, collaborationMode: "default" }, + { workMode: "standard", executionMode: "default", readOnly: false, collaborationMode: "default" }, + { workMode: "planning", executionMode: "plan", readOnly: true, collaborationMode: "plan" }, + ] as const)("launches a $workMode Codex run in an external sandbox without a read-only Codex sandbox claim", (mode) => { + const thread = createSecuredCodexThreadParams("/execution/workspace", mode.executionMode, true, false, + { PAPERCLIP_RUNNER_EXTERNAL_SANDBOX: "1" }); + const selected = runnerProviderLaunchRequirements({ provider: "codex", externallySandboxed: true }, + requirements.identity, { model: requirements.model, ...thread }, null, ""); + expect(selected.readOnly).toBe(mode.readOnly); + const launch = createRunnerProviderLaunch(selected, { ...environment, + options: { ...environment.options, externallySandboxed: true } }); + expect(launch).toMatchObject({ externallySandboxed: true, collaborationMode: mode.collaborationMode }); + expect("readOnly" in launch).toBe(mode.readOnly); + }); + it("preserves probe project-instruction suppression through a prepared launch", () => { const selected = runnerProviderLaunchRequirements({ provider: "codex", disableCodexProjectInstructions: true, disableCodexWebSearch: true }, requirements.identity, diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 09fff02dfce..654dba53d8d 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -435,6 +435,8 @@ import { nativeProviderUsageLimitFromEvent, nativeSessionFailureSourceCode, NATIVE_PROVIDER_SESSION_CLOSED, + NATIVE_PROVIDER_STARTUP_REJECTED, + nativeRemoteStartupRejectionCode, nativeSessionRecoveryProjection, nativeGovernedWaitResult, nativeConversationReplyResult, @@ -14060,6 +14062,62 @@ describe("native session bounded recovery", () => { }); }); + describe("a session.open the provider itself rejected before any provider work", () => { + const sessionOpenFailure = (providerFailure: Record) => { + const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2, + result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" }; + const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } }; + return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`), + "runnerCommandResult", { value: commandResult, enumerable: false }); + }; + const startupFailure = (fields: Record) => ({ schema: "paperclip.provider-failure.v1", provider: "openai", + phase: "startup", category: "unknown", retryability: "unknown", + message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted", + ...fields }); + + it.each(["authentication", "permission", "invalid_request"])("stops on the first %s rejection and hands the run to the board", (category) => { + const code = nativeSessionFailureSourceCode(sessionOpenFailure(startupFailure({ category, retryability: "requires_action" }))); + expect(code).toBe(NATIVE_PROVIDER_STARTUP_REJECTED); + const disposition = nativeSessionFailureDisposition(1, new Date(), code); + expect(disposition).toEqual({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null }); + expect(nativeSessionRecoveryProjection({ ...disposition, agentId: "agent" })).toMatchObject({ + issueStatus: "blocked", recoveryOwner: { kind: "board" }, + }); + }); + + it("keeps bounded same-run retries for a local startup failure without a definite permanent cause", () => { + for (const fields of [ + {}, + { message: "Query closed before response received" }, + { category: "overloaded", retryability: "transient", httpStatus: 503 }, + { category: "timeout", retryability: "transient", httpStatus: 504 }, + { category: "rate_limit", httpStatus: 429 }, + { category: "quota_exhausted", retryability: "requires_action", providerCode: "usageLimitExceeded" }, + { category: "model_unavailable", retryability: "requires_action" }, + { phase: "turn", category: "authentication", retryability: "requires_action" }, + ]) { + const failure = sessionOpenFailure(startupFailure(fields)); + const code = nativeSessionFailureSourceCode(failure); + expect(code).toBe("provider_session_open_rejected"); + expect(nativeRemoteStartupRejectionCode(code, failure, false)).toBe("provider_session_open_rejected"); + } + expect(nativeSessionFailureDisposition(2, new Date(), "provider_session_open_rejected")).toMatchObject({ phase: "retryable_failure" }); + expect(nativeSessionFailureDisposition(3, new Date(), "provider_session_open_rejected")).toEqual({ + phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null, + }); + }); + + it("stops a remote startup rejection at once because a remote same-run bootstrap cannot be verified", () => { + const failure = sessionOpenFailure(startupFailure({ message: "Query closed before response received" })); + expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(failure), failure, true)) + .toBe(NATIVE_PROVIDER_STARTUP_REJECTED); + const unreported = new Error(`PRP command session.open failed: {"result":{"code":"command_execution_failed","message":"Provider request failed"},"status":"failed"}`); + expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(unreported), unreported, true)) + .toBe("provider_session_open_rejected"); + expect(nativeRemoteStartupRejectionCode("native_session_interrupted", failure, true)).toBe("native_session_interrupted"); + }); + }); + it("stops same-run retries for an operator while a saved side-effecting call has no result", () => { const failure = new NativeProviderSideEffectPendingError( "Error: native_runner_source_semantic_reconciliation_required", diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 2029f7c7442..6c7b4b5685d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -8443,7 +8443,8 @@ export function nativeSessionFailureDisposition( sourceFailureCode === "native_session_cleanup_quarantined" || sourceFailureCode === "native_adopted_runner_authentication_timeout" || sourceFailureCode === "native_provider_usage_limit" || - sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED; + sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED || + sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED; const permissionDeclined = sourceFailureCode === "native_permission_declined"; const exhausted = permanentFailure || permissionDeclined || attempt >= 3; return { @@ -8496,6 +8497,26 @@ export const NATIVE_PROVIDER_SESSION_CLOSED_ACTION = "The provider session closed after a turn start that may have reached the provider, and this run cannot reopen it. Retry starts a new run with a fresh provider conversation and the saved task context."; const RUNNER_PROVIDER_SESSION_CLOSED = "provider_session_closed"; +export const NATIVE_PROVIDER_STARTUP_REJECTED = "native_provider_startup_rejected"; +export const NATIVE_PROVIDER_STARTUP_REJECTED_ACTION = + "The provider refused to start a session, so no provider work began. Read the provider message and fix its cause in the agent's environment or AI connection if needed, then Retry. Retry starts a new run with a fresh provider session."; + +function deterministicProviderStartupFailure(error: unknown): boolean { + const failure = providerFailureFromError(error); + return failure?.phase === "startup" && failure.retryability === "requires_action" + && ["authentication", "permission", "invalid_request"].includes(failure.category); +} + +export function nativeRemoteStartupRejectionCode( + sourceFailureCode: Code, + error: unknown, + remoteTarget: boolean, +): Code | typeof NATIVE_PROVIDER_STARTUP_REJECTED { + return remoteTarget && sourceFailureCode === "provider_session_open_rejected" + && providerFailureFromError(error)?.phase === "startup" + ? NATIVE_PROVIDER_STARTUP_REJECTED : sourceFailureCode; +} + /** Reads the Runner's closed class from an authenticated turn.start failure. */ function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean { if (!error || typeof error !== "object" || depth > 5) return false; @@ -8522,6 +8543,7 @@ export function nativeSessionFailureSourceCode( | "native_provider_approval_required" | "native_provider_usage_limit" | typeof NATIVE_PROVIDER_SESSION_CLOSED + | typeof NATIVE_PROVIDER_STARTUP_REJECTED | "native_session_cleanup_quarantined" | "native_adopted_runner_authentication_timeout" | "runner_remote_provider_artifact_incompatible" @@ -8663,6 +8685,7 @@ export function nativeSessionFailureSourceCode( if (/classification=(?:requested_model_unsupported|grok_model_mismatch)\)/.test(message)) { return "native_provider_model_rejected"; } + if (deterministicProviderStartupFailure(error)) return NATIVE_PROVIDER_STARTUP_REJECTED; return "provider_session_open_rejected"; } // The controller stopped waiting for session.open while runner startup was @@ -8811,6 +8834,7 @@ export async function nativeProviderRecoveryEvidence(input: { "provider_request_protocol_error", "native_runner_process_exited", "provider_session_open_rejected", + NATIVE_PROVIDER_STARTUP_REJECTED, "provider_session_open_timeout", "local_runtime_api_listener_unavailable", ]).has(input.sourceFailureCode); @@ -12297,7 +12321,7 @@ async function executePaperclipNativeSessionAttempt( ? classifiedFailureCode : providerUsageLimitObserved ? "native_provider_usage_limit" - : classifiedFailureCode; + : nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote"); const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote" && !input.restartRecovery && !retainedTransition && !remoteWarmRecovery ? { @@ -12456,6 +12480,8 @@ async function executePaperclipNativeSessionAttempt( ? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler." : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION + : sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED + ? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION : sourceFailureCode === "native_session_cleanup_quarantined" ? nativeCleanupRefusalAction(completedTurnCleanup) : sourceFailureCode === "runner_remote_provider_verification_unavailable" @@ -12731,6 +12757,8 @@ async function executePaperclipNativeSessionAttempt( ? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler." : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION + : sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED + ? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION : sourceFailureCode === "native_session_cleanup_quarantined" ? nativeCleanupRefusalAction(completedTurnCleanup) : sourceFailureCode === "runner_task_admission_rejected" diff --git a/server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts b/server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts new file mode 100644 index 00000000000..20586d0b943 --- /dev/null +++ b/server/src/services/native-runtime/native-startup-rejected-retry.integration.test.ts @@ -0,0 +1,121 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db"; +import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; + +const provider = vi.hoisted(() => ({ execute: vi.fn() })); +vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({ + ...(await importOriginal()), + executeNativeSession: provider.execute, +})); + +import { executePaperclipNativeSession, NATIVE_PROVIDER_STARTUP_REJECTED } from "./native-session-executor.js"; +import { buildNativeCompletionContract } from "./completion-contracts.js"; +import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; +import { heartbeatService } from "../heartbeat.js"; +import { requestTaskRetry } from "../task-retry.js"; +import { taskInterruptionPresentation } from "../task-interruption-presentation.js"; + +describe("native session.open the provider rejected before any provider work", () => { + let temporary: Awaited>; + let db: ReturnType; + const companyId = randomUUID(); + const agentId = randomUUID(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-startup-rejected-"); + db = createDb(temporary.connectionString); + await db.insert(companies).values({ id: companyId, name: "Startup rejected", issuePrefix: "NSR" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Codex agent", status: "active", adapterType: "paperclip_runner", + adapterConfig: { provider: "codex", model: "gpt-test" } }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" }); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + const rejection = (providerFailure: Record) => { + const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2, + result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" }; + const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } }; + return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`), + "runnerCommandResult", { value: commandResult, enumerable: false }); + }; + const startupFailure = (fields: Record = {}) => ({ schema: "paperclip.provider-failure.v1", provider: "openai", + phase: "startup", category: "unknown", retryability: "unknown", + message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted", + ...fields }); + + async function rejectSessionOpen(failure: Error, attempt = 0) { + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Plan the change", status: "in_progress", workMode: "standard", + assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + const runId = randomUUID(); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "assignment", triggerDetail: "system", + contextSnapshot: { issueId } }).returning(); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() }); + const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId)); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt }); + const execution: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", + binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId }, + provider: { kind: "codex", model: "gpt-test" }, + task: { identifier: issueId, title: issue.title, description: null, prompt: "Plan the change.", workMode: "standard" }, + workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + interactionResponses: [], credentialBindings: [], + }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId)); + provider.execute.mockReset().mockRejectedValue(failure); + await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { + descriptor: async () => { throw new Error("Unexpected provider call"); }, + openSession: async () => { throw new Error("Unexpected provider call"); }, + } }).catch(() => undefined); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + return { issueId, runId, coordinator }; + } + + it("fails the run on the first deterministic rejection with a plain explanation, and Retry starts a new run", async () => { + const { issueId, runId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ category: "authentication", + retryability: "requires_action", message: "error creating thread: the AI connection was revoked" }))); + expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null, + failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry", providerSessionEstablished: false, + providerFailure: expect.objectContaining({ phase: "startup" }), + nextAction: expect.stringContaining("Retry starts a new run") }) }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true, + providerFailure: expect.objectContaining({ phase: "startup" }) }); + const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session", + idempotencyKey: randomUUID(), observedRunId: runId }); + expect(receipt).toMatchObject({ accepted: true, stale: false }); + const heartbeat = heartbeatService(db); + await heartbeat.resumeTaskRetry(companyId, receipt.requestId!); + const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId))) + .filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId); + expect(retried).toHaveLength(1); + await heartbeat.drainActiveRunExecutions(); + }, 120_000); + + it.each([ + startupFailure(), + startupFailure({ message: "Query closed before response received" }), + startupFailure({ category: "overloaded", retryability: "transient", httpStatus: 503 }), + ])("retries an unknown or transient local startup failure on the same run: $message", async (failure) => { + const { coordinator } = await rejectSessionOpen(rejection(failure)); + expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted", + failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry" }) }); + }, 120_000); + + it("ends with Retry once the bounded startup retries are spent", async () => { + const { issueId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ message: "Query closed before response received" })), 2); + expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null, + failureDetail: expect.objectContaining({ nextAction: expect.stringContaining("retry the task") }) }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true }); + }, 120_000); +});