fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.

## Linked Issues or Issue Description

**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.

**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.

**Paperclip version or commit**
master at 84fc67317e.

## What Changed

- Server (`native-session-executor.ts`): new permanent code
`native_provider_startup_rejected`.
- A definite `session.open` failure maps to it when its startup
`providerFailure` is `requires_action` for `authentication`,
`permission` or `invalid_request`.
- On a remote execution target (`nativeRemoteStartupRejectionCode`), any
definite `session.open` failure with a startup `providerFailure` maps to
it. A remote same-run bootstrap after the runner launched cannot prove
ownership and stops on `ambiguous_remote_recovery_evidence`.
- The run ends as `terminal_failure` with a board owner and a plain next
action. The board offers Retry, which starts one new run with a fresh
provider session.
- On a local target, unknown, transient, rate-limit, quota and model
startup failures keep the bounded same-run retries and end as
`native_session_retry_exhausted` with Retry. That code is unchanged
because the chat-channel failed-run retry depends on it.
- Runner: planning runs keep Codex's
`paperclip-runner-workspace-read-only` profile, also in an external
sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode
(Conversational)") and does not block writes or commands, so it cannot
replace that enforcement.
- Ask mode: an ask task builds `executionMode: "default"`. Its launch
has `readOnly: false`, the external sandbox profile, and no
collaboration mode. Ask mode restricts only Paperclip mutation tools,
not the Codex sandbox.
- Tests:
- `native-startup-rejected-retry.integration.test.ts` (new, embedded
Postgres, Codex provider): a deterministic rejection ends at once with
`canRetry: true`, and Retry starts exactly one new run. Unknown and
transient local failures stay `retryable_failure`. A spent retry budget
ends `terminal_failure` with `canRetry: true`.
  - Unit tests for the deterministic, local-retry and remote rules.
- `runner-provider-launch.test.ts`: ask, standard and planning launches
in an external sandbox.
- The Rust external-sandbox test now fails if a non-planning turn sends
a collaboration mode (`--forbid-plan-collaboration` switch on the fake
app server).
- Docs: `packages/paperclip-runner/docs/codex-driver.md`,
`packages/paperclip-runner/docs/durable-recovery.md`,
`doc/execution-semantics.md`.

## Verification

- Red on master, green here: the commands in "Steps to reproduce".
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1364 pass.
- `npx vitest run src/live/runner-provider-launch.test.ts`
(packages/paperclip-runner): 22 pass.
- `cargo test -p paperclip-runner-core --test codex_provider
external_sandbox`: pass. `cargo fmt --check`: clean.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Codex planning on Daytona still fails at thread creation. With this
change it ends at once with the provider message and Retry instead of an
ownership hold. A follow-up must make Codex's read-only sandbox work in
that container, or add a verified Paperclip read-only guard. This PR
does not relax read-only enforcement.
- A remote startup failure that could clear by itself now needs a Retry
click instead of an automatic same-run retry. Before this change, that
retry ended in an ownership hold, so no working automatic path is lost.
- A false `native_provider_startup_rejected` costs automatic retries and
shows Retry earlier. It never replays provider work, because
`session.open` precedes every prompt.
- Saved state: no persisted shape or launch argument changes. A run that
an older release held on `ambiguous_remote_recovery_evidence` after the
same rejection already offers Retry once its saved workspace is
recovered. No migration is needed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit, browser reads) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-11 07:48:18 -06:00
1 parent 6dd8611a2f
commit fa3cd6f330
9 files changed
+273 -5

No files matched your search

+9 -2
View File
@@ -1355,8 +1355,15 @@ provider cannot run the selected model (`requested_model_unsupported`,
`grok_model_mismatch`) is a model rejection, not a startup interruption. It
stops on its first attempt as `native_provider_model_rejected`, with no
same-run bootstrap retry, and leaves an active board recovery action that asks
for a supported model. Other `session.open` rejections keep the bounded
startup retry. An indeterminate `session.open` outcome stays unclassified.
for a supported model. A definite rejection whose startup `providerFailure`
is `requires_action` for `authentication`, `permission` or `invalid_request`
stops on its first attempt as `native_provider_startup_rejected`. On a remote
execution target, any definite rejection with a startup `providerFailure` stops
the same way, because a remote same-run bootstrap cannot prove ownership. No
provider work began, so the board offers Retry, which starts a new run with a
fresh provider session. Other local `session.open` rejections keep the bounded
startup retry and end with Retry when it is spent. An
indeterminate `session.open` outcome stays unclassified.
### Workspace scan failures before provider startup
@@ -82,6 +82,17 @@ valid, but when `PAPERCLIP_WORKSPACE_CWD` is present its canonical path must be
equal to or below that assigned workspace so sibling and symlink escapes fail
before provider startup.
When the execution target is itself the sandbox (a remote sandbox such as
Daytona sets `PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`), standard and ask runs use
the default execution mode. The Runner starts their Codex threads and turns
with the external sandbox policy and sends no collaboration mode. Ask mode
restricts Paperclip mutation tools, not the Codex sandbox. Planning runs keep
the `paperclip-runner-workspace-read-only` profile everywhere, because Codex's
plan collaboration mode is a prompt and does not block writes or commands.
Inside a container without network-namespace rights, that profile's helper can
fail thread creation (`bwrap: loopback: Failed RTM_NEWADDR`); the run then ends
as a provider startup failure with Retry (see `durable-recovery.md`).
The returned sandbox facts remain authoritative. Codex 0.132.0 may inject a
provider-managed writable root such as `~/.codex/memories` after a first run,
even with `features.memories=false` and an explicit Codex-home deny. That makes
@@ -151,6 +151,20 @@ which proves that the sidecar and the provider are gone.
the saved task context. An older Runner sends the same text without the class; the
controller then keeps its bounded same-run retries, because the same text
also covers a session that can reopen after a restart.
- A definite `session.open` failure whose command result carries a startup
`providerFailure` with `retryability: requires_action` and category
`authentication`, `permission` or `invalid_request` records
`native_provider_startup_rejected`. No provider work began, so the
controller stops same-run bootstrap retries and the board offers Retry at once
with the provider message. On a remote execution target, every definite
`session.open` failure with a startup `providerFailure` records the same code,
because a remote same-run bootstrap after the runner launched cannot prove
ownership and would stop on `ambiguous_remote_recovery_evidence`. On a local
target, other startup failures (unknown, transient, rate limit, quota, model)
keep the bounded same-run startup retries and end as
`native_session_retry_exhausted` with Retry. A run an older release held on
`ambiguous_remote_recovery_evidence` after such a rejection offers Retry once
its saved workspace is recovered; no migration is needed.
Until a proof settles it, the saved session stays `turn_starting`, so a Runner
killed during the proof restores through the same check. The cut-off command is
@@ -803,6 +803,9 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
let require_external_sandbox = args
.iter()
.any(|value| value == "--require-external-sandbox");
let forbid_plan_collaboration = args
.iter()
.any(|value| value == "--forbid-plan-collaboration");
let expected_canonical_task_context = argument(&args, "--expected-canonical-task-context")
.map(|value| serde_json::from_str::<Value>(&value))
.transpose()?;
@@ -1756,6 +1759,11 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
{
return Err("turn/start omitted the external sandbox boundary".into());
}
if forbid_plan_collaboration
&& message.pointer("/params/collaborationMode").is_some()
{
return Err("turn/start requested a collaboration mode outside planning".into());
}
if let Some(pid_file) = escaped_tool_pid_file.as_deref() {
if turn_start_count == 0 {
std::process::Command::new("perl")
@@ -632,8 +632,12 @@ fn ordinary_recovery_preserves_command_replay_and_collision_checks() {
#[test]
fn delegates_command_isolation_to_an_explicit_external_sandbox() {
let directory = temporary_directory("external-sandbox");
let mut config = provider_config(&directory, &["--require-external-sandbox"]);
let mut config = provider_config(
&directory,
&["--require-external-sandbox", "--forbid-plan-collaboration"],
);
config.externally_sandboxed = true;
config.read_only = false;
let mut provider = CodexProvider::start(&config, None)
.expect("start Codex with an externally owned sandbox boundary");
@@ -1,6 +1,7 @@
import { describe, expect, it } from "vitest";
import { createRunnerProviderLaunch, preparedRunnerLaunchWorkingDirectory, runnerLaunchRequirementsDigest, runnerLaunchSettingsMatch, validatePreparedRunnerLaunch,
runnerProviderLaunchRequirements, type RunnerProviderLaunchEnvironment, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js";
import { createSecuredCodexThreadParams } from "../drivers/codex/codex-security-config.js";
const requirements: RunnerProviderLaunchRequirements = {
schema: "paperclip.runner.launch-requirements.v1",
@@ -27,6 +28,22 @@ describe("built-in provider launch preparation", () => {
expect(frozen.settings).toEqual({});
});
it.each([
{ workMode: "ask", executionMode: "default", readOnly: false, collaborationMode: "default" },
{ workMode: "standard", executionMode: "default", readOnly: false, collaborationMode: "default" },
{ workMode: "planning", executionMode: "plan", readOnly: true, collaborationMode: "plan" },
] as const)("launches a $workMode Codex run in an external sandbox without a read-only Codex sandbox claim", (mode) => {
const thread = createSecuredCodexThreadParams("/execution/workspace", mode.executionMode, true, false,
{ PAPERCLIP_RUNNER_EXTERNAL_SANDBOX: "1" });
const selected = runnerProviderLaunchRequirements({ provider: "codex", externallySandboxed: true },
requirements.identity, { model: requirements.model, ...thread }, null, "");
expect(selected.readOnly).toBe(mode.readOnly);
const launch = createRunnerProviderLaunch(selected, { ...environment,
options: { ...environment.options, externallySandboxed: true } });
expect(launch).toMatchObject({ externallySandboxed: true, collaborationMode: mode.collaborationMode });
expect("readOnly" in launch).toBe(mode.readOnly);
});
it("preserves probe project-instruction suppression through a prepared launch", () => {
const selected = runnerProviderLaunchRequirements({ provider: "codex", disableCodexProjectInstructions: true,
disableCodexWebSearch: true }, requirements.identity,
@@ -435,6 +435,8 @@ import {
nativeProviderUsageLimitFromEvent,
nativeSessionFailureSourceCode,
NATIVE_PROVIDER_SESSION_CLOSED,
NATIVE_PROVIDER_STARTUP_REJECTED,
nativeRemoteStartupRejectionCode,
nativeSessionRecoveryProjection,
nativeGovernedWaitResult,
nativeConversationReplyResult,
@@ -14060,6 +14062,62 @@ describe("native session bounded recovery", () => {
});
});
describe("a session.open the provider itself rejected before any provider work", () => {
const sessionOpenFailure = (providerFailure: Record<string, unknown>) => {
const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2,
result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" };
const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } };
return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`),
"runnerCommandResult", { value: commandResult, enumerable: false });
};
const startupFailure = (fields: Record<string, unknown>) => ({ schema: "paperclip.provider-failure.v1", provider: "openai",
phase: "startup", category: "unknown", retryability: "unknown",
message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
...fields });
it.each(["authentication", "permission", "invalid_request"])("stops on the first %s rejection and hands the run to the board", (category) => {
const code = nativeSessionFailureSourceCode(sessionOpenFailure(startupFailure({ category, retryability: "requires_action" })));
expect(code).toBe(NATIVE_PROVIDER_STARTUP_REJECTED);
const disposition = nativeSessionFailureDisposition(1, new Date(), code);
expect(disposition).toEqual({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null });
expect(nativeSessionRecoveryProjection({ ...disposition, agentId: "agent" })).toMatchObject({
issueStatus: "blocked", recoveryOwner: { kind: "board" },
});
});
it("keeps bounded same-run retries for a local startup failure without a definite permanent cause", () => {
for (const fields of [
{},
{ message: "Query closed before response received" },
{ category: "overloaded", retryability: "transient", httpStatus: 503 },
{ category: "timeout", retryability: "transient", httpStatus: 504 },
{ category: "rate_limit", httpStatus: 429 },
{ category: "quota_exhausted", retryability: "requires_action", providerCode: "usageLimitExceeded" },
{ category: "model_unavailable", retryability: "requires_action" },
{ phase: "turn", category: "authentication", retryability: "requires_action" },
]) {
const failure = sessionOpenFailure(startupFailure(fields));
const code = nativeSessionFailureSourceCode(failure);
expect(code).toBe("provider_session_open_rejected");
expect(nativeRemoteStartupRejectionCode(code, failure, false)).toBe("provider_session_open_rejected");
}
expect(nativeSessionFailureDisposition(2, new Date(), "provider_session_open_rejected")).toMatchObject({ phase: "retryable_failure" });
expect(nativeSessionFailureDisposition(3, new Date(), "provider_session_open_rejected")).toEqual({
phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null,
});
});
it("stops a remote startup rejection at once because a remote same-run bootstrap cannot be verified", () => {
const failure = sessionOpenFailure(startupFailure({ message: "Query closed before response received" }));
expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(failure), failure, true))
.toBe(NATIVE_PROVIDER_STARTUP_REJECTED);
const unreported = new Error(`PRP command session.open failed: {"result":{"code":"command_execution_failed","message":"Provider request failed"},"status":"failed"}`);
expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(unreported), unreported, true))
.toBe("provider_session_open_rejected");
expect(nativeRemoteStartupRejectionCode("native_session_interrupted", failure, true)).toBe("native_session_interrupted");
});
});
it("stops same-run retries for an operator while a saved side-effecting call has no result", () => {
const failure = new NativeProviderSideEffectPendingError(
"Error: native_runner_source_semantic_reconciliation_required",
@@ -8443,7 +8443,8 @@ export function nativeSessionFailureDisposition(
sourceFailureCode === "native_session_cleanup_quarantined" ||
sourceFailureCode === "native_adopted_runner_authentication_timeout" ||
sourceFailureCode === "native_provider_usage_limit" ||
sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED;
sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED ||
sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED;
const permissionDeclined = sourceFailureCode === "native_permission_declined";
const exhausted = permanentFailure || permissionDeclined || attempt >= 3;
return {
@@ -8496,6 +8497,26 @@ export const NATIVE_PROVIDER_SESSION_CLOSED_ACTION =
"The provider session closed after a turn start that may have reached the provider, and this run cannot reopen it. Retry starts a new run with a fresh provider conversation and the saved task context.";
const RUNNER_PROVIDER_SESSION_CLOSED = "provider_session_closed";
export const NATIVE_PROVIDER_STARTUP_REJECTED = "native_provider_startup_rejected";
export const NATIVE_PROVIDER_STARTUP_REJECTED_ACTION =
"The provider refused to start a session, so no provider work began. Read the provider message and fix its cause in the agent's environment or AI connection if needed, then Retry. Retry starts a new run with a fresh provider session.";
function deterministicProviderStartupFailure(error: unknown): boolean {
const failure = providerFailureFromError(error);
return failure?.phase === "startup" && failure.retryability === "requires_action"
&& ["authentication", "permission", "invalid_request"].includes(failure.category);
}
export function nativeRemoteStartupRejectionCode<Code extends string>(
sourceFailureCode: Code,
error: unknown,
remoteTarget: boolean,
): Code | typeof NATIVE_PROVIDER_STARTUP_REJECTED {
return remoteTarget && sourceFailureCode === "provider_session_open_rejected"
&& providerFailureFromError(error)?.phase === "startup"
? NATIVE_PROVIDER_STARTUP_REJECTED : sourceFailureCode;
}
/** Reads the Runner's closed class from an authenticated turn.start failure. */
function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean {
if (!error || typeof error !== "object" || depth > 5) return false;
@@ -8522,6 +8543,7 @@ export function nativeSessionFailureSourceCode(
| "native_provider_approval_required"
| "native_provider_usage_limit"
| typeof NATIVE_PROVIDER_SESSION_CLOSED
| typeof NATIVE_PROVIDER_STARTUP_REJECTED
| "native_session_cleanup_quarantined"
| "native_adopted_runner_authentication_timeout"
| "runner_remote_provider_artifact_incompatible"
@@ -8663,6 +8685,7 @@ export function nativeSessionFailureSourceCode(
if (/classification=(?:requested_model_unsupported|grok_model_mismatch)\)/.test(message)) {
return "native_provider_model_rejected";
}
if (deterministicProviderStartupFailure(error)) return NATIVE_PROVIDER_STARTUP_REJECTED;
return "provider_session_open_rejected";
}
// The controller stopped waiting for session.open while runner startup was
@@ -8811,6 +8834,7 @@ export async function nativeProviderRecoveryEvidence(input: {
"provider_request_protocol_error",
"native_runner_process_exited",
"provider_session_open_rejected",
NATIVE_PROVIDER_STARTUP_REJECTED,
"provider_session_open_timeout",
"local_runtime_api_listener_unavailable",
]).has(input.sourceFailureCode);
@@ -12297,7 +12321,7 @@ async function executePaperclipNativeSessionAttempt(
? classifiedFailureCode
: providerUsageLimitObserved
? "native_provider_usage_limit"
: classifiedFailureCode;
: nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote");
const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote"
&& !input.restartRecovery && !retainedTransition && !remoteWarmRecovery
? {
@@ -12456,6 +12480,8 @@ async function executePaperclipNativeSessionAttempt(
? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler."
: sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
: sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED
? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION
: sourceFailureCode === "native_session_cleanup_quarantined"
? nativeCleanupRefusalAction(completedTurnCleanup)
: sourceFailureCode === "runner_remote_provider_verification_unavailable"
@@ -12731,6 +12757,8 @@ async function executePaperclipNativeSessionAttempt(
? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler."
: sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
: sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED
? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION
: sourceFailureCode === "native_session_cleanup_quarantined"
? nativeCleanupRefusalAction(completedTurnCleanup)
: sourceFailureCode === "runner_task_admission_rejected"
@@ -0,0 +1,121 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
executeNativeSession: provider.execute,
}));
import { executePaperclipNativeSession, NATIVE_PROVIDER_STARTUP_REJECTED } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { heartbeatService } from "../heartbeat.js";
import { requestTaskRetry } from "../task-retry.js";
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
describe("native session.open the provider rejected before any provider work", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID();
const agentId = randomUUID();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("native-startup-rejected-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Startup rejected", issuePrefix: "NSR" });
await db.insert(agents).values({ id: agentId, companyId, name: "Codex agent", status: "active", adapterType: "paperclip_runner",
adapterConfig: { provider: "codex", model: "gpt-test" } });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" });
});
afterAll(async () => { await temporary?.cleanup(); });
const rejection = (providerFailure: Record<string, unknown>) => {
const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2,
result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" };
const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } };
return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`),
"runnerCommandResult", { value: commandResult, enumerable: false });
};
const startupFailure = (fields: Record<string, unknown> = {}) => ({ schema: "paperclip.provider-failure.v1", provider: "openai",
phase: "startup", category: "unknown", retryability: "unknown",
message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
...fields });
async function rejectSessionOpen(failure: Error, attempt = 0) {
const issueId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Plan the change", status: "in_progress", workMode: "standard",
assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
const runId = randomUUID();
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "assignment", triggerDetail: "system",
contextSnapshot: { issueId } }).returning();
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt });
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1",
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
provider: { kind: "codex", model: "gpt-test" },
task: { identifier: issueId, title: issue.title, description: null, prompt: "Plan the change.", workMode: "standard" },
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
provider.execute.mockReset().mockRejectedValue(failure);
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
descriptor: async () => { throw new Error("Unexpected provider call"); },
openSession: async () => { throw new Error("Unexpected provider call"); },
} }).catch(() => undefined);
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
return { issueId, runId, coordinator };
}
it("fails the run on the first deterministic rejection with a plain explanation, and Retry starts a new run", async () => {
const { issueId, runId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ category: "authentication",
retryability: "requires_action", message: "error creating thread: the AI connection was revoked" })));
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null,
failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry", providerSessionEstablished: false,
providerFailure: expect.objectContaining({ phase: "startup" }),
nextAction: expect.stringContaining("Retry starts a new run") }) });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true,
providerFailure: expect.objectContaining({ phase: "startup" }) });
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session",
idempotencyKey: randomUUID(), observedRunId: runId });
expect(receipt).toMatchObject({ accepted: true, stale: false });
const heartbeat = heartbeatService(db);
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
.filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId);
expect(retried).toHaveLength(1);
await heartbeat.drainActiveRunExecutions();
}, 120_000);
it.each([
startupFailure(),
startupFailure({ message: "Query closed before response received" }),
startupFailure({ category: "overloaded", retryability: "transient", httpStatus: 503 }),
])("retries an unknown or transient local startup failure on the same run: $message", async (failure) => {
const { coordinator } = await rejectSessionOpen(rejection(failure));
expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted",
failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry" }) });
}, 120_000);
it("ends with Retry once the bounded startup retries are spent", async () => {
const { issueId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ message: "Query closed before response received" })), 2);
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null,
failureDetail: expect.objectContaining({ nextAction: expect.stringContaining("retry the task") }) });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true });
}, 120_000);
});