mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at 84fc67317e.
## What Changed
- Server (`native-session-executor.ts`): new permanent code
`native_provider_startup_rejected`.
- A definite `session.open` failure maps to it when its startup
`providerFailure` is `requires_action` for `authentication`,
`permission` or `invalid_request`.
- On a remote execution target (`nativeRemoteStartupRejectionCode`), any
definite `session.open` failure with a startup `providerFailure` maps to
it. A remote same-run bootstrap after the runner launched cannot prove
ownership and stops on `ambiguous_remote_recovery_evidence`.
- The run ends as `terminal_failure` with a board owner and a plain next
action. The board offers Retry, which starts one new run with a fresh
provider session.
- On a local target, unknown, transient, rate-limit, quota and model
startup failures keep the bounded same-run retries and end as
`native_session_retry_exhausted` with Retry. That code is unchanged
because the chat-channel failed-run retry depends on it.
- Runner: planning runs keep Codex's
`paperclip-runner-workspace-read-only` profile, also in an external
sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode
(Conversational)") and does not block writes or commands, so it cannot
replace that enforcement.
- Ask mode: an ask task builds `executionMode: "default"`. Its launch
has `readOnly: false`, the external sandbox profile, and no
collaboration mode. Ask mode restricts only Paperclip mutation tools,
not the Codex sandbox.
- Tests:
- `native-startup-rejected-retry.integration.test.ts` (new, embedded
Postgres, Codex provider): a deterministic rejection ends at once with
`canRetry: true`, and Retry starts exactly one new run. Unknown and
transient local failures stay `retryable_failure`. A spent retry budget
ends `terminal_failure` with `canRetry: true`.
- Unit tests for the deterministic, local-retry and remote rules.
- `runner-provider-launch.test.ts`: ask, standard and planning launches
in an external sandbox.
- The Rust external-sandbox test now fails if a non-planning turn sends
a collaboration mode (`--forbid-plan-collaboration` switch on the fake
app server).
- Docs: `packages/paperclip-runner/docs/codex-driver.md`,
`packages/paperclip-runner/docs/durable-recovery.md`,
`doc/execution-semantics.md`.
## Verification
- Red on master, green here: the commands in "Steps to reproduce".
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1364 pass.
- `npx vitest run src/live/runner-provider-launch.test.ts`
(packages/paperclip-runner): 22 pass.
- `cargo test -p paperclip-runner-core --test codex_provider
external_sandbox`: pass. `cargo fmt --check`: clean.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.
## Risks
- Codex planning on Daytona still fails at thread creation. With this
change it ends at once with the provider message and Retry instead of an
ownership hold. A follow-up must make Codex's read-only sandbox work in
that container, or add a verified Paperclip read-only guard. This PR
does not relax read-only enforcement.
- A remote startup failure that could clear by itself now needs a Retry
click instead of an automatic same-run retry. Before this change, that
retry ended in an ownership hold, so no working automatic path is lost.
- A false `native_provider_startup_rejected` costs automatic retries and
shows Retry earlier. It never replays provider work, because
`session.open` precedes every prompt.
- Saved state: no persisted shape or launch argument changes. A run that
an older release held on `ambiguous_remote_recovery_evidence` after the
same rejection already offers Retry once its saved workspace is
recovered. No migration is needed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit, browser reads) and extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6dd8611a2f
commit
fa3cd6f330
9 files changed
+273
-5
No files matched your search
@@ -1355,8 +1355,15 @@ provider cannot run the selected model (`requested_model_unsupported`,
|
||||
`grok_model_mismatch`) is a model rejection, not a startup interruption. It
|
||||
stops on its first attempt as `native_provider_model_rejected`, with no
|
||||
same-run bootstrap retry, and leaves an active board recovery action that asks
|
||||
for a supported model. Other `session.open` rejections keep the bounded
|
||||
startup retry. An indeterminate `session.open` outcome stays unclassified.
|
||||
for a supported model. A definite rejection whose startup `providerFailure`
|
||||
is `requires_action` for `authentication`, `permission` or `invalid_request`
|
||||
stops on its first attempt as `native_provider_startup_rejected`. On a remote
|
||||
execution target, any definite rejection with a startup `providerFailure` stops
|
||||
the same way, because a remote same-run bootstrap cannot prove ownership. No
|
||||
provider work began, so the board offers Retry, which starts a new run with a
|
||||
fresh provider session. Other local `session.open` rejections keep the bounded
|
||||
startup retry and end with Retry when it is spent. An
|
||||
indeterminate `session.open` outcome stays unclassified.
|
||||
|
||||
### Workspace scan failures before provider startup
|
||||
|
||||
|
||||
@@ -82,6 +82,17 @@ valid, but when `PAPERCLIP_WORKSPACE_CWD` is present its canonical path must be
|
||||
equal to or below that assigned workspace so sibling and symlink escapes fail
|
||||
before provider startup.
|
||||
|
||||
When the execution target is itself the sandbox (a remote sandbox such as
|
||||
Daytona sets `PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`), standard and ask runs use
|
||||
the default execution mode. The Runner starts their Codex threads and turns
|
||||
with the external sandbox policy and sends no collaboration mode. Ask mode
|
||||
restricts Paperclip mutation tools, not the Codex sandbox. Planning runs keep
|
||||
the `paperclip-runner-workspace-read-only` profile everywhere, because Codex's
|
||||
plan collaboration mode is a prompt and does not block writes or commands.
|
||||
Inside a container without network-namespace rights, that profile's helper can
|
||||
fail thread creation (`bwrap: loopback: Failed RTM_NEWADDR`); the run then ends
|
||||
as a provider startup failure with Retry (see `durable-recovery.md`).
|
||||
|
||||
The returned sandbox facts remain authoritative. Codex 0.132.0 may inject a
|
||||
provider-managed writable root such as `~/.codex/memories` after a first run,
|
||||
even with `features.memories=false` and an explicit Codex-home deny. That makes
|
||||
|
||||
@@ -151,6 +151,20 @@ which proves that the sidecar and the provider are gone.
|
||||
the saved task context. An older Runner sends the same text without the class; the
|
||||
controller then keeps its bounded same-run retries, because the same text
|
||||
also covers a session that can reopen after a restart.
|
||||
- A definite `session.open` failure whose command result carries a startup
|
||||
`providerFailure` with `retryability: requires_action` and category
|
||||
`authentication`, `permission` or `invalid_request` records
|
||||
`native_provider_startup_rejected`. No provider work began, so the
|
||||
controller stops same-run bootstrap retries and the board offers Retry at once
|
||||
with the provider message. On a remote execution target, every definite
|
||||
`session.open` failure with a startup `providerFailure` records the same code,
|
||||
because a remote same-run bootstrap after the runner launched cannot prove
|
||||
ownership and would stop on `ambiguous_remote_recovery_evidence`. On a local
|
||||
target, other startup failures (unknown, transient, rate limit, quota, model)
|
||||
keep the bounded same-run startup retries and end as
|
||||
`native_session_retry_exhausted` with Retry. A run an older release held on
|
||||
`ambiguous_remote_recovery_evidence` after such a rejection offers Retry once
|
||||
its saved workspace is recovered; no migration is needed.
|
||||
|
||||
Until a proof settles it, the saved session stays `turn_starting`, so a Runner
|
||||
killed during the proof restores through the same check. The cut-off command is
|
||||
|
||||
@@ -803,6 +803,9 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let require_external_sandbox = args
|
||||
.iter()
|
||||
.any(|value| value == "--require-external-sandbox");
|
||||
let forbid_plan_collaboration = args
|
||||
.iter()
|
||||
.any(|value| value == "--forbid-plan-collaboration");
|
||||
let expected_canonical_task_context = argument(&args, "--expected-canonical-task-context")
|
||||
.map(|value| serde_json::from_str::<Value>(&value))
|
||||
.transpose()?;
|
||||
@@ -1756,6 +1759,11 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
{
|
||||
return Err("turn/start omitted the external sandbox boundary".into());
|
||||
}
|
||||
if forbid_plan_collaboration
|
||||
&& message.pointer("/params/collaborationMode").is_some()
|
||||
{
|
||||
return Err("turn/start requested a collaboration mode outside planning".into());
|
||||
}
|
||||
if let Some(pid_file) = escaped_tool_pid_file.as_deref() {
|
||||
if turn_start_count == 0 {
|
||||
std::process::Command::new("perl")
|
||||
|
||||
@@ -632,8 +632,12 @@ fn ordinary_recovery_preserves_command_replay_and_collision_checks() {
|
||||
#[test]
|
||||
fn delegates_command_isolation_to_an_explicit_external_sandbox() {
|
||||
let directory = temporary_directory("external-sandbox");
|
||||
let mut config = provider_config(&directory, &["--require-external-sandbox"]);
|
||||
let mut config = provider_config(
|
||||
&directory,
|
||||
&["--require-external-sandbox", "--forbid-plan-collaboration"],
|
||||
);
|
||||
config.externally_sandboxed = true;
|
||||
config.read_only = false;
|
||||
|
||||
let mut provider = CodexProvider::start(&config, None)
|
||||
.expect("start Codex with an externally owned sandbox boundary");
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createRunnerProviderLaunch, preparedRunnerLaunchWorkingDirectory, runnerLaunchRequirementsDigest, runnerLaunchSettingsMatch, validatePreparedRunnerLaunch,
|
||||
runnerProviderLaunchRequirements, type RunnerProviderLaunchEnvironment, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js";
|
||||
import { createSecuredCodexThreadParams } from "../drivers/codex/codex-security-config.js";
|
||||
|
||||
const requirements: RunnerProviderLaunchRequirements = {
|
||||
schema: "paperclip.runner.launch-requirements.v1",
|
||||
@@ -27,6 +28,22 @@ describe("built-in provider launch preparation", () => {
|
||||
expect(frozen.settings).toEqual({});
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ workMode: "ask", executionMode: "default", readOnly: false, collaborationMode: "default" },
|
||||
{ workMode: "standard", executionMode: "default", readOnly: false, collaborationMode: "default" },
|
||||
{ workMode: "planning", executionMode: "plan", readOnly: true, collaborationMode: "plan" },
|
||||
] as const)("launches a $workMode Codex run in an external sandbox without a read-only Codex sandbox claim", (mode) => {
|
||||
const thread = createSecuredCodexThreadParams("/execution/workspace", mode.executionMode, true, false,
|
||||
{ PAPERCLIP_RUNNER_EXTERNAL_SANDBOX: "1" });
|
||||
const selected = runnerProviderLaunchRequirements({ provider: "codex", externallySandboxed: true },
|
||||
requirements.identity, { model: requirements.model, ...thread }, null, "");
|
||||
expect(selected.readOnly).toBe(mode.readOnly);
|
||||
const launch = createRunnerProviderLaunch(selected, { ...environment,
|
||||
options: { ...environment.options, externallySandboxed: true } });
|
||||
expect(launch).toMatchObject({ externallySandboxed: true, collaborationMode: mode.collaborationMode });
|
||||
expect("readOnly" in launch).toBe(mode.readOnly);
|
||||
});
|
||||
|
||||
it("preserves probe project-instruction suppression through a prepared launch", () => {
|
||||
const selected = runnerProviderLaunchRequirements({ provider: "codex", disableCodexProjectInstructions: true,
|
||||
disableCodexWebSearch: true }, requirements.identity,
|
||||
|
||||
@@ -435,6 +435,8 @@ import {
|
||||
nativeProviderUsageLimitFromEvent,
|
||||
nativeSessionFailureSourceCode,
|
||||
NATIVE_PROVIDER_SESSION_CLOSED,
|
||||
NATIVE_PROVIDER_STARTUP_REJECTED,
|
||||
nativeRemoteStartupRejectionCode,
|
||||
nativeSessionRecoveryProjection,
|
||||
nativeGovernedWaitResult,
|
||||
nativeConversationReplyResult,
|
||||
@@ -14060,6 +14062,62 @@ describe("native session bounded recovery", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("a session.open the provider itself rejected before any provider work", () => {
|
||||
const sessionOpenFailure = (providerFailure: Record<string, unknown>) => {
|
||||
const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2,
|
||||
result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" };
|
||||
const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } };
|
||||
return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`),
|
||||
"runnerCommandResult", { value: commandResult, enumerable: false });
|
||||
};
|
||||
const startupFailure = (fields: Record<string, unknown>) => ({ schema: "paperclip.provider-failure.v1", provider: "openai",
|
||||
phase: "startup", category: "unknown", retryability: "unknown",
|
||||
message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
|
||||
...fields });
|
||||
|
||||
it.each(["authentication", "permission", "invalid_request"])("stops on the first %s rejection and hands the run to the board", (category) => {
|
||||
const code = nativeSessionFailureSourceCode(sessionOpenFailure(startupFailure({ category, retryability: "requires_action" })));
|
||||
expect(code).toBe(NATIVE_PROVIDER_STARTUP_REJECTED);
|
||||
const disposition = nativeSessionFailureDisposition(1, new Date(), code);
|
||||
expect(disposition).toEqual({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null });
|
||||
expect(nativeSessionRecoveryProjection({ ...disposition, agentId: "agent" })).toMatchObject({
|
||||
issueStatus: "blocked", recoveryOwner: { kind: "board" },
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps bounded same-run retries for a local startup failure without a definite permanent cause", () => {
|
||||
for (const fields of [
|
||||
{},
|
||||
{ message: "Query closed before response received" },
|
||||
{ category: "overloaded", retryability: "transient", httpStatus: 503 },
|
||||
{ category: "timeout", retryability: "transient", httpStatus: 504 },
|
||||
{ category: "rate_limit", httpStatus: 429 },
|
||||
{ category: "quota_exhausted", retryability: "requires_action", providerCode: "usageLimitExceeded" },
|
||||
{ category: "model_unavailable", retryability: "requires_action" },
|
||||
{ phase: "turn", category: "authentication", retryability: "requires_action" },
|
||||
]) {
|
||||
const failure = sessionOpenFailure(startupFailure(fields));
|
||||
const code = nativeSessionFailureSourceCode(failure);
|
||||
expect(code).toBe("provider_session_open_rejected");
|
||||
expect(nativeRemoteStartupRejectionCode(code, failure, false)).toBe("provider_session_open_rejected");
|
||||
}
|
||||
expect(nativeSessionFailureDisposition(2, new Date(), "provider_session_open_rejected")).toMatchObject({ phase: "retryable_failure" });
|
||||
expect(nativeSessionFailureDisposition(3, new Date(), "provider_session_open_rejected")).toEqual({
|
||||
phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null,
|
||||
});
|
||||
});
|
||||
|
||||
it("stops a remote startup rejection at once because a remote same-run bootstrap cannot be verified", () => {
|
||||
const failure = sessionOpenFailure(startupFailure({ message: "Query closed before response received" }));
|
||||
expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(failure), failure, true))
|
||||
.toBe(NATIVE_PROVIDER_STARTUP_REJECTED);
|
||||
const unreported = new Error(`PRP command session.open failed: {"result":{"code":"command_execution_failed","message":"Provider request failed"},"status":"failed"}`);
|
||||
expect(nativeRemoteStartupRejectionCode(nativeSessionFailureSourceCode(unreported), unreported, true))
|
||||
.toBe("provider_session_open_rejected");
|
||||
expect(nativeRemoteStartupRejectionCode("native_session_interrupted", failure, true)).toBe("native_session_interrupted");
|
||||
});
|
||||
});
|
||||
|
||||
it("stops same-run retries for an operator while a saved side-effecting call has no result", () => {
|
||||
const failure = new NativeProviderSideEffectPendingError(
|
||||
"Error: native_runner_source_semantic_reconciliation_required",
|
||||
|
||||
@@ -8443,7 +8443,8 @@ export function nativeSessionFailureDisposition(
|
||||
sourceFailureCode === "native_session_cleanup_quarantined" ||
|
||||
sourceFailureCode === "native_adopted_runner_authentication_timeout" ||
|
||||
sourceFailureCode === "native_provider_usage_limit" ||
|
||||
sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED;
|
||||
sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED ||
|
||||
sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED;
|
||||
const permissionDeclined = sourceFailureCode === "native_permission_declined";
|
||||
const exhausted = permanentFailure || permissionDeclined || attempt >= 3;
|
||||
return {
|
||||
@@ -8496,6 +8497,26 @@ export const NATIVE_PROVIDER_SESSION_CLOSED_ACTION =
|
||||
"The provider session closed after a turn start that may have reached the provider, and this run cannot reopen it. Retry starts a new run with a fresh provider conversation and the saved task context.";
|
||||
const RUNNER_PROVIDER_SESSION_CLOSED = "provider_session_closed";
|
||||
|
||||
export const NATIVE_PROVIDER_STARTUP_REJECTED = "native_provider_startup_rejected";
|
||||
export const NATIVE_PROVIDER_STARTUP_REJECTED_ACTION =
|
||||
"The provider refused to start a session, so no provider work began. Read the provider message and fix its cause in the agent's environment or AI connection if needed, then Retry. Retry starts a new run with a fresh provider session.";
|
||||
|
||||
function deterministicProviderStartupFailure(error: unknown): boolean {
|
||||
const failure = providerFailureFromError(error);
|
||||
return failure?.phase === "startup" && failure.retryability === "requires_action"
|
||||
&& ["authentication", "permission", "invalid_request"].includes(failure.category);
|
||||
}
|
||||
|
||||
export function nativeRemoteStartupRejectionCode<Code extends string>(
|
||||
sourceFailureCode: Code,
|
||||
error: unknown,
|
||||
remoteTarget: boolean,
|
||||
): Code | typeof NATIVE_PROVIDER_STARTUP_REJECTED {
|
||||
return remoteTarget && sourceFailureCode === "provider_session_open_rejected"
|
||||
&& providerFailureFromError(error)?.phase === "startup"
|
||||
? NATIVE_PROVIDER_STARTUP_REJECTED : sourceFailureCode;
|
||||
}
|
||||
|
||||
/** Reads the Runner's closed class from an authenticated turn.start failure. */
|
||||
function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean {
|
||||
if (!error || typeof error !== "object" || depth > 5) return false;
|
||||
@@ -8522,6 +8543,7 @@ export function nativeSessionFailureSourceCode(
|
||||
| "native_provider_approval_required"
|
||||
| "native_provider_usage_limit"
|
||||
| typeof NATIVE_PROVIDER_SESSION_CLOSED
|
||||
| typeof NATIVE_PROVIDER_STARTUP_REJECTED
|
||||
| "native_session_cleanup_quarantined"
|
||||
| "native_adopted_runner_authentication_timeout"
|
||||
| "runner_remote_provider_artifact_incompatible"
|
||||
@@ -8663,6 +8685,7 @@ export function nativeSessionFailureSourceCode(
|
||||
if (/classification=(?:requested_model_unsupported|grok_model_mismatch)\)/.test(message)) {
|
||||
return "native_provider_model_rejected";
|
||||
}
|
||||
if (deterministicProviderStartupFailure(error)) return NATIVE_PROVIDER_STARTUP_REJECTED;
|
||||
return "provider_session_open_rejected";
|
||||
}
|
||||
// The controller stopped waiting for session.open while runner startup was
|
||||
@@ -8811,6 +8834,7 @@ export async function nativeProviderRecoveryEvidence(input: {
|
||||
"provider_request_protocol_error",
|
||||
"native_runner_process_exited",
|
||||
"provider_session_open_rejected",
|
||||
NATIVE_PROVIDER_STARTUP_REJECTED,
|
||||
"provider_session_open_timeout",
|
||||
"local_runtime_api_listener_unavailable",
|
||||
]).has(input.sourceFailureCode);
|
||||
@@ -12297,7 +12321,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
? classifiedFailureCode
|
||||
: providerUsageLimitObserved
|
||||
? "native_provider_usage_limit"
|
||||
: classifiedFailureCode;
|
||||
: nativeRemoteStartupRejectionCode(classifiedFailureCode, error, input.runnerExecutionTarget?.kind === "remote");
|
||||
const bootstrap = input.useRunnerd && input.runnerExecutionTarget?.kind === "remote"
|
||||
&& !input.restartRecovery && !retainedTransition && !remoteWarmRecovery
|
||||
? {
|
||||
@@ -12456,6 +12480,8 @@ async function executePaperclipNativeSessionAttempt(
|
||||
? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler."
|
||||
: sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
|
||||
? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
|
||||
: sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED
|
||||
? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION
|
||||
: sourceFailureCode === "native_session_cleanup_quarantined"
|
||||
? nativeCleanupRefusalAction(completedTurnCleanup)
|
||||
: sourceFailureCode === "runner_remote_provider_verification_unavailable"
|
||||
@@ -12731,6 +12757,8 @@ async function executePaperclipNativeSessionAttempt(
|
||||
? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler."
|
||||
: sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
|
||||
? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
|
||||
: sourceFailureCode === NATIVE_PROVIDER_STARTUP_REJECTED
|
||||
? NATIVE_PROVIDER_STARTUP_REJECTED_ACTION
|
||||
: sourceFailureCode === "native_session_cleanup_quarantined"
|
||||
? nativeCleanupRefusalAction(completedTurnCleanup)
|
||||
: sourceFailureCode === "runner_task_admission_rejected"
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
|
||||
executeNativeSession: provider.execute,
|
||||
}));
|
||||
|
||||
import { executePaperclipNativeSession, NATIVE_PROVIDER_STARTUP_REJECTED } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { heartbeatService } from "../heartbeat.js";
|
||||
import { requestTaskRetry } from "../task-retry.js";
|
||||
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
|
||||
|
||||
describe("native session.open the provider rejected before any provider work", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-startup-rejected-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Startup rejected", issuePrefix: "NSR" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Codex agent", status: "active", adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "codex", model: "gpt-test" } });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
const rejection = (providerFailure: Record<string, unknown>) => {
|
||||
const commandResult = { commandId: "command_prp_00000002", commandType: "session.open", controllerSeq: 2,
|
||||
result: { code: "command_execution_failed", message: String(providerFailure.message), providerFailure }, status: "failed" };
|
||||
const reported = { ...commandResult, result: { code: "command_execution_failed", message: "Provider request failed" } };
|
||||
return Object.defineProperty(new Error(`PRP command session.open failed: ${JSON.stringify(reported)}`),
|
||||
"runnerCommandResult", { value: commandResult, enumerable: false });
|
||||
};
|
||||
const startupFailure = (fields: Record<string, unknown> = {}) => ({ schema: "paperclip.provider-failure.v1", provider: "openai",
|
||||
phase: "startup", category: "unknown", retryability: "unknown",
|
||||
message: "error creating thread: failed to load AGENTS.md instructions for environment `local`: fs sandbox helper failed with status exit status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
|
||||
...fields });
|
||||
|
||||
async function rejectSessionOpen(failure: Error, attempt = 0) {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Plan the change", status: "in_progress", workMode: "standard",
|
||||
assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
const runId = randomUUID();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
|
||||
nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "assignment", triggerDetail: "system",
|
||||
contextSnapshot: { issueId } }).returning();
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1",
|
||||
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "codex", model: "gpt-test" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Plan the change.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
||||
provider.execute.mockReset().mockRejectedValue(failure);
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
|
||||
descriptor: async () => { throw new Error("Unexpected provider call"); },
|
||||
openSession: async () => { throw new Error("Unexpected provider call"); },
|
||||
} }).catch(() => undefined);
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
return { issueId, runId, coordinator };
|
||||
}
|
||||
|
||||
it("fails the run on the first deterministic rejection with a plain explanation, and Retry starts a new run", async () => {
|
||||
const { issueId, runId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ category: "authentication",
|
||||
retryability: "requires_action", message: "error creating thread: the AI connection was revoked" })));
|
||||
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_STARTUP_REJECTED, nextAttemptAt: null,
|
||||
failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry", providerSessionEstablished: false,
|
||||
providerFailure: expect.objectContaining({ phase: "startup" }),
|
||||
nextAction: expect.stringContaining("Retry starts a new run") }) });
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true,
|
||||
providerFailure: expect.objectContaining({ phase: "startup" }) });
|
||||
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session",
|
||||
idempotencyKey: randomUUID(), observedRunId: runId });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false });
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
|
||||
const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
|
||||
.filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId);
|
||||
expect(retried).toHaveLength(1);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}, 120_000);
|
||||
|
||||
it.each([
|
||||
startupFailure(),
|
||||
startupFailure({ message: "Query closed before response received" }),
|
||||
startupFailure({ category: "overloaded", retryability: "transient", httpStatus: 503 }),
|
||||
])("retries an unknown or transient local startup failure on the same run: $message", async (failure) => {
|
||||
const { coordinator } = await rejectSessionOpen(rejection(failure));
|
||||
expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted",
|
||||
failureDetail: expect.objectContaining({ recoveryMode: "bootstrap_retry" }) });
|
||||
}, 120_000);
|
||||
|
||||
it("ends with Retry once the bounded startup retries are spent", async () => {
|
||||
const { issueId, coordinator } = await rejectSessionOpen(rejection(startupFailure({ message: "Query closed before response received" })), 2);
|
||||
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_session_retry_exhausted", nextAttemptAt: null,
|
||||
failureDetail: expect.objectContaining({ nextAction: expect.stringContaining("retry the task") }) });
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true });
|
||||
}, 120_000);
|
||||
});
|
||||
Reference in new issue
Block a user