mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
1143abaf92
commit
08ff0828d6
8 files changed
+255
-20
No files matched your search
@@ -143,6 +143,14 @@ which proves that the sidecar and the provider are gone.
|
||||
no file (an older sidecar) proves nothing. The earlier behavior applies
|
||||
unchanged: a failed start closes the session, and a killed Runner uses
|
||||
lost-turn recovery.
|
||||
- A session this run closed cannot reopen for the same run. The failed start,
|
||||
and every later `turn.start` of the run, fail with the closed class
|
||||
`provider_session_closed` in the command result. The controller records
|
||||
`native_provider_session_closed`, stops same-run retries, and the board offers
|
||||
Retry at once. Retry starts a new run with a fresh provider conversation and
|
||||
the saved task context. An older Runner sends the same text without the class; the
|
||||
controller then keeps its bounded same-run retries, because the same text
|
||||
also covers a session that can reopen after a restart.
|
||||
|
||||
Until a proof settles it, the saved session stays `turn_starting`, so a Runner
|
||||
killed during the proof restores through the same check. The cut-off command is
|
||||
|
||||
@@ -28,7 +28,8 @@ use crate::durable::QualifiedLaunchArtifact;
|
||||
use crate::durable::{
|
||||
create_private_temporary_file, open_private_regular_file, verify_private_directory,
|
||||
AcpxLaunchProfile, Command, CommandExecution, CommandExecutor, DurableRunnerConfig,
|
||||
DurableRunnerError, EventPriority, PolledEvent, PROVIDER_STATE_INCOMPATIBLE,
|
||||
DurableRunnerError, EventPriority, PolledEvent, PROVIDER_SESSION_CLOSED,
|
||||
PROVIDER_STATE_INCOMPATIBLE,
|
||||
};
|
||||
use crate::generated_acpx_sidecar_contract::GeneratedAcpxSidecarCommand;
|
||||
use crate::process_supervisor::{VerifiedProcessArgument, VerifiedProcessLaunch};
|
||||
@@ -2061,7 +2062,8 @@ impl AcpxCommandExecutor {
|
||||
DurableRunnerError::invalid(format!("ACPX turn start failed closed: {error}"))
|
||||
}
|
||||
None => DurableRunnerError::invalid("ACPX session is not open"),
|
||||
});
|
||||
}
|
||||
.with_classification(PROVIDER_SESSION_CLOSED));
|
||||
}
|
||||
let reopenable = self.session.is_none()
|
||||
&& self.state.as_ref().is_some_and(|state| {
|
||||
@@ -2128,7 +2130,17 @@ impl AcpxCommandExecutor {
|
||||
self.retain_buffered_provider_events()?;
|
||||
}
|
||||
if self.session.is_none() {
|
||||
return Err(DurableRunnerError::invalid("ACPX session is not open"));
|
||||
let error = DurableRunnerError::invalid("ACPX session is not open");
|
||||
// Only a session this run closed is permanently closed for it. A
|
||||
// session awaiting provider-exit proof can reopen after a restart.
|
||||
let closed_for_run = self.state.as_ref().is_some_and(|state| {
|
||||
state.lifecycle == "closed" && state.descriptor.run_id == self.context.run_id
|
||||
});
|
||||
return Err(if closed_for_run {
|
||||
error.with_classification(PROVIDER_SESSION_CLOSED)
|
||||
} else {
|
||||
error
|
||||
});
|
||||
}
|
||||
let (provider_process_will_be_replaced, previous_process_id) = self
|
||||
.session
|
||||
@@ -2206,6 +2218,7 @@ impl AcpxCommandExecutor {
|
||||
}
|
||||
.to_owned();
|
||||
state.provider_exit_unconfirmed = rejected_checkpoint.is_some();
|
||||
let rejected_checkpoint_reopenable = rejected_checkpoint.is_some();
|
||||
state.active_turn_id = None;
|
||||
self.context.provider_turn_id = None;
|
||||
self.session = None;
|
||||
@@ -2225,9 +2238,15 @@ impl AcpxCommandExecutor {
|
||||
return Err(error);
|
||||
}
|
||||
}
|
||||
return Err(DurableRunnerError::invalid(format!(
|
||||
"ACPX turn start failed closed: {error}"
|
||||
)));
|
||||
let failure =
|
||||
DurableRunnerError::invalid(format!("ACPX turn start failed closed: {error}"));
|
||||
// A rejected checkpoint stays `prepared` and reopens on the next
|
||||
// start. A `closed` session cannot reopen for this run.
|
||||
return Err(if rejected_checkpoint_reopenable {
|
||||
failure
|
||||
} else {
|
||||
failure.with_classification(PROVIDER_SESSION_CLOSED)
|
||||
});
|
||||
}
|
||||
// Copy numeric timing fields only. Sidecar contents never enter the receipt.
|
||||
let timing = if measure {
|
||||
|
||||
@@ -51,6 +51,12 @@ pub const PROVIDER_ATTACH_REFUSED: &str = "provider_attach_refused";
|
||||
/// recorded result. Replacing the session could repeat or lose that action.
|
||||
pub const PROVIDER_SIDE_EFFECT_PENDING: &str = "provider_side_effect_pending";
|
||||
|
||||
/// The runner closed this run's provider session after a turn start whose
|
||||
/// prompt may have reached the provider. No later `turn.start` of the same run
|
||||
/// can reopen it; only a new run's attachment can. Resending could duplicate
|
||||
/// the prompt, so the controller stops same-run retries and offers Retry.
|
||||
pub const PROVIDER_SESSION_CLOSED: &str = "provider_session_closed";
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct DurableRunnerError(String, Option<serde_json::Value>, Option<&'static str>);
|
||||
|
||||
|
||||
+21
-10
@@ -3,6 +3,7 @@
|
||||
//! journal decides whether the prompt could have reached the provider.
|
||||
|
||||
use super::*;
|
||||
use paperclip_runner_core::durable::PROVIDER_SESSION_CLOSED;
|
||||
|
||||
const JOURNAL: &str = "paperclip-turn-dispatch.jsonl";
|
||||
|
||||
@@ -27,8 +28,14 @@ fn kill(pid: &Value) {
|
||||
}
|
||||
|
||||
/// Starts `turn` against a sidecar that freezes at `fault`, then kills the
|
||||
/// sidecar once the journal shows `phases`, and returns the start's error.
|
||||
fn cut_turn_start(harness: &mut Harness, turn: &str, fault: &str, phases: &[&str]) -> String {
|
||||
/// sidecar once the journal shows `phases`, and returns the start's error and
|
||||
/// its closed classification.
|
||||
fn cut_turn_start(
|
||||
harness: &mut Harness,
|
||||
turn: &str,
|
||||
fault: &str,
|
||||
phases: &[&str],
|
||||
) -> (String, Option<&'static str>) {
|
||||
harness.fixture.fault(fault);
|
||||
let sidecar = harness.fixture.process_starts().last().unwrap()["processStart"].clone();
|
||||
let root = harness.fixture.0.clone();
|
||||
@@ -68,21 +75,23 @@ fn cut_turn_start(harness: &mut Harness, turn: &str, fault: &str, phases: &[&str
|
||||
"turn.start",
|
||||
json!({"turnId": turn, "text": "Write the brief."}),
|
||||
))
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
.unwrap_err();
|
||||
killer.join().unwrap();
|
||||
harness.fixture.fault("");
|
||||
error
|
||||
(error.to_string(), error.classification())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once() {
|
||||
let mut harness = Harness::new();
|
||||
let error = cut_turn_start(&mut harness, "cut", "hang-before-dispatch", &["received"]);
|
||||
let (error, classification) =
|
||||
cut_turn_start(&mut harness, "cut", "hang-before-dispatch", &["received"]);
|
||||
assert!(
|
||||
error.contains("before its prompt reached the provider"),
|
||||
"{error}"
|
||||
);
|
||||
// The run recovers by itself: the controller must keep retrying it.
|
||||
assert_eq!(classification, None);
|
||||
let saved = harness.fixture.state();
|
||||
assert_eq!(saved["lifecycle"], "prepared", "{saved}");
|
||||
assert_eq!(saved["activeTurnId"], Value::Null);
|
||||
@@ -107,13 +116,15 @@ fn a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once() {
|
||||
#[test]
|
||||
fn a_start_cut_off_after_the_prompt_left_stays_closed_and_never_resends() {
|
||||
let mut harness = Harness::new();
|
||||
let error = cut_turn_start(
|
||||
let (error, classification) = cut_turn_start(
|
||||
&mut harness,
|
||||
"cut",
|
||||
"hang-after-dispatch",
|
||||
&["received", "dispatched"],
|
||||
);
|
||||
assert!(error.contains("failed closed"), "{error}");
|
||||
// No later start of this run can reopen it: the controller offers Retry at once.
|
||||
assert_eq!(classification, Some(PROVIDER_SESSION_CLOSED));
|
||||
assert_eq!(harness.fixture.state()["lifecycle"], "closed");
|
||||
let starts = harness.fixture.process_starts().len();
|
||||
harness.sequence += 1;
|
||||
@@ -124,9 +135,9 @@ fn a_start_cut_off_after_the_prompt_left_stays_closed_and_never_resends() {
|
||||
"turn.start",
|
||||
json!({"turnId": "resent", "text": "Write the brief."}),
|
||||
))
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert_eq!(refused, "ACPX session is not open");
|
||||
.unwrap_err();
|
||||
assert_eq!(refused.to_string(), "ACPX session is not open");
|
||||
assert_eq!(refused.classification(), Some(PROVIDER_SESSION_CLOSED));
|
||||
assert_eq!(harness.fixture.process_starts().len(), starts);
|
||||
}
|
||||
|
||||
|
||||
+7
-3
@@ -437,9 +437,13 @@ fn a_runner_restart_in_the_same_run_keeps_the_failed_turn_end_restart_closed() {
|
||||
"turn.start",
|
||||
json!({"turnId": "second", "text": "Complete the work."}),
|
||||
))
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert_eq!(failed, "ACPX session is not open");
|
||||
.unwrap_err();
|
||||
assert_eq!(failed.to_string(), "ACPX session is not open");
|
||||
// No same-run start can reopen it: the controller offers Retry at once.
|
||||
assert_eq!(
|
||||
failed.classification(),
|
||||
Some(paperclip_runner_core::durable::PROVIDER_SESSION_CLOSED)
|
||||
);
|
||||
assert_eq!(harness.fixture.process_starts().len(), starts);
|
||||
// Neither session.open nor a same-run attach reopens it.
|
||||
let reopened = harness
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
|
||||
executeNativeSession: provider.execute,
|
||||
}));
|
||||
|
||||
import { executePaperclipNativeSession, NATIVE_PROVIDER_SESSION_CLOSED } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { heartbeatService } from "../heartbeat.js";
|
||||
import { requestTaskRetry } from "../task-retry.js";
|
||||
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
|
||||
|
||||
// Chaos w-20261111T011704-s9107/t05 (claude, TM2): a cut-off ACPX turn start
|
||||
// left this run's session closed with the prompt possibly sent. Every same-run
|
||||
// resume got "ACPX session is not open"; the board offered Retry only after
|
||||
// three attempts (about 95 s). The Runner now classifies that refusal, and
|
||||
// Retry starts a new run with a fresh provider session.
|
||||
describe("native turn start refused on a closed provider session", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-closed-session-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Closed session", issuePrefix: "NCS" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Claude agent", status: "active", adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "acpx", agent: "claude", model: "claude-test" } });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
const refusal = (result: Record<string, unknown>) => {
|
||||
const commandResult = { commandId: "command_lab_1", commandType: "turn.start", controllerSeq: 10, result, status: "failed" };
|
||||
return Object.defineProperty(new Error(`PRP command turn.start failed: ${JSON.stringify(commandResult)}`),
|
||||
"runnerCommandResult", { value: commandResult, enumerable: false });
|
||||
};
|
||||
|
||||
async function refuseTurnStart(failure: Error) {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Write the brief", status: "in_progress", workMode: "standard",
|
||||
assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
const runId = randomUUID();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
|
||||
nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "automation", triggerDetail: "system",
|
||||
contextSnapshot: { issueId } }).returning();
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1",
|
||||
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "opencode", model: "openrouter/test-model" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Write the brief.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "opencode_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
// An established provider session: same-run resume is the ordinary recovery.
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution,
|
||||
sessionCheckpoint: { backendKind: "mock", providerSessionId: "claude-session" } } }).where(eq(heartbeatRuns.id, runId));
|
||||
provider.execute.mockReset().mockRejectedValue(failure);
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
|
||||
descriptor: async () => { throw new Error("Unexpected provider call"); },
|
||||
openSession: async () => { throw new Error("Unexpected provider call"); },
|
||||
} }).catch(() => undefined);
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
return { issueId, runId, coordinator };
|
||||
}
|
||||
|
||||
it("offers Retry on the first refusal, and Retry starts a new run", async () => {
|
||||
const { issueId, runId, coordinator } = await refuseTurnStart(refusal({ code: "command_execution_failed",
|
||||
message: "ACPX session is not open", classification: "provider_session_closed" }));
|
||||
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_SESSION_CLOSED, nextAttemptAt: null });
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true });
|
||||
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session",
|
||||
idempotencyKey: randomUUID(), observedRunId: runId });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false });
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
|
||||
const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
|
||||
.filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId);
|
||||
expect(retried).toHaveLength(1);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}, 120_000);
|
||||
|
||||
it("keeps a same-run retry when the Runner proved the prompt never left", async () => {
|
||||
const { coordinator } = await refuseTurnStart(refusal({ code: "command_execution_failed",
|
||||
message: "ACPX turn start failed before its prompt reached the provider: ACPX sidecar request timed out at turn.start" }));
|
||||
expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted" });
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -429,6 +429,7 @@ import {
|
||||
retainedNativeCleanupJournalMatches,
|
||||
nativeProviderUsageLimitFromEvent,
|
||||
nativeSessionFailureSourceCode,
|
||||
NATIVE_PROVIDER_SESSION_CLOSED,
|
||||
nativeSessionRecoveryProjection,
|
||||
nativeGovernedWaitResult,
|
||||
nativeConversationReplyResult,
|
||||
@@ -13976,6 +13977,52 @@ describe("native session bounded recovery", () => {
|
||||
expect(nativeSessionFailureSourceCode(new Error(failure.message))).toBe("native_session_interrupted");
|
||||
});
|
||||
|
||||
// Chaos w-20261111T011704-s9107/t05: after a cut-off ACPX turn start the
|
||||
// Runner saved the session closed; three same-run resumes each got the same
|
||||
// refusal before the board offered Retry.
|
||||
describe("a turn start the Runner refused because this run's session is closed", () => {
|
||||
const turnStartFailure = (result: Record<string, unknown>) => {
|
||||
const commandResult = { commandId: "command_lab_1", commandType: "turn.start", controllerSeq: 10, result, status: "failed" };
|
||||
return Object.defineProperty(new Error(`PRP command turn.start failed: ${JSON.stringify(commandResult)}`),
|
||||
"runnerCommandResult", { value: commandResult, enumerable: false });
|
||||
};
|
||||
const closed = (message: string) => turnStartFailure({ code: "command_execution_failed", message,
|
||||
classification: "provider_session_closed" });
|
||||
|
||||
it.each(["ACPX session is not open", "ACPX turn start failed closed: ACPX sidecar request timed out at turn.start"])(
|
||||
"stops on the first refusal and hands the run to the board: %s", (message) => {
|
||||
const code = nativeSessionFailureSourceCode(closed(message));
|
||||
expect(code).toBe(NATIVE_PROVIDER_SESSION_CLOSED);
|
||||
const disposition = nativeSessionFailureDisposition(1, new Date(), code);
|
||||
expect(disposition).toEqual({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_SESSION_CLOSED, nextAttemptAt: null });
|
||||
expect(nativeSessionRecoveryProjection({ ...disposition, agentId: "agent" })).toMatchObject({
|
||||
issueStatus: "blocked", recoveryOwner: { kind: "board" },
|
||||
});
|
||||
// A rewrapped failure keeps the class through its authenticated text.
|
||||
expect(nativeSessionFailureSourceCode(new Error(closed(message).message))).toBe(NATIVE_PROVIDER_SESSION_CLOSED);
|
||||
expect(nativeSessionFailureSourceCode(Object.assign(new Error("native session failed"), { cause: closed(message) })))
|
||||
.toBe(NATIVE_PROVIDER_SESSION_CLOSED);
|
||||
});
|
||||
|
||||
it("keeps retrying a start the Runner proved never reached the provider", () => {
|
||||
const failure = turnStartFailure({ code: "command_execution_failed",
|
||||
message: "ACPX turn start failed before its prompt reached the provider: ACPX sidecar request timed out at turn.start" });
|
||||
expect(nativeSessionFailureSourceCode(failure)).toBe("native_session_interrupted");
|
||||
expect(nativeSessionFailureDisposition(1, new Date(), "native_session_interrupted").phase).toBe("retryable_failure");
|
||||
});
|
||||
|
||||
it("keeps today's same-run retries for an unclassified refusal from an older Runner", () => {
|
||||
// An older Runner sends the same text for a session that can reopen after
|
||||
// a restart (provider exit not yet proven), so the text alone proves nothing.
|
||||
const failure = turnStartFailure({ code: "command_execution_failed", message: "ACPX session is not open" });
|
||||
expect(nativeSessionFailureSourceCode(failure)).toBe("native_session_interrupted");
|
||||
// Another command's class is not a turn-start refusal.
|
||||
const other = Object.defineProperty(new Error("PRP command session.open failed"), "runnerCommandResult",
|
||||
{ value: { commandType: "session.open", result: { classification: "provider_session_closed" } }, enumerable: false });
|
||||
expect(nativeSessionFailureSourceCode(other)).not.toBe(NATIVE_PROVIDER_SESSION_CLOSED);
|
||||
});
|
||||
});
|
||||
|
||||
it("stops same-run retries for an operator while a saved side-effecting call has no result", () => {
|
||||
const failure = new NativeProviderSideEffectPendingError(
|
||||
"Error: native_runner_source_semantic_reconciliation_required",
|
||||
|
||||
@@ -8312,7 +8312,8 @@ export function nativeSessionFailureDisposition(
|
||||
sourceFailureCode === NATIVE_CURRENT_WAKE_COMMENTS_CHANGED_AFTER_READ ||
|
||||
sourceFailureCode === "native_session_cleanup_quarantined" ||
|
||||
sourceFailureCode === "native_adopted_runner_authentication_timeout" ||
|
||||
sourceFailureCode === "native_provider_usage_limit";
|
||||
sourceFailureCode === "native_provider_usage_limit" ||
|
||||
sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED;
|
||||
const permissionDeclined = sourceFailureCode === "native_permission_declined";
|
||||
const exhausted = permanentFailure || permissionDeclined || attempt >= 3;
|
||||
return {
|
||||
@@ -8353,6 +8354,33 @@ export function nativeSessionRecoveryProjection(input: {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The Runner closed this run's provider session after a turn start whose prompt
|
||||
* may have reached the provider. Every later same-run `turn.start` is refused
|
||||
* the same way, and resending could duplicate the prompt, so the run stops and
|
||||
* the board offers Retry (a new run with a fresh provider session). A start the
|
||||
* Runner proved never dispatched is not classified and still retries.
|
||||
*/
|
||||
export const NATIVE_PROVIDER_SESSION_CLOSED = "native_provider_session_closed";
|
||||
export const NATIVE_PROVIDER_SESSION_CLOSED_ACTION =
|
||||
"The provider session closed after a turn start that may have reached the provider, and this run cannot reopen it. Retry starts a new run with a fresh provider conversation and the saved task context.";
|
||||
const RUNNER_PROVIDER_SESSION_CLOSED = "provider_session_closed";
|
||||
|
||||
/** Reads the Runner's closed class from an authenticated turn.start failure. */
|
||||
function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean {
|
||||
if (!error || typeof error !== "object" || depth > 5) return false;
|
||||
const value = error as Record<string, unknown>;
|
||||
const result = record(value.runnerCommandResult);
|
||||
if (result.commandType === "turn.start" && record(result.result).classification === RUNNER_PROVIDER_SESSION_CLOSED) {
|
||||
return true;
|
||||
}
|
||||
// The structured result can be lost when the failure is rewrapped; the
|
||||
// authenticated command failure text carries the same closed class.
|
||||
const message = error instanceof Error ? error.message : "";
|
||||
if (/\bPRP command turn\.start failed: \{.*"classification":"provider_session_closed"/.test(message)) return true;
|
||||
return runnerClosedTurnStartSession(value.cause, depth + 1);
|
||||
}
|
||||
|
||||
export function nativeSessionFailureSourceCode(
|
||||
error: unknown,
|
||||
):
|
||||
@@ -8363,6 +8391,7 @@ export function nativeSessionFailureSourceCode(
|
||||
| typeof NATIVE_PROVIDER_SIDE_EFFECT_PENDING
|
||||
| "native_provider_approval_required"
|
||||
| "native_provider_usage_limit"
|
||||
| typeof NATIVE_PROVIDER_SESSION_CLOSED
|
||||
| "native_session_cleanup_quarantined"
|
||||
| "native_adopted_runner_authentication_timeout"
|
||||
| "runner_remote_provider_artifact_incompatible"
|
||||
@@ -8424,6 +8453,7 @@ export function nativeSessionFailureSourceCode(
|
||||
return error.failure.httpStatus === 401 || error.failure.httpStatus === 403
|
||||
? "runner_ingress_access_rejected" : "runner_ingress_acquisition_failed";
|
||||
}
|
||||
if (runnerClosedTurnStartSession(error)) return NATIVE_PROVIDER_SESSION_CLOSED;
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (/classification=runtime_package_missing\)/.test(message))
|
||||
return "native_provider_runtime_missing";
|
||||
@@ -12261,6 +12291,8 @@ async function executePaperclipNativeSessionAttempt(
|
||||
? "Provider permission was declined. Inspect the task and give explicit direction before starting further provider work. Automatic recovery is stopped."
|
||||
: sourceFailureCode === "native_provider_approval_required"
|
||||
? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler."
|
||||
: sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
|
||||
? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
|
||||
: sourceFailureCode === "native_session_cleanup_quarantined"
|
||||
? nativeCleanupRefusalAction(completedTurnCleanup)
|
||||
: sourceFailureCode === "runner_remote_provider_verification_unavailable"
|
||||
@@ -12530,6 +12562,8 @@ async function executePaperclipNativeSessionAttempt(
|
||||
? "Provider permission was declined. Inspect the task and give explicit direction before starting further provider work. Automatic recovery is stopped."
|
||||
: sourceFailureCode === "native_provider_approval_required"
|
||||
? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler."
|
||||
: sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED
|
||||
? NATIVE_PROVIDER_SESSION_CLOSED_ACTION
|
||||
: sourceFailureCode === "native_session_cleanup_quarantined"
|
||||
? nativeCleanupRefusalAction(completedTurnCleanup)
|
||||
: sourceFailureCode === "runner_ingress_access_rejected"
|
||||
|
||||
Reference in new issue
Block a user