mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(runner): reject only the overflow call when parallel tool calls exceed the result reserve (#21913)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs call Paperclip tools through the runner's provider tool bridge > - The bridge reserves 768 KiB for each pending call against an 8 MiB budget per turn > - When an agent sends many tool calls in parallel, the reserve for the pending calls can go over the budget, although the bytes the turn really holds are small > - The bridge then reports the durable receipt limit, and the backend stops the whole turn. The run ends as cancelled with `needs_review`, and the task stays blocked with a Retry button > - This pull request rejects only the overflow call as retryable, and keeps the turn stop for real held bytes. It also adds a server backstop: a turn stopped at the receipt limit continues by itself in a fresh session > - The benefit is that a parallel burst of tool calls no longer leaves a task waiting for a person ## Linked Issues or Issue Description **What happened?** In a restart-chaos wave, an opencode agent sent 10 parallel `get_task` calls. The turn already held about 0.7 MB of results. 10 × 768 KiB plus 0.7 MB is more than 8 MiB, so `begin_call` returned `active_turn_receipt_limit()` for the 10th call. This branch does not set `durable_run_receipt_limit_reached`, so it is a temporary reserve overflow. `handle_tool_call` sends every receipt-limit error to `stop_turn_at_tool_receipt_limit`, which interrupts the turn. The run ended as cancelled with `needs_review` and a human review request. The release then blocked the task (`native_continuation_requires_reconciliation`) and showed Retry. No continuation was scheduled. The task moved again only because a person posted a comment ten minutes later. **Expected behavior** The overflow call gets a normal, retryable rejection, and the turn continues. The agent can send the call again after a pending result arrives. If a turn does stop at the receipt limit, Paperclip continues the task in a fresh session, as it does for `agent_tool_call_loop` (#21734). The task does not show Retry. **Steps to reproduce** 1. Start a native turn that holds about 600 KiB of settled tool results. 2. Send 10 tool calls in parallel. 3. On master, the 10th call stops the turn, and the task becomes blocked with Retry. **Paperclip version or commit** `master` atc5218f5ec4(wave), rebased on991f773d7f. ## What Changed - `provider_bridge.rs`: the two `begin_call` branches that fail only because of the per-call result reserve now return a new retryable error (`tool_calls_in_flight`). The provider sees "Too many tool calls are in flight. This call was not executed. Wait for the pending tool results, then retry it." The branches that set `durable_run_receipt_limit_reached` (real held bytes, exact identity capacity, saturated replay history) still stop the turn. - `provider_backend.rs`: the invalid tool call result sets `retryable` from the error, so this rejection is `retryable: true`. Other rejections stay `false`. - `native-tool-receipt-limit.ts` (new): reads the persisted, source-bound `harness.diagnostic` event with code `semantic_tool_turn_receipt_limit` for the run, and holds the lane constants and the hint for the next session. - `status-arbiter.ts`: a run that did not succeed and whose runner stopped it at the receipt limit preserves the task status and schedules a retry with cause `semantic_tool_turn_receipt_limit` (and releases run resources when it was cancelled). This applies only when the task owner holds the decision, with no review, governance gate, open blocker, pause hold or prior `blocked` status, and at most 2 times per chain. Other cases keep their current path. - `native-provider-capacity-retry.ts`: the capacity retry is now one lane of a shared fresh-session retry helper. The new receipt-limit lane creates a `scheduled_retry` run due in 5 seconds with `forceFreshSession` and a liveness instruction that tells the agent to make fewer parallel calls. - `status-decision-committer.ts` and `native-run-finalizer.ts`: wire the new lane, and require the task owner under the issue lock, as for the capacity retry. - `acpx_provider_session.rs` (Claude, Cursor and other ACPX agents) handles the same retryable error per call: it drops the pending sidecar call and resolves only that call with the retryable error. Other `begin_call` errors still fail closed. - The external provider path (Dot/Muse) is out of scope and unchanged: it still fails closed on this overflow, as on master. Making a same-requestId retry safe there needs retry generations across the executor, the driver command ids and the broker receipts. That is a follow-up. - The receipt-limit backstop reads only the diagnostic of the final turn in the run's session. - A person's Stop always wins: the finalizer re-reads the saved Stop on each arbitration attempt, the retry helper refuses under the run lock, and the #21734 tool-call loop continuation is not queued, or is cancelled, for a stopped run. The runner change is runner-image content. The Daytona runner image must be rebuilt before sandboxes get the runner fix. The server backstop works with older images. ## Verification - Red then green, `provider_bridge.rs` unit test `parallel_calls_over_the_result_reserve_reject_only_the_overflow_call`: with about 600 KiB settled, 9 calls are admitted. On master the 10th returns `is_active_turn_receipt_limit()`. Now it is a retryable rejection, the durable flag stays false, and the same call is admitted after one pending result settles. - Negative, `held_result_bytes_over_the_limit_still_stop_the_turn`: real held bytes over the limit still return the receipt limit with the durable flag set. - `cargo test -p paperclip-runner-core -- --skip task_memory::`: all pass (640 lib tests plus the integration suites). The `task_memory` monitor tests are timing-based kernel OOM monitor tests that do not run correctly on macOS (one hangs); this change does not touch them, and CI runs them on Linux. - Red then green, `native-tool-receipt-limit.integration.test.ts` (real Postgres): a run stopped at the receipt limit gets one `scheduled_retry` successor with `forceFreshSession` and the hint. The release does not block the task and creates no recovery action. The task stays `in_progress`. The presentation shows `retry_scheduled` with no Retry, then `working` after promotion. With the master arbiter, no successor is created. A negative case (another diagnostic code) keeps `cancellation_run_only`. - Red then green: ACPX `receiver_rejects_only_the_tool_call_that_overflows_the_result_reserve` (11-call burst through the fake sidecar). - Red then green, server: an earlier turn's diagnostic gives no continuation; a Stop saved before or right after the finalizer's first read gives no continuation and settles the cancellation with no recovery action; the retry helper throws `NativeStatusRaceError` for a stopped run; the loop continuation is not queued, and a queued one is cancelled, for a stopped run. - `status-arbiter.test.ts`: the new branch, the 2-continuation bound, and the `blocked` exclusion. - Focused server tests pass: the new test, `native-provider-overload.integration.test.ts`, `status-arbiter.test.ts`, `native-status-arbiter-corpus.test.ts`, `heartbeat-run-dispatch-work.test.ts`, `native-run-finalizer-telemetry.test.ts`, `native-workspace-finalizer.test.ts` (306 tests). `tsc --noEmit` passes for `server`. - `pnpm verify` exits 2 (a runner-core change needs broad coverage), so required CI supplies it. ## Risks - Dot/Muse agents still lose the turn on this overflow until the follow-up lands (no change from master). - An agent that keeps more than about ten large calls in flight now gets rejections for the extra calls instead of a stopped turn. It can send them again after results arrive. Real held bytes still stop the turn. - The receipt-limit continuation starts a fresh session. The in-turn conversation is lost; the task, comments, documents and saved workspace stay. It runs at most 2 times per chain, then the current path applies. - The retry lane shares the failure-retry count with the capacity lane. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21734 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d16c06308d
commit
3452639a81
14 files changed
+668
-30
No files matched your search
@@ -963,6 +963,7 @@ impl AcpxProviderSession {
|
||||
let mut next_bridge = self.tool_bridge.clone();
|
||||
let mut next_reserved_bridge = self.reserved_tool_bridge.clone();
|
||||
let mut reconciled_events = Vec::with_capacity(events.len());
|
||||
let mut retryable_rejections = Vec::new();
|
||||
for event in events {
|
||||
let mut expose_event = true;
|
||||
match &event {
|
||||
@@ -998,9 +999,19 @@ impl AcpxProviderSession {
|
||||
if let Err(error) =
|
||||
bridge.begin_call(call_id.clone(), operation_id.clone(), input.clone())
|
||||
{
|
||||
return Err(self.fail_closed(LocalRunnerError::invalid(format!(
|
||||
"ACPX provider tool authorization failed: {error}"
|
||||
))));
|
||||
if !error.is_retryable() {
|
||||
return Err(self.fail_closed(LocalRunnerError::invalid(format!(
|
||||
"ACPX provider tool authorization failed: {error}"
|
||||
))));
|
||||
}
|
||||
if let Err(error) = next_state.complete_tool(call_id, operation_id) {
|
||||
return Err(self.fail_closed(error));
|
||||
}
|
||||
retryable_rejections.push((
|
||||
call_id.clone(),
|
||||
error.safe_provider_message().unwrap_or_default().to_owned(),
|
||||
));
|
||||
expose_event = false;
|
||||
}
|
||||
}
|
||||
AcpxProviderStateEvent::SemanticResult(result) => {
|
||||
@@ -1125,6 +1136,22 @@ impl AcpxProviderSession {
|
||||
self.state = next_state;
|
||||
self.tool_bridge = next_bridge;
|
||||
self.reserved_tool_bridge = next_reserved_bridge;
|
||||
if let Some(turn_id) = (!retryable_rejections.is_empty())
|
||||
.then(|| self.state.active_turn_id().map(str::to_owned))
|
||||
.flatten()
|
||||
{
|
||||
for (call_id, message) in retryable_rejections {
|
||||
let response = self
|
||||
.transport
|
||||
.request_tool_resolve(json!({
|
||||
"callId": call_id,
|
||||
"turnId": turn_id,
|
||||
"error": {"message": message},
|
||||
}))
|
||||
.map_err(|(error, _)| self.fail_closed(error))?;
|
||||
self.verify_resolution(&response, "tool")?;
|
||||
}
|
||||
}
|
||||
if let Some(turn_id) = terminal_turn {
|
||||
self.offer_provider_turn(&turn_id, terminal_completed, terminal_live);
|
||||
}
|
||||
|
||||
@@ -104,7 +104,10 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
.try_into()
|
||||
.map_err(|_| "lifetime fence requires three ports")?;
|
||||
// Only the receiver-admission fixture writes this task-owned command journal.
|
||||
let mut admission_journal = if matches!(mode, "admission-tool" | "admission-tool-oversized") {
|
||||
let mut admission_journal = if matches!(
|
||||
mode,
|
||||
"admission-tool" | "admission-tool-oversized" | "admission-tool-burst"
|
||||
) {
|
||||
let path = args
|
||||
.windows(2)
|
||||
.find(|pair| pair[0] == "--journal")
|
||||
@@ -641,7 +644,10 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
if let Some(journal) = admission_journal.as_mut() {
|
||||
write_json(journal, &json!({"request":request}))?;
|
||||
let response = if command == "tool.resolve" {
|
||||
let response = if command == "tool.resolve" && mode == "admission-tool-burst" {
|
||||
json!({"protocolVersion":GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION,
|
||||
"id":id,"ok":true,"result":{"resolved":true}})
|
||||
} else if command == "tool.resolve" {
|
||||
json!({"protocolVersion":GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION,
|
||||
"id":id,"ok":true,"result":{"resolved":
|
||||
request["params"]["callId"] == "call-admission"
|
||||
@@ -660,7 +666,20 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
)
|
||||
};
|
||||
write_json(&mut stdout, &response)?;
|
||||
if command == "turn.start" {
|
||||
if command == "turn.start" && mode == "admission-tool-burst" {
|
||||
for index in 0..11 {
|
||||
let frame = json!({
|
||||
"protocolVersion":GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION,
|
||||
"sequence":next_sequence,"eventType":"runtime.tool_called",
|
||||
"runId":"run-1","turnId":request["params"]["turnId"],
|
||||
"payload":{"callId":format!("call-burst-{index}"),"operationId":"issues.read",
|
||||
"input":{"id":format!("issue-{index}")}},
|
||||
});
|
||||
write_json(journal, &json!({"emitted":frame}))?;
|
||||
write_json(&mut stdout, &frame)?;
|
||||
next_sequence += 1;
|
||||
}
|
||||
} else if command == "turn.start" {
|
||||
let frame = json!({
|
||||
"protocolVersion":GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION,
|
||||
"sequence":next_sequence,"eventType":"runtime.tool_called",
|
||||
|
||||
@@ -72,7 +72,7 @@ fn invalid_tool_call_result(
|
||||
"message": error
|
||||
.safe_provider_message()
|
||||
.unwrap_or(GENERIC_INVALID_TOOL_CALL_MESSAGE),
|
||||
"retryable": false,
|
||||
"retryable": error.is_retryable(),
|
||||
},
|
||||
}),
|
||||
is_error: true,
|
||||
|
||||
@@ -39,6 +39,9 @@ const MAX_SETTLED_CALL_IDS: usize = 65_536;
|
||||
const REPLAY_FILTER_WORDS: usize = 32_768;
|
||||
const ACTIVE_TURN_RECEIPT_LIMIT_MESSAGE: &str =
|
||||
"durable provider tool receipt limit reached for the active turn";
|
||||
const TOOL_CALLS_IN_FLIGHT_MESSAGE: &str =
|
||||
"too many provider tool calls are in flight for the retained result reserve";
|
||||
const TOOL_CALLS_IN_FLIGHT_HINT: &str = "Too many tool calls are in flight. This call was not executed. Wait for the pending tool results, then retry it.";
|
||||
const COMPLETION_INPUT_SCHEMA_HINT: &str = "Invalid paperclip_finish arguments. Required fields: reportedWorkDisposition, summary, completionClaim, evidence, and verification. When reportedWorkDisposition is yielded, continuation must include kind=response_wake or monitor, summary, and idempotencyKey.";
|
||||
const HUMAN_INPUT_SCHEMA_HINT: &str = "Invalid request_human_input arguments. Put version and supersedeOnUserComment at payload.version and payload.supersedeOnUserComment, never at the root. For questions, use payload.version=1 and a complete payload.questionSet. Preserve question and option IDs. Correct the arguments and retry.";
|
||||
const APPROVAL_INPUT_SCHEMA_HINT: &str = "Invalid request_human_input arguments. Put version and supersedeOnUserComment at payload.version and payload.supersedeOnUserComment, never at the root. Keep approval requests in their confirmation or checkbox interactionKind; do not replace them with questions. Correct the arguments and retry.";
|
||||
@@ -310,6 +313,17 @@ impl ProviderBridgeError {
|
||||
self.message == ACTIVE_TURN_RECEIPT_LIMIT_MESSAGE
|
||||
}
|
||||
|
||||
fn tool_calls_in_flight() -> Self {
|
||||
Self {
|
||||
message: TOOL_CALLS_IN_FLIGHT_MESSAGE.to_owned(),
|
||||
safe_provider_message: Some(TOOL_CALLS_IN_FLIGHT_HINT.to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_retryable(&self) -> bool {
|
||||
self.message == TOOL_CALLS_IN_FLIGHT_MESSAGE
|
||||
}
|
||||
|
||||
pub fn safe_provider_message(&self) -> Option<&str> {
|
||||
self.safe_provider_message.as_deref()
|
||||
}
|
||||
@@ -779,7 +793,7 @@ impl ProviderToolBridge {
|
||||
.and_then(|total| total.checked_add(result_reserve))
|
||||
.is_none_or(|total| total > MAX_RETAINED_TOOL_VALUE_BYTES)
|
||||
{
|
||||
return Err(ProviderBridgeError::active_turn_receipt_limit());
|
||||
return Err(ProviderBridgeError::tool_calls_in_flight());
|
||||
}
|
||||
// Reserve the complete encoded receipt, including this exact input and
|
||||
// a maximum-sized result, before accepting work. An admitted call can
|
||||
@@ -791,7 +805,7 @@ impl ProviderToolBridge {
|
||||
return Err(ProviderBridgeError::active_turn_receipt_limit());
|
||||
}
|
||||
if self.ensure_settled_result_capacity(Some(&call)).is_err() {
|
||||
return Err(ProviderBridgeError::active_turn_receipt_limit());
|
||||
return Err(ProviderBridgeError::tool_calls_in_flight());
|
||||
}
|
||||
self.pending.insert(call_id, call.clone());
|
||||
Ok(call)
|
||||
@@ -1656,6 +1670,108 @@ mod tests {
|
||||
bridge
|
||||
}
|
||||
|
||||
fn get_task_bridge() -> ProviderToolBridge {
|
||||
let operation = AuthorizedTool {
|
||||
operation_id: "get_task".to_owned(),
|
||||
version: 1,
|
||||
description: "Read a task.".to_owned(),
|
||||
input_schema: json!({"type": "object"}),
|
||||
response_schema: json!({"type": "object"}),
|
||||
};
|
||||
let mut bridge = ProviderToolBridge::default();
|
||||
bridge
|
||||
.prepare(AuthorizedToolSet {
|
||||
schema: TOOL_SET_SCHEMA.to_owned(),
|
||||
schema_version: 1,
|
||||
catalog_digest: authorized_tool_catalog_digest(std::slice::from_ref(&operation))
|
||||
.unwrap(),
|
||||
operations: vec![operation],
|
||||
})
|
||||
.unwrap();
|
||||
bridge
|
||||
}
|
||||
|
||||
fn settle_get_task(bridge: &mut ProviderToolBridge, call_id: &str, result_bytes: usize) {
|
||||
bridge
|
||||
.begin_call(call_id.to_owned(), "get_task".to_owned(), json!({}))
|
||||
.unwrap();
|
||||
bridge
|
||||
.apply_result(ToolResult {
|
||||
call_id: call_id.to_owned(),
|
||||
operation_id: "get_task".to_owned(),
|
||||
result: json!({"body": "x".repeat(result_bytes)}),
|
||||
is_error: false,
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parallel_calls_over_the_result_reserve_reject_only_the_overflow_call() {
|
||||
let mut bridge = get_task_bridge();
|
||||
settle_get_task(&mut bridge, "settled", 600 * 1024);
|
||||
for index in 0..9 {
|
||||
bridge
|
||||
.begin_call(
|
||||
format!("parallel-{index}"),
|
||||
"get_task".to_owned(),
|
||||
json!({}),
|
||||
)
|
||||
.expect("calls within the result reserve are admitted");
|
||||
}
|
||||
let error = bridge
|
||||
.begin_call("parallel-9".to_owned(), "get_task".to_owned(), json!({}))
|
||||
.expect_err("the call that overflows the reserve is rejected");
|
||||
assert!(!error.is_active_turn_receipt_limit());
|
||||
assert!(error.is_retryable());
|
||||
assert_eq!(
|
||||
error.safe_provider_message(),
|
||||
Some(TOOL_CALLS_IN_FLIGHT_HINT)
|
||||
);
|
||||
assert!(!bridge.durable_run_receipt_limit_reached());
|
||||
assert!(!bridge.has_call_receipt("parallel-9"));
|
||||
assert_eq!(bridge.pending_calls().count(), 9);
|
||||
|
||||
bridge
|
||||
.apply_result(ToolResult {
|
||||
call_id: "parallel-0".to_owned(),
|
||||
operation_id: "get_task".to_owned(),
|
||||
result: json!({"ok": true}),
|
||||
is_error: false,
|
||||
})
|
||||
.unwrap();
|
||||
bridge
|
||||
.begin_call("parallel-9".to_owned(), "get_task".to_owned(), json!({}))
|
||||
.expect("the rejected call is admitted once a pending result settles");
|
||||
assert!(!bridge.durable_run_receipt_limit_reached());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn held_result_bytes_over_the_limit_still_stop_the_turn() {
|
||||
let mut bridge = get_task_bridge();
|
||||
let mut settled = 0;
|
||||
let error = loop {
|
||||
match bridge.begin_call(format!("held-{settled}"), "get_task".to_owned(), json!({})) {
|
||||
Ok(_) => {
|
||||
bridge
|
||||
.apply_result(ToolResult {
|
||||
call_id: format!("held-{settled}"),
|
||||
operation_id: "get_task".to_owned(),
|
||||
result: json!({"body": "x".repeat(760 * 1024)}),
|
||||
is_error: false,
|
||||
})
|
||||
.unwrap();
|
||||
settled += 1;
|
||||
}
|
||||
Err(error) => break error,
|
||||
}
|
||||
};
|
||||
assert!(settled >= 9);
|
||||
assert_eq!(bridge.pending_calls().count(), 0);
|
||||
assert!(error.is_active_turn_receipt_limit());
|
||||
assert!(!error.is_retryable());
|
||||
assert!(bridge.durable_run_receipt_limit_reached());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn completion_summary_enforces_the_canonical_unicode_character_limit() {
|
||||
let mut within_limit = completion_bridge();
|
||||
|
||||
@@ -385,6 +385,68 @@ fn receiver_admits_normal_tool_event_and_resolves_only_correlated_call_once() {
|
||||
check_tool_receiver_admission(false);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn receiver_rejects_only_the_tool_call_that_overflows_the_result_reserve() {
|
||||
let root = std::env::temp_dir().join(format!("acpx-burst-{}", uuid::Uuid::new_v4()));
|
||||
std::fs::create_dir(&root).unwrap();
|
||||
let journal = root.join("commands.jsonl");
|
||||
let mut fixture = config("admission-tool-burst");
|
||||
fixture
|
||||
.transport
|
||||
.args
|
||||
.extend(["--journal".to_owned(), journal.to_str().unwrap().to_owned()]);
|
||||
let mut session = AcpxProviderSession::start(&fixture).unwrap();
|
||||
session
|
||||
.start_turn("turn-admission", "Read the issues", &std::env::temp_dir())
|
||||
.unwrap();
|
||||
let mut exposed = 0;
|
||||
while (0..10).any(|index| {
|
||||
session
|
||||
.state()
|
||||
.pending_tool(&format!("call-burst-{index}"))
|
||||
.is_none()
|
||||
}) || exposed < 10
|
||||
{
|
||||
exposed += session
|
||||
.poll_event(Duration::from_secs(1))
|
||||
.unwrap()
|
||||
.map_or(0, |events| events.len());
|
||||
}
|
||||
for _ in 0..3 {
|
||||
let _ = session.poll_event(Duration::from_millis(100)).unwrap();
|
||||
}
|
||||
assert_eq!(exposed, 10);
|
||||
assert!(session.state().pending_tool("call-burst-10").is_none());
|
||||
session
|
||||
.deliver_tool_result(&paperclip_runner_core::provider_bridge::ToolResult {
|
||||
call_id: "call-burst-0".to_owned(),
|
||||
operation_id: "issues.read".to_owned(),
|
||||
result: json!({"id":"issue-0"}),
|
||||
is_error: false,
|
||||
})
|
||||
.unwrap();
|
||||
session.shutdown("burst test complete").unwrap();
|
||||
let rows: Vec<serde_json::Value> = std::fs::read_to_string(&journal)
|
||||
.unwrap()
|
||||
.lines()
|
||||
.map(|line| serde_json::from_str(line).unwrap())
|
||||
.collect();
|
||||
let resolutions: Vec<_> = rows
|
||||
.iter()
|
||||
.filter_map(|row| row.get("request"))
|
||||
.filter(|request| request["command"] == "tool.resolve")
|
||||
.map(|request| request["params"].clone())
|
||||
.collect();
|
||||
assert_eq!(resolutions.len(), 2);
|
||||
assert_eq!(resolutions[0]["callId"], "call-burst-10");
|
||||
assert!(resolutions[0]["error"]["message"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("Too many tool calls are in flight"));
|
||||
assert_eq!(resolutions[1]["callId"], "call-burst-0");
|
||||
std::fs::remove_dir_all(root).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pi_thinking_effective_identity_is_admitted_before_any_turn() {
|
||||
use paperclip_runner_core::acpx_provider_session::{
|
||||
|
||||
@@ -5,29 +5,74 @@ import { and, eq } from "drizzle-orm";
|
||||
import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { accountingForScheduledRetry, executionFailureRetryCount } from "../execution-recovery-attempt.js";
|
||||
import { NATIVE_PROVIDER_CAPACITY_MAX_RETRIES, NATIVE_PROVIDER_OVERLOADED_CODE, NATIVE_PROVIDER_OVERLOADED_MESSAGE } from "./native-provider-failure.js";
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE, NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS, NATIVE_TOOL_RECEIPT_LIMIT_HINT,
|
||||
NATIVE_TOOL_RECEIPT_LIMIT_MAX_CONTINUATIONS, NATIVE_TOOL_RECEIPT_LIMIT_MESSAGE, nativeRunHasStopRequest } from "./native-tool-receipt-limit.js";
|
||||
import { NativeStatusRaceError } from "./status-decision-committer.js";
|
||||
|
||||
/** Called under the finalizer's issue/coordinator locks, in the status-effect transaction. */
|
||||
export async function materializeNativeProviderCapacityRetry(input: {
|
||||
type NativeRetryTarget = {
|
||||
tx: Db;
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
runId: string;
|
||||
agentId: string;
|
||||
}) {
|
||||
};
|
||||
|
||||
type NativeRetryLane = {
|
||||
reason: string;
|
||||
maxRetries: number;
|
||||
delayMs: (attempt: number) => number;
|
||||
message: string;
|
||||
idempotencyPrefix: string;
|
||||
exhaustedError: string;
|
||||
context?: (run: typeof heartbeatRuns.$inferSelect) => Record<string, unknown>;
|
||||
refuseStopped?: boolean;
|
||||
};
|
||||
|
||||
/** Called under the finalizer's issue/coordinator locks, in the status-effect transaction. */
|
||||
export async function materializeNativeProviderCapacityRetry(input: NativeRetryTarget) {
|
||||
return materializeNativeFreshSessionRetry(input, {
|
||||
reason: NATIVE_PROVIDER_OVERLOADED_CODE,
|
||||
maxRetries: NATIVE_PROVIDER_CAPACITY_MAX_RETRIES,
|
||||
delayMs: attempt => 60_000 * 2 ** (attempt - 1),
|
||||
message: NATIVE_PROVIDER_OVERLOADED_MESSAGE,
|
||||
idempotencyPrefix: "native-capacity",
|
||||
exhaustedError: "native_capacity_retry_exhausted",
|
||||
});
|
||||
}
|
||||
|
||||
export async function materializeNativeToolReceiptLimitContinuation(input: NativeRetryTarget) {
|
||||
return materializeNativeFreshSessionRetry(input, {
|
||||
reason: NATIVE_TOOL_RECEIPT_LIMIT_CODE,
|
||||
maxRetries: NATIVE_TOOL_RECEIPT_LIMIT_MAX_CONTINUATIONS,
|
||||
delayMs: () => NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS,
|
||||
message: NATIVE_TOOL_RECEIPT_LIMIT_MESSAGE,
|
||||
idempotencyPrefix: "native-tool-receipt-limit",
|
||||
exhaustedError: "native_tool_receipt_limit_continuation_exhausted",
|
||||
refuseStopped: true,
|
||||
context: run => ({
|
||||
livenessContinuationSourceRunId: run.id,
|
||||
livenessContinuationReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE,
|
||||
livenessContinuationInstruction: NATIVE_TOOL_RECEIPT_LIMIT_HINT,
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
async function materializeNativeFreshSessionRetry(input: NativeRetryTarget, lane: NativeRetryLane) {
|
||||
const [run] = await input.tx.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId),
|
||||
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
||||
)).for("update");
|
||||
if (!run || run.runtimeMode !== "native") throw new Error("native_capacity_retry_run_binding_mismatch");
|
||||
if (lane.refuseStopped && nativeRunHasStopRequest(run.resultJson)) throw new NativeStatusRaceError();
|
||||
// Share the predecessor claim with every other failure-retry lane.
|
||||
const [existing] = await input.tx.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.retryOfRunId, run.id),
|
||||
)).limit(1);
|
||||
if (existing) return existing.id;
|
||||
const attempt = executionFailureRetryCount(run) + 1;
|
||||
if (attempt > NATIVE_PROVIDER_CAPACITY_MAX_RETRIES) throw new Error("native_capacity_retry_exhausted");
|
||||
if (attempt > lane.maxRetries) throw new Error(lane.exhaustedError);
|
||||
const now = new Date();
|
||||
const dueAt = new Date(now.getTime() + 60_000 * 2 ** (attempt - 1));
|
||||
const dueAt = new Date(now.getTime() + lane.delayMs(attempt));
|
||||
const retryId = randomUUID();
|
||||
const predecessorContext = { ...run.contextSnapshot };
|
||||
// Resume receipts and delivered wake input belong to the failed run. The
|
||||
@@ -45,18 +90,19 @@ export async function materializeNativeProviderCapacityRetry(input: {
|
||||
taskId: input.issueId,
|
||||
forceFreshSession: true,
|
||||
retryOfRunId: run.id,
|
||||
retryReason: NATIVE_PROVIDER_OVERLOADED_CODE,
|
||||
wakeReason: NATIVE_PROVIDER_OVERLOADED_CODE,
|
||||
retryReason: lane.reason,
|
||||
wakeReason: lane.reason,
|
||||
scheduledRetryAttempt: attempt,
|
||||
scheduledRetryAt: dueAt.toISOString(),
|
||||
executionRetryAccounting: accountingForScheduledRetry(run, NATIVE_PROVIDER_OVERLOADED_CODE, attempt),
|
||||
executionRetryAccounting: accountingForScheduledRetry(run, lane.reason, attempt),
|
||||
...lane.context?.(run),
|
||||
};
|
||||
await notifyIssueAttentionRetentionWork(input.tx, { companyId: input.companyId, issueId: input.issueId, runId: run.id });
|
||||
await notifyScheduledRetryDispatch(input.tx, { companyId: run.companyId, runId: run.id });
|
||||
const [wake] = await input.tx.insert(agentWakeupRequests).values({
|
||||
companyId: run.companyId, agentId: run.agentId, source: "automation", triggerDetail: "system",
|
||||
reason: NATIVE_PROVIDER_OVERLOADED_CODE, status: "queued", runId: retryId,
|
||||
idempotencyKey: `native-capacity:${run.companyId}:${run.id}`,
|
||||
reason: lane.reason, status: "queued", runId: retryId,
|
||||
idempotencyKey: `${lane.idempotencyPrefix}:${run.companyId}:${run.id}`,
|
||||
requestedByActorType: "system", requestedByActorId: "native_finalizer",
|
||||
payload: contextSnapshot,
|
||||
}).returning({ id: agentWakeupRequests.id });
|
||||
@@ -64,9 +110,9 @@ export async function materializeNativeProviderCapacityRetry(input: {
|
||||
id: retryId, companyId: run.companyId, agentId: run.agentId,
|
||||
invocationSource: "automation", triggerDetail: "system", status: "scheduled_retry",
|
||||
responsibleUserId: run.responsibleUserId, wakeupRequestId: wake.id,
|
||||
retryOfRunId: run.id, scheduledRetryReason: NATIVE_PROVIDER_OVERLOADED_CODE,
|
||||
retryOfRunId: run.id, scheduledRetryReason: lane.reason,
|
||||
scheduledRetryAttempt: attempt, scheduledRetryAt: dueAt,
|
||||
error: NATIVE_PROVIDER_OVERLOADED_MESSAGE, errorCode: NATIVE_PROVIDER_OVERLOADED_CODE,
|
||||
error: lane.message, errorCode: lane.reason,
|
||||
contextSnapshot,
|
||||
});
|
||||
return retryId;
|
||||
|
||||
@@ -32,6 +32,7 @@ import { settleExternalConversation } from "../slack-conversation-lifecycle.js";
|
||||
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
|
||||
import { isFailedTurnContinuationRun } from "./native-failed-turn-continuation.js";
|
||||
import { readPersistedNativeFailureEvidence } from "./native-provider-failure-evidence.js";
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE, cancelStoppedRunContinuation, nativeRunHasStopRequest, readPersistedNativeToolReceiptLimitStop } from "./native-tool-receipt-limit.js";
|
||||
import { explainFailedRun, explainNativeTurnFailure, NATIVE_RUN_FAILED_UNEXPLAINED } from "./native-run-failure-explanation.js";
|
||||
import { dismissAutomaticCompletionReviews } from "./automatic-completion-reviews.js";
|
||||
import { findSupersededExternalReviewRequests } from "./superseded-external-review-requests.js";
|
||||
@@ -1607,6 +1608,10 @@ export async function finalizeNativeRun(input: {
|
||||
const allowIncompleteContinuation = record(sourceWake?.payload).continuationIdempotencyKey !== "native-completion-incomplete";
|
||||
let supersedesAssessmentId: string | null = coordinator.assessmentId;
|
||||
for (let attempt = 0; attempt < 3; attempt += 1) {
|
||||
const savedStop = attempt === 0 ? run.resultJson : await input.db.select({ resultJson: heartbeatRuns.resultJson })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId)))
|
||||
.then(rows => rows[0]?.resultJson ?? run.resultJson);
|
||||
const stopRequested = nativeRunHasStopRequest(savedStop);
|
||||
const authoritativeIssue = await input.db
|
||||
.select()
|
||||
.from(issues)
|
||||
@@ -1747,6 +1752,7 @@ export async function finalizeNativeRun(input: {
|
||||
// issue route decides `status: "done"`. Review runs keep their own path.
|
||||
const reviewStage = isConversation(authoritativeIssue) || reviewContext
|
||||
? { kind: "none" as const } : completionReviewStageEntry(authoritativeIssue, run.agentId);
|
||||
await cancelStoppedRunContinuation(input.db, { companyId: run.companyId, resultJson: savedStop }, toolCallLoopContinuationKey(run.id));
|
||||
const statusInput: Parameters<typeof resolveNativeFinalizerStatus>[0] = {
|
||||
requiredQualityReviewSatisfied: qualityReview.satisfied,
|
||||
executionStageApprovedByRun: await executionStageApprovedByRun(input.db, run, authoritativeIssue),
|
||||
@@ -1761,13 +1767,16 @@ export async function finalizeNativeRun(input: {
|
||||
providerModelRejected: providerFailure?.errorCode === "native_provider_model_rejected" && ownsProviderFailureDecision,
|
||||
providerOverloaded: providerFailure?.errorCode === "native_provider_overloaded" && ownsProviderFailureDecision,
|
||||
providerFailureSuperseded: providerFailure?.errorCode === "native_provider_overloaded" && !ownsProviderFailureDecision,
|
||||
toolReceiptLimitStopped: terminalState !== "succeeded" && ownsProviderFailureDecision && !reviewContext
|
||||
&& !stopRequested && await readPersistedNativeToolReceiptLimitStop(input.db, run, resultRow.turnId),
|
||||
failureRetryCount: executionFailureRetryCount(run),
|
||||
// Only the task's current owner may turn its own failed continuation
|
||||
// into a blocker; the committer rechecks that under the issue lock.
|
||||
failedTurnContinuationUsed: terminalState === "failed" && ownsProviderFailureDecision && !reviewContext
|
||||
&& await isFailedTurnContinuationRun(input.db, run, authoritativeIssue.id),
|
||||
toolCallLoopContinuation: assessment.reportedDisposition === "yielded"
|
||||
&& assessment.continuation?.idempotencyKey === toolCallLoopContinuationKey(run.id),
|
||||
&& assessment.continuation?.idempotencyKey === toolCallLoopContinuationKey(run.id)
|
||||
&& !stopRequested,
|
||||
...(reviewContext ? { nativeReviewOutcome: nativeReview
|
||||
? nativeReview.interaction.status === "pending" ? "pending" as const : "resolved" as const
|
||||
: "stale" as const } : {}),
|
||||
@@ -1922,6 +1931,7 @@ export async function finalizeNativeRun(input: {
|
||||
: undefined,
|
||||
requireProviderFailureOwner: decision.reasonCode?.startsWith("native_provider_overloaded")
|
||||
|| decision.reasonCode === "native_run_failed_continuation_exhausted"
|
||||
|| decision.reasonCode === NATIVE_TOOL_RECEIPT_LIMIT_CODE
|
||||
? { agentId: run.agentId, reviewContext } : undefined,
|
||||
requireBoardResponseWaitSource:
|
||||
decision.reasonCode === "board_response_waiting" || repairBoardResponseWait
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, agentWakeupRequests, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, statusDecisionEffects, statusDecisions } from "@paperclipai/db";
|
||||
import type { ControlPlanePort, NativeExecutionInputV1, PrpEvent, PrpTerminalState } from "@paperclipai/paperclip-runner";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
const receiptLimitRead = vi.hoisted(() => ({ after: null as null | (() => Promise<unknown>) }));
|
||||
vi.mock("./native-tool-receipt-limit.js", async (importOriginal) => {
|
||||
const original = await importOriginal<typeof import("./native-tool-receipt-limit.js")>();
|
||||
return { ...original, readPersistedNativeToolReceiptLimitStop: async (...args: Parameters<typeof original.readPersistedNativeToolReceiptLimitStop>) => {
|
||||
const stopped = await original.readPersistedNativeToolReceiptLimitStop(...args);
|
||||
const after = receiptLimitRead.after;
|
||||
receiptLimitRead.after = null;
|
||||
await after?.();
|
||||
return stopped;
|
||||
} };
|
||||
});
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
|
||||
}));
|
||||
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { finalizeNativeRun } from "./native-run-finalizer.js";
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE, NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS, NATIVE_TOOL_RECEIPT_LIMIT_HINT,
|
||||
readPersistedNativeToolReceiptLimitStop } from "./native-tool-receipt-limit.js";
|
||||
import { heartbeatService } from "../heartbeat.js";
|
||||
import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js";
|
||||
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
|
||||
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
|
||||
import { materializeNativeToolReceiptLimitContinuation } from "./native-provider-capacity-retry.js";
|
||||
import { NativeStatusRaceError } from "./status-decision-committer.js";
|
||||
|
||||
describe("native turn stopped at its tool receipt limit", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-tool-receipt-limit-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Receipt limit regression", issuePrefix: "RCL" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Receipt worker", status: "active", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "test-model" } });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
let finalTurnId: string | null = null;
|
||||
async function stopAtReceiptLimit(runId: string, issueId: string, diagnosticCode: string, beforeFinalize?: () => Promise<unknown>,
|
||||
diagnosticTurnId?: string) {
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
const [run] = await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "codex", model: "test-model" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task using its history.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
||||
const event: PrpEvent = {
|
||||
schema: "paperclip.prp.event.v1", runId, normalizedSessionId: native.normalizedSessionId, turnId: diagnosticTurnId ?? native.turnId,
|
||||
sourceInstanceId: native.runnerInstanceId, sourceEventId: "receipt-limit", sourceSeq: 1, sourceKind: "runner", eventType: "harness.diagnostic", schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(),
|
||||
payload: { provider: "opencode", code: diagnosticCode, operationId: "get_task", callId: "call-80",
|
||||
message: "The active provider turn reached its durable semantic-tool receipt limit and was interrupted", paperclipExecuted: false },
|
||||
};
|
||||
const terminal: PrpTerminalState = { schema: "paperclip.prp.terminal.v1", turnTerminalState: "interrupted", runTerminalState: "cancelled", reportedWorkDisposition: "needs_review" };
|
||||
const result = { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "needs_review" as const,
|
||||
summary: "The OpenCode run stopped before it completed.",
|
||||
completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: contractInput.revision },
|
||||
attentionRequests: [{ kind: "review" as const, summary: "Review the stopped OpenCode run before continuing.", ownerClass: "human" as const }] };
|
||||
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId }, backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
|
||||
await options.controlPlane.appendEvent(event);
|
||||
await options.controlPlane.completeRun({ result, terminal, turnId: native.turnId });
|
||||
return { result, terminal, turnId: native.turnId, normalizedSessionId: native.normalizedSessionId, providerSessionId: "fixture-session", driverKind: "codex_app_server", driverVersion: "fixture", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
const unexpected = async () => { throw new Error("Unexpected provider call"); };
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } });
|
||||
await beforeFinalize?.();
|
||||
for (let replay = 0; replay < 2; replay++) await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
expect(provider.execute).toHaveBeenCalledTimes(1);
|
||||
finalTurnId = native.turnId;
|
||||
return (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!;
|
||||
}
|
||||
|
||||
function release() {
|
||||
const unexpected = vi.fn(async () => { throw new Error("Unexpected recovery"); });
|
||||
return createReleaseIssueExecution({
|
||||
issueLock: createPostgresWakeQueueAdapter(db, { resolveResponsibleUserId: unexpected, getRoutineEnv: unexpected, resolveSessionBeforeForWakeup: unexpected }),
|
||||
recovery: { escalateStrandedAssignedIssue: unexpected, escalateStrandedRecoveryIssueInPlace: unexpected },
|
||||
});
|
||||
}
|
||||
|
||||
it("continues the task at once in a fresh session and shows no Retry", async () => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Parallel reads task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId, wakeCommentId: "original-message" } });
|
||||
const stopped = await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE);
|
||||
expect(stopped.status).toBe("cancelled");
|
||||
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, finalTurnId)).toBe(true);
|
||||
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, "another-turn")).toBe(false);
|
||||
|
||||
const [retry, ...others] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId)));
|
||||
expect(others).toHaveLength(0);
|
||||
expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, scheduledRetryAttempt: 1,
|
||||
contextSnapshot: { issueId, forceFreshSession: true, livenessContinuationSourceRunId: runId,
|
||||
livenessContinuationReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, livenessContinuationInstruction: NATIVE_TOOL_RECEIPT_LIMIT_HINT } });
|
||||
expect(retry!.contextSnapshot).not.toHaveProperty("wakeCommentId");
|
||||
expect(retry!.scheduledRetryAt!.getTime() - Date.now()).toBeLessThanOrEqual(NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS);
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision).toMatchObject({ reasonCode: NATIVE_TOOL_RECEIPT_LIMIT_CODE });
|
||||
expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, decision!.id)))
|
||||
.toEqual(expect.arrayContaining([expect.objectContaining({ effectKind: "schedule_retry", targetType: "heartbeat_run", targetId: retry!.id })]));
|
||||
|
||||
expect((await release()({ companyId, runId, now: new Date() })).outcome.kind).toBe("released");
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
expect(issue).toMatchObject({ status: "in_progress", assigneeAgentId: agentId });
|
||||
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "retry_scheduled", canRetry: false, userAction: null });
|
||||
const scheduler = heartbeatService(db, { runtimeEnv: {} });
|
||||
expect(await scheduler.promoteDueScheduledRetries(new Date())).toMatchObject({ promoted: 0 });
|
||||
expect(await scheduler.promoteDueScheduledRetries(retry!.scheduledRetryAt!)).toMatchObject({ promoted: 1, runIds: [retry!.id] });
|
||||
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "working", canRetry: false, userAction: null });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, retry!.id))).toHaveLength(1);
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, retry!.id));
|
||||
});
|
||||
|
||||
it("does not continue a turn a person stopped while it settled", async () => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Stopped task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
|
||||
await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE, () => db.update(heartbeatRuns)
|
||||
.set({ resultJson: { cancelledByActorType: "user", cancelledByUserId: "receipt-owner" } }).where(eq(heartbeatRuns.id, runId)));
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision?.reasonCode).not.toBe(NATIVE_TOOL_RECEIPT_LIMIT_CODE);
|
||||
});
|
||||
|
||||
it("settles the cancellation when a person stops the run after finalization first read it", async () => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Late stop task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
|
||||
await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE, async () => {
|
||||
receiptLimitRead.after = () => db.update(heartbeatRuns)
|
||||
.set({ resultJson: { cancelledByActorType: "user", cancelledByUserId: "receipt-owner" } }).where(eq(heartbeatRuns.id, runId));
|
||||
});
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(coordinator).toMatchObject({ phase: "committed", failureCode: null });
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not continue when only an earlier turn hit the receipt limit", async () => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Earlier turn task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
|
||||
const stopped = await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE, undefined, "earlier-provider-turn");
|
||||
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, finalTurnId)).toBe(false);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
|
||||
});
|
||||
|
||||
it("refuses to create the continuation under the run lock once a person's Stop is saved", async () => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Locked stop task", status: "in_progress", assigneeAgentId: agentId });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "cancelled", runtimeMode: "native", nativeIssueId: issueId,
|
||||
contextSnapshot: { issueId }, resultJson: { cancelledByActorType: "user", cancelledByUserId: "receipt-owner" } });
|
||||
await expect(db.transaction(tx => materializeNativeToolReceiptLimitContinuation({ tx: tx as never, companyId, issueId, runId, agentId })))
|
||||
.rejects.toBeInstanceOf(NativeStatusRaceError);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, `native-tool-receipt-limit:${companyId}:${runId}`))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("leaves another cancelled turn to its existing recovery", async () => {
|
||||
const issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Other diagnostic task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
|
||||
const stopped = await stopAtReceiptLimit(runId, issueId, "semantic_tool_denied");
|
||||
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, finalTurnId)).toBe(false);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, heartbeatRunEvents, type heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { validatePrpEvent } from "../../vendor/paperclip-runner/index.js";
|
||||
import { nativeSha256 } from "./canonical.js";
|
||||
import { readRunCancellation } from "../run-cancellation.js";
|
||||
|
||||
export const NATIVE_TOOL_RECEIPT_LIMIT_CODE = "semantic_tool_turn_receipt_limit";
|
||||
export const NATIVE_TOOL_RECEIPT_LIMIT_MAX_CONTINUATIONS = 2;
|
||||
export const NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS = 5_000;
|
||||
export const NATIVE_TOOL_RECEIPT_LIMIT_MESSAGE = "The agent's turn held too many tool results at once. Paperclip continues the task in a fresh session.";
|
||||
export const NATIVE_TOOL_RECEIPT_LIMIT_HINT = "The previous turn stopped because it held too many tool results at once. "
|
||||
+ "Continue the task from its saved work. Make fewer tool calls in parallel, and do not re-read data you already have.";
|
||||
|
||||
export function nativeRunHasStopRequest(resultJson: Record<string, unknown> | null | undefined): boolean {
|
||||
if (!resultJson) return false;
|
||||
const cancellation = readRunCancellation(resultJson);
|
||||
return Boolean(resultJson.nativeCancellation || resultJson.executionCancellation || resultJson.cancelledByActorType
|
||||
|| resultJson.cancelledByUserId || resultJson.interruptedByActorType || resultJson.queuedCommentInterruptQueueId
|
||||
|| resultJson.issueMutationStopId || resultJson.operatorInterrupted === true
|
||||
|| (cancellation && cancellation.initiator.type !== "system" && cancellation.initiator.type !== "provider"));
|
||||
}
|
||||
|
||||
export async function cancelStoppedRunContinuation(
|
||||
db: Pick<Db, "update">,
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "companyId" | "resultJson">,
|
||||
idempotencyKey: string,
|
||||
): Promise<number> {
|
||||
if (!nativeRunHasStopRequest(run.resultJson)) return 0;
|
||||
const now = new Date();
|
||||
const cancelled = await db.update(agentWakeupRequests).set({
|
||||
status: "cancelled", finishedAt: now, updatedAt: now, error: "The run that asked for this continuation was stopped",
|
||||
}).where(and(
|
||||
eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.idempotencyKey, idempotencyKey),
|
||||
inArray(agentWakeupRequests.status, ["queued", "claimed", "deferred_issue_execution"]), isNull(agentWakeupRequests.runId),
|
||||
)).returning({ id: agentWakeupRequests.id });
|
||||
return cancelled.length;
|
||||
}
|
||||
|
||||
export async function readPersistedNativeToolReceiptLimitStop(
|
||||
db: Pick<Db, "select">,
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runtimeMode" | "runnerInstanceId" | "nativeSessionId">,
|
||||
turnId: string | null,
|
||||
): Promise<boolean> {
|
||||
if (run.runtimeMode !== "native" || !run.runnerInstanceId || !run.nativeSessionId || !turnId) return false;
|
||||
const rows = await db.select().from(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
|
||||
eq(heartbeatRunEvents.sourceInstanceId, run.runnerInstanceId), eq(heartbeatRunEvents.eventType, "harness.diagnostic"),
|
||||
sql`${heartbeatRunEvents.payload}->'prpEvent'->'payload'->>'code' = ${NATIVE_TOOL_RECEIPT_LIMIT_CODE}`,
|
||||
sql`${heartbeatRunEvents.payload}->'prpEvent'->>'turnId' = ${turnId}`,
|
||||
)).limit(4);
|
||||
return rows.some(row => {
|
||||
const prpEvent = (row.payload as Record<string, unknown> | null)?.prpEvent;
|
||||
const parsed = validatePrpEvent(prpEvent);
|
||||
return parsed.ok && parsed.event.runId === run.id && parsed.event.sourceKind === "runner"
|
||||
&& parsed.event.turnId === turnId && parsed.event.normalizedSessionId === run.nativeSessionId
|
||||
&& parsed.event.sourceInstanceId === run.runnerInstanceId && parsed.event.eventType === row.eventType
|
||||
&& parsed.event.sourceEventId === row.sourceEventId && parsed.event.sourceSeq === row.sourceSeq
|
||||
&& row.sourcePayloadSha256 === nativeSha256(parsed.event);
|
||||
});
|
||||
}
|
||||
@@ -32,6 +32,7 @@ import { documentService } from "../documents.js";
|
||||
import { issueService } from "../issues.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
import { TOOL_CALL_LOOP_HINT, ToolCallLoopError } from "./repeated-tool-failure-guard.js";
|
||||
import { cancelStoppedRunContinuation } from "./native-tool-receipt-limit.js";
|
||||
import { PaperclipRunnerSemanticAuthority } from "./runner-semantic-authority.js";
|
||||
import { createAssignedMcpTools } from "./assigned-mcp-tools.js";
|
||||
import type { ToolGatewayService } from "../tool-gateway.js";
|
||||
@@ -1638,6 +1639,53 @@ describe("PaperclipRunnerToolAuthority", () => {
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key));
|
||||
});
|
||||
|
||||
it("queues no tool-call loop continuation for a run a person stopped, and cancels one queued before the Stop", async () => {
|
||||
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workMode: "standard" });
|
||||
const key = `agent_tool_call_loop:${runId}`;
|
||||
const loop = new ToolCallLoopError("call_api", 6, 6, ["operationId"], 400, "refused");
|
||||
await expect(authority.scheduleToolCallLoopContinuation(loop)).resolves.toBe(true);
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
const stopped = { ...before!, resultJson: { ...before!.resultJson, cancelledByActorType: "user", cancelledByUserId: "board-user" } };
|
||||
await db.update(heartbeatRuns).set({ resultJson: stopped.resultJson }).where(eq(heartbeatRuns.id, runId));
|
||||
try {
|
||||
await expect(cancelStoppedRunContinuation(db, stopped, key)).resolves.toBe(1);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key)))
|
||||
.map(row => row.status)).toEqual(["cancelled"]);
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key));
|
||||
await expect(authority.scheduleToolCallLoopContinuation(loop)).resolves.toBe(false);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key))).toHaveLength(0);
|
||||
} finally {
|
||||
await db.update(heartbeatRuns).set({ resultJson: before!.resultJson }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key));
|
||||
}
|
||||
});
|
||||
|
||||
it("orders the loop continuation's locks task first, like Stop, so a concurrent Stop wins without a deadlock", async () => {
|
||||
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workMode: "standard" });
|
||||
const key = `agent_tool_call_loop:${runId}`;
|
||||
const loop = new ToolCallLoopError("call_api", 6, 6, ["operationId"], 400, "refused");
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
let holdingTask!: () => void;
|
||||
const taskLocked = new Promise<void>(resolve => { holdingTask = resolve; });
|
||||
try {
|
||||
const stop = db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, issueId)).for("update");
|
||||
holdingTask();
|
||||
await new Promise(resolve => setTimeout(resolve, 300));
|
||||
await tx.update(heartbeatRuns).set({ resultJson: { ...before!.resultJson, cancelledByActorType: "user", cancelledByUserId: "board-user" } })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
});
|
||||
await taskLocked;
|
||||
const scheduled = authority.scheduleToolCallLoopContinuation(loop);
|
||||
await expect(stop).resolves.toBeUndefined();
|
||||
await expect(scheduled).resolves.toBe(false);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key))).toHaveLength(0);
|
||||
} finally {
|
||||
await db.update(heartbeatRuns).set({ resultJson: before!.resultJson }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, key));
|
||||
}
|
||||
});
|
||||
|
||||
it("creates ordinary children, preserves blockers, and deduplicates across runs", async () => {
|
||||
const wakes: Array<{ agentId: string; options: Record<string, unknown> }> =
|
||||
[];
|
||||
|
||||
@@ -80,6 +80,7 @@ import { finishPendingRunnerApiCaptureDeletes, markRunnerApiCaptureEvictions, RU
|
||||
import { acquireRunnerApiResponseSlot, RUNNER_API_RESPONSE_DEADLINE_MS, RunnerApiCaptureReservationReclaimedError, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js";
|
||||
import { and, desc, eq, inArray, isNull, notInArray, sql } from "drizzle-orm";
|
||||
import { AGENT_TOOL_CALL_LOOP, TOOL_CALL_LOOP_HINT, toolCallLoopContinuationKey, type ToolCallLoopError } from "./repeated-tool-failure-guard.js";
|
||||
import { nativeRunHasStopRequest } from "./native-tool-receipt-limit.js";
|
||||
import { enqueueWake } from "./status-decision-committer.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
@@ -357,7 +358,13 @@ export class PaperclipRunnerToolAuthority {
|
||||
const review = this.binding.nativeReview;
|
||||
const loopContext = { sourceRunId: this.binding.runId, continuationCause: AGENT_TOOL_CALL_LOOP,
|
||||
tool: loop.tool, refusals: loop.refusals, runRefusals: loop.runRefusals, argumentNames: loop.argumentNames };
|
||||
await this.db.transaction(tx => enqueueWake({
|
||||
return this.db.transaction(async tx => {
|
||||
await tx.select({ id: issues.id }).from(issues).where(and(
|
||||
eq(issues.id, this.binding.issueId), eq(issues.companyId, this.binding.companyId))).for("update");
|
||||
const [run] = await tx.select({ resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId))).for("share");
|
||||
if (!run || nativeRunHasStopRequest(run.resultJson)) return false;
|
||||
await enqueueWake({
|
||||
tx: tx as unknown as Db,
|
||||
originRunId: this.binding.runId,
|
||||
companyId: this.binding.companyId,
|
||||
@@ -368,8 +375,9 @@ export class PaperclipRunnerToolAuthority {
|
||||
payload: { ...(review ?? {}), ...loopContext },
|
||||
contextSnapshot: { ...(review ?? {}), forceFreshSession: true, livenessContinuationSourceRunId: this.binding.runId,
|
||||
livenessContinuationReason: AGENT_TOOL_CALL_LOOP, livenessContinuationInstruction: TOOL_CALL_LOOP_HINT },
|
||||
}));
|
||||
return true;
|
||||
});
|
||||
return true;
|
||||
});
|
||||
}
|
||||
constructor(readonly db: Db, readonly binding: Binding) {}
|
||||
|
||||
|
||||
@@ -157,6 +157,20 @@ describe("native status authority", () => {
|
||||
expect(decision.effects).toEqual([{ kind: "release_checkout" }]);
|
||||
});
|
||||
|
||||
it("continues a turn stopped at its tool receipt limit in a fresh session, within its bound", () => {
|
||||
const stopped = arbitrate({ terminalState: "cancelled", toolReceiptLimitStopped: true });
|
||||
expect(stopped).toMatchObject({ statusAction: "preserve", toStatus: "in_progress", reasonCode: "semantic_tool_turn_receipt_limit" });
|
||||
expect(stopped.effects).toEqual([
|
||||
{ kind: "schedule_retry", cause: "semantic_tool_turn_receipt_limit", summary: "Continue the task in a fresh session.", agentId: expect.any(String) },
|
||||
{ kind: "release_run_resources" },
|
||||
]);
|
||||
expect(arbitrate({ terminalState: "failed", toolReceiptLimitStopped: true }).effects)
|
||||
.toEqual([expect.objectContaining({ kind: "schedule_retry", cause: "semantic_tool_turn_receipt_limit" })]);
|
||||
expect(arbitrate({ terminalState: "cancelled", toolReceiptLimitStopped: true, failureRetryCount: 2 }).reasonCode).toBe("cancellation_run_only");
|
||||
expect(arbitrate({ terminalState: "cancelled", toolReceiptLimitStopped: true, priorIssueStatus: "blocked" }).reasonCode).toBe("cancellation_run_only");
|
||||
expect(arbitrate({ terminalState: "cancelled" }).reasonCode).toBe("cancellation_run_only");
|
||||
});
|
||||
|
||||
it("treats only the authorized Board response_wake as passive and preserves governance", () => {
|
||||
const passive = assessment({
|
||||
reportedDisposition: "yielded",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { COMPLETION_REVIEW_STAGE_REASON, COMPLETION_REVIEW_STAGE_REJECTED_REASON, type CompletionReviewStageEntry } from "../issue-execution-policy.js";
|
||||
import type { NativeEvidenceAssessment } from "./evidence-classifier.js";
|
||||
import { NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_CAPACITY_MAX_RETRIES, NATIVE_PROVIDER_OVERLOADED_CODE, NATIVE_PROVIDER_OVERLOADED_MESSAGE } from "./native-provider-failure.js";
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE, NATIVE_TOOL_RECEIPT_LIMIT_MAX_CONTINUATIONS } from "./native-tool-receipt-limit.js";
|
||||
|
||||
export const NATIVE_STATUS_ARBITER_POLICY_VERSION = "phase6-v14";
|
||||
|
||||
@@ -256,6 +257,7 @@ export function arbitrateNativeStatus(input: {
|
||||
providerModelRejected?: boolean;
|
||||
providerOverloaded?: boolean;
|
||||
providerFailureSuperseded?: boolean;
|
||||
toolReceiptLimitStopped?: boolean;
|
||||
failureRetryCount?: number;
|
||||
/**
|
||||
* This run was itself the automatic continuation of an earlier failed turn.
|
||||
@@ -368,6 +370,20 @@ export function arbitrateNativeStatus(input: {
|
||||
],
|
||||
};
|
||||
}
|
||||
if (input.toolReceiptLimitStopped && input.terminalState !== "succeeded" && !input.nativeReviewOutcome &&
|
||||
!input.governanceGate && !input.hasUnresolvedIssueBlockers && !input.hasActivePauseHold &&
|
||||
input.priorIssueStatus !== "blocked" &&
|
||||
(input.failureRetryCount ?? 0) < NATIVE_TOOL_RECEIPT_LIMIT_MAX_CONTINUATIONS) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "preserve", toStatus: input.priorIssueStatus,
|
||||
reasonCode: NATIVE_TOOL_RECEIPT_LIMIT_CODE, unblockDescriptor: null,
|
||||
effects: [
|
||||
{ kind: "schedule_retry", cause: NATIVE_TOOL_RECEIPT_LIMIT_CODE, summary: "Continue the task in a fresh session.", agentId: input.agentId },
|
||||
...(input.terminalState === "cancelled" ? [{ kind: "release_run_resources" as const }] : []),
|
||||
],
|
||||
};
|
||||
}
|
||||
if (input.nativeReviewOutcome) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
|
||||
@@ -87,8 +87,9 @@ import {
|
||||
type ActivityPublication,
|
||||
} from "../activity-log.js";
|
||||
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
||||
import { materializeNativeProviderCapacityRetry } from "./native-provider-capacity-retry.js";
|
||||
import { materializeNativeProviderCapacityRetry, materializeNativeToolReceiptLimitContinuation } from "./native-provider-capacity-retry.js";
|
||||
import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-provider-failure.js";
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE } from "./native-tool-receipt-limit.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
|
||||
export class NativeStatusRaceError extends Error {
|
||||
@@ -824,6 +825,19 @@ async function materializeDecisionEffect(input: {
|
||||
summary: effect.summary,
|
||||
},
|
||||
};
|
||||
const [stopped] = effect.idempotencyKey === toolCallLoopContinuationKey(input.runId)
|
||||
? await input.tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, effect.agentId),
|
||||
eq(agentWakeupRequests.idempotencyKey, effect.idempotencyKey), eq(agentWakeupRequests.status, "cancelled"),
|
||||
)).limit(1)
|
||||
: [];
|
||||
if (stopped)
|
||||
return {
|
||||
effectKind: effect.kind,
|
||||
targetType: "agent_wakeup_request",
|
||||
targetId: stopped.id,
|
||||
payload: { continuationKind: effect.continuationKind, summary: effect.summary, cancelled: true },
|
||||
};
|
||||
}
|
||||
if (effect.continuationKind === "monitor") throw new Error("Monitor delivery belongs to the issue monitor scheduler");
|
||||
const wakeId = await enqueueWake({
|
||||
@@ -895,8 +909,10 @@ async function materializeDecisionEffect(input: {
|
||||
}
|
||||
if (effect.kind === "schedule_retry") {
|
||||
failAt("continuation_materialization", input.failpoint);
|
||||
if (effect.cause === NATIVE_PROVIDER_OVERLOADED_CODE) {
|
||||
const retryId = await materializeNativeProviderCapacityRetry({
|
||||
if (effect.cause === NATIVE_PROVIDER_OVERLOADED_CODE || effect.cause === NATIVE_TOOL_RECEIPT_LIMIT_CODE) {
|
||||
const materialize = effect.cause === NATIVE_PROVIDER_OVERLOADED_CODE
|
||||
? materializeNativeProviderCapacityRetry : materializeNativeToolReceiptLimitContinuation;
|
||||
const retryId = await materialize({
|
||||
tx: input.tx as unknown as Db, companyId: input.companyId,
|
||||
issueId: input.issue.id, runId: input.runId, agentId: effect.agentId,
|
||||
});
|
||||
@@ -1699,8 +1715,8 @@ export async function commitNativeStatusDecision(input: {
|
||||
if (reasonCode === "native_provider_model_rejected" && !input.requireModelRejectionOwner) {
|
||||
throw new Error("native_model_rejection_owner_required");
|
||||
}
|
||||
if ((reasonCode.startsWith("native_provider_overloaded") || reasonCode === "native_run_failed_continuation_exhausted")
|
||||
&& !input.requireProviderFailureOwner) {
|
||||
if ((reasonCode.startsWith("native_provider_overloaded") || reasonCode === "native_run_failed_continuation_exhausted"
|
||||
|| reasonCode === NATIVE_TOOL_RECEIPT_LIMIT_CODE) && !input.requireProviderFailureOwner) {
|
||||
throw new Error("native_provider_failure_owner_required");
|
||||
}
|
||||
const publications: ActivityPublication[] = [];
|
||||
|
||||
Reference in new issue
Block a user