mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(native): deliver the retry after a failed native turn, once per chain (#20681)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime decides a task's next status after each run, through the status arbiter and the status-decision committer > - When a native turn fails without a classified provider error, the arbiter keeps the task's status and asks for one retry (`schedule_retry`) > - The native-status wake dispatcher refused that retry wake, because its provenance check accepted only `enqueue_continuation` effects > - So the task stayed `in_progress` with no live run, no queued wake and no visible cause, until a person noticed it > - This pull request lets the dispatcher accept the committed retry effect, keeps the retry on its chat route, and limits the automatic continuation to one per chain > - The benefit is that a failed turn always ends with either queued work or a blocked task with a working Retry ## Linked Issues or Issue Description **What happened?** A native run that ends with a failed turn and no classified provider error leaves its task stranded. Examples: an OpenCode server that the sandbox killed with SIGKILL during a turn (`opencode_server_exited`), or an OpenCode turn that ended without an answer (`opencode_empty_response`). The run's accepted result is finalized as `run_failed_partial_evidence_preserved`. The committer queues a `schedule_retry` wake. The dispatcher then skips the wake with `chat_control_recovery_proof_unresolved`. The task stays `in_progress` with no run, no wake and no blocker. **Steps to reproduce** 1. Run a native task. Make the provider turn fail with `opencode_server_exited` or `opencode_empty_response` and an accepted `needs_review` result. 2. Finalize the run. 3. Run the native-status wake dispatcher. 4. See that the retry wake and its dispatch receipt are `skipped`, and that no run starts. **Expected behavior** The task gets its automatic continuation. If that continuation fails the same way, the task is blocked with a clear board blocker and a Retry that starts a new run. **Paperclip version or commit** `origin/master` atc444da7d05. **Additional context** The task stays `in_progress` for hours with no owner. On a staging deployment, two tasks stayed in this state for more than 7 and 11 hours. ## What Changed - `server/src/services/chat-control-recovery-stop.ts`: the native-parent proof accepts the `schedule_retry` effect, as well as `enqueue_continuation`, when the effect targets the wake intent. Other effect kinds (for example `notify_owner`) are not changed. The other checks (applied decision, delivered effect, same company, task and agent) do not change. - `server/src/services/native-runtime/status-arbiter.ts`: new input `failedTurnContinuationUsed`. When it is true, a failed run is decided as `native_run_failed_continuation_exhausted`: the task is blocked with a board blocker and no wake. New constants `NATIVE_RUN_FAILED_RETRY_CAUSE` and `NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION`. - `server/src/services/native-runtime/native-failed-turn-continuation.ts` (new): `isFailedTurnContinuationRun` reads the run's dispatcher-set `nativeStatusWakeIntentId` and proves from the intent row (company, agent, task, committer actor, `retryCause`) that the run is the automatic continuation of a failed turn. A copied or unknown id does not count. - `server/src/services/native-runtime/native-run-finalizer.ts`: passes `failedTurnContinuationUsed` to the arbiter for failed runs that the run still owns; the committer rechecks that ownership under the issue lock (`requireProviderFailureOwner`). - `server/src/services/native-runtime/native-continuation-delivery.ts`: the failed-turn retry keeps the failed run as its chat conversation source, so its answer reaches the original chat. - `doc/execution-semantics.md`: documents the retry wake, its provenance and the one-continuation bound. - New test `native-failed-turn-continuation.integration.test.ts`. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-failed-turn-continuation.integration.test.ts` - RED on `origin/master` (c444da7d05): the dispatcher reports `dispatched: 0` and the wake is skipped with `chat_control_recovery_proof_unresolved`. - GREEN with this change: for both `opencode_server_exited` and `opencode_empty_response`, the dispatcher starts one continuation run. When that continuation fails the same way, the task is blocked with the board blocker, no new wake is queued, and the task Retry is accepted and starts a new run. A copied wake id does not exhaust the continuation. - Related suites pass: `status-arbiter.test.ts`, `native-provider-failure.integration.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `chat-control-admission-retry.test.ts`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` selects broad verification (exit 2) because `doc/execution-semantics.md` changed. Required CI supplies that coverage. ## Risks - Behavior change: a failed native turn now gets the automatic continuation that the arbiter already requested. That continuation is a new run of the same task, with the normal admission gates. It does not replay the failed turn. - The continuation is bounded to one per chain, so a deterministic failure cannot make a wake loop. - Tasks that are already stranded by the old behavior keep their skipped wake. Their task Retry still works. - Upgrade safety: an old pending `schedule_retry` intent is accepted by the new dispatcher. A run that started before this change has no proven intent, so it gets one continuation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, read-only API access). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
52160eba35
commit
06af074452
8 files changed
+251
-8
No files matched your search
@@ -1579,6 +1579,8 @@ An explicit recovery action is a typed liveness repair path for a source issue.
|
||||
|
||||
A terminal native failure can retain a result accepted before checkpoint or cleanup failed. That result is historical evidence, not a live controller. New user input may start a fresh turn after the controller and execution environment have stopped and ordinary admission checks pass. Preserve the failed run, its result, and its recovery budget. Do not commit the old result, infer action outcomes, or replay the failed turn. A message saved while cleanup is pending must be reconsidered after verified cleanup and delivered once. External chat messages retain their original inbound action, wake receipt and actor. Cleanup delegates reconsideration to the live inbound owner, which revalidates the endpoint, conversation generation, principal and exact saved source before normal admission. The existing stranded-queue sweep uses the same owner after a lost callback or restart; it cannot substitute a generic assignment wake. A rejected source must not prevent other saved messages from being reconsidered. Admission must consume its deferred receipt in the same transaction that creates the new run, including in agent chat, where later messages keep their separate turns. Completing that run must not promote the consumed receipt again. Workspace-export repair retains its separate saved-result recovery path.
|
||||
|
||||
When a native run's accepted result reports a failed turn without a classified provider error (`run_failed_partial_evidence_preserved`, for example an OpenCode server that was killed or ended without an answer), the status decision keeps the task's status and queues one same-agent `schedule_retry` wake. The wake dispatcher accepts that committed decision effect as the continuation's provenance, exactly like an `enqueue_continuation` effect, so the task always has queued work, and the retry answers the same external chat conversation. The automatic continuation is used once per chain: if the run started from that wake fails the same way, the decision is `native_run_failed_continuation_exhausted`, the task is blocked with a board blocker, and the visible Retry starts the next attempt.
|
||||
|
||||
A host-verified workspace copy-back receipt also permits new user input after
|
||||
a native coordinator reaches terminal failure with a retained result. The receipt
|
||||
must bind the company, task, run, allocation, descriptor and final file digest.
|
||||
|
||||
@@ -92,6 +92,9 @@ export type ChatControlRecoveryStop =
|
||||
sourceRunId: string;
|
||||
};
|
||||
|
||||
/** Status-decision effects whose target is the native-status wake intent. */
|
||||
const NATIVE_STATUS_WAKE_EFFECT_KINDS = ["enqueue_continuation", "schedule_retry"];
|
||||
|
||||
/** A dispatched native continuation inherits only its committed decision's
|
||||
* source run. Arbitrary payload.nativeDecisionId / context.source is not proof. */
|
||||
export async function readChatControlNativeParent(
|
||||
@@ -137,7 +140,10 @@ export async function readChatControlNativeParent(
|
||||
eq(statusDecisionEffects.companyId, scope.companyId),
|
||||
eq(statusDecisionEffects.issueId, scope.issueId),
|
||||
eq(statusDecisionEffects.decisionId, statusDecisions.id),
|
||||
eq(statusDecisionEffects.effectKind, "enqueue_continuation"),
|
||||
// A committed continuation or failed-turn retry effect that targets
|
||||
// this intent is its proof. Refusing the retry stranded the task
|
||||
// with no queued work.
|
||||
inArray(statusDecisionEffects.effectKind, NATIVE_STATUS_WAKE_EFFECT_KINDS),
|
||||
eq(statusDecisionEffects.targetType, "agent_wakeup_request"),
|
||||
eq(statusDecisionEffects.targetId, intent.id),
|
||||
eq(statusDecisionEffects.deliveryState, "delivered"),
|
||||
|
||||
@@ -2,6 +2,7 @@ import { agentWakeupRequests, heartbeatRuns, statusDecisions, statusDecisionEffe
|
||||
import { and, eq, inArray, isNull, ne, or, sql } from "drizzle-orm";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
import { boundRunIssueIdSql } from "../run-task-binding.js";
|
||||
import { NATIVE_RUN_FAILED_RETRY_CAUSE } from "./status-arbiter.js";
|
||||
|
||||
/** Recover conversation provenance from a committed continuation, never from
|
||||
* provider-supplied source IDs or the mere presence of a chat on the task. */
|
||||
@@ -34,18 +35,29 @@ export async function readNativeContinuationSource(
|
||||
))
|
||||
.innerJoin(statusDecisionEffects, and(
|
||||
eq(statusDecisionEffects.companyId, binding.companyId), eq(statusDecisionEffects.issueId, binding.issueId),
|
||||
eq(statusDecisionEffects.effectKind, "enqueue_continuation"),
|
||||
eq(statusDecisionEffects.targetType, "agent_wakeup_request"),
|
||||
sql`${statusDecisionEffects.targetId} = ${agentWakeupRequests.id}::text`,
|
||||
eq(statusDecisionEffects.deliveryState, "delivered"),
|
||||
sql`${statusDecisionEffects.payload}->>'continuationKind' = 'same_agent'`,
|
||||
sql`${agentWakeupRequests.payload}->>'continuationKind' = 'same_agent'`,
|
||||
or(and(
|
||||
eq(statusDecisionEffects.effectKind, "enqueue_continuation"),
|
||||
sql`${statusDecisionEffects.payload}->>'continuationKind' = 'same_agent'`,
|
||||
sql`${agentWakeupRequests.payload}->>'continuationKind' = 'same_agent'`,
|
||||
), and(
|
||||
// The one automatic retry after a failed turn answers the same chat.
|
||||
eq(statusDecisionEffects.effectKind, "schedule_retry"),
|
||||
sql`${agentWakeupRequests.payload}->>'retryCause' = ${NATIVE_RUN_FAILED_RETRY_CAUSE}`,
|
||||
)),
|
||||
))
|
||||
.innerJoin(statusDecisions, and(
|
||||
eq(statusDecisions.id, statusDecisionEffects.decisionId), eq(statusDecisions.companyId, binding.companyId),
|
||||
eq(statusDecisions.issueId, binding.issueId), eq(statusDecisions.applicationState, "applied"),
|
||||
eq(statusDecisions.reasonCode, "live_continuation_registered"),
|
||||
sql`${agentWakeupRequests.idempotencyKey} = 'native-status:' || ${statusDecisions.id}::text || ':continuation'`,
|
||||
or(and(
|
||||
eq(statusDecisions.reasonCode, "live_continuation_registered"),
|
||||
sql`${agentWakeupRequests.idempotencyKey} = 'native-status:' || ${statusDecisions.id}::text || ':continuation'`,
|
||||
), and(
|
||||
eq(statusDecisions.reasonCode, "run_failed_partial_evidence_preserved"),
|
||||
sql`${agentWakeupRequests.idempotencyKey} = 'native-status:' || ${statusDecisions.id}::text || ':retry'`,
|
||||
)),
|
||||
sql`${agentWakeupRequests.payload}->>'nativeDecisionId' = ${statusDecisions.id}::text`,
|
||||
))
|
||||
.innerJoin(source, and(
|
||||
|
||||
+158
@@ -0,0 +1,158 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, agentWakeupRequests, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations, statusDecisions } from "@paperclipai/db";
|
||||
import type { ControlPlanePort, NativeExecutionInputV1, PrpEvent, PrpTerminalState } from "@paperclipai/paperclip-runner";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
|
||||
executeNativeSession: provider.execute,
|
||||
}));
|
||||
|
||||
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { finalizeNativeRun } from "./native-run-finalizer.js";
|
||||
import { readNativeContinuationSource } from "./native-continuation-delivery.js";
|
||||
import { NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION } from "./status-arbiter.js";
|
||||
import { heartbeatService } from "../heartbeat.js";
|
||||
import { requestTaskRetry } from "../task-retry.js";
|
||||
|
||||
type FailureCode = "opencode_server_exited" | "opencode_empty_response";
|
||||
|
||||
// Seen on a staging deployment: an OpenCode turn
|
||||
// failed (its server was killed, or it ended without an answer). The accepted
|
||||
// result kept the task in_progress and the arbiter scheduled a retry, but the
|
||||
// dispatcher refused that wake (`chat_control_recovery_proof_unresolved`), so
|
||||
// the task sat for hours with no owner and no queued work.
|
||||
describe("native failed-turn continuation", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-failed-turn-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Native failed turn", issuePrefix: "NFT" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "OpenCode agent", status: "active", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "openrouter/test-model" } });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
async function failTurn(issueId: string, code: FailureCode, context: Record<string, unknown> = {}, dispatchedFrom?: string) {
|
||||
const runId = randomUUID();
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
// A dispatched continuation carries the dispatcher's own receipt for its intent.
|
||||
const [receipt] = dispatchedFrom ? await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation",
|
||||
triggerDetail: "system", reason: "issue_status_changed", status: "claimed", requestedByActorType: "system",
|
||||
requestedByActorId: `native-status-wake-dispatch:${dispatchedFrom}`, payload: { issueId } }).returning() : [];
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "automation", triggerDetail: "system", wakeupRequestId: receipt?.id ?? null, contextSnapshot: { issueId, ...context } }).returning();
|
||||
if (receipt) await db.update(agentWakeupRequests).set({ runId }).where(eq(agentWakeupRequests.id, receipt.id));
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
const completionInput = buildNativeCompletionContract(issue, { revision: contract.revision });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1",
|
||||
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "opencode", model: "openrouter/test-model" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Review the PR.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "opencode_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: completionInput },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
||||
const event: PrpEvent = {
|
||||
schema: "paperclip.prp.event.v1", runId, normalizedSessionId: native.normalizedSessionId,
|
||||
turnId: native.turnId, sourceInstanceId: native.runnerInstanceId, sourceEventId: "turn-terminal", sourceSeq: 1,
|
||||
sourceKind: "runner", eventType: "turn.failed", schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(),
|
||||
payload: code === "opencode_server_exited"
|
||||
? { status: "failed", error: { code, message: "OpenCode stopped while the turn was running.", opencodeFailure: { kind: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" } } }
|
||||
: { status: "failed", error: { code, message: "OpenCode ended without a final answer.", retryable: false } },
|
||||
};
|
||||
const terminal: PrpTerminalState = { schema: "paperclip.prp.terminal.v1", turnTerminalState: "failed", runTerminalState: "failed", reportedWorkDisposition: "needs_review" };
|
||||
const result = { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "needs_review" as const,
|
||||
completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: completionInput.revision } };
|
||||
const openInput = { identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId }, backendKind: "mock" as const, sourceInstanceId: native.runnerInstanceId };
|
||||
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun(openInput);
|
||||
await options.controlPlane.appendEvent(event);
|
||||
await options.controlPlane.completeRun({ result, terminal, turnId: native.turnId });
|
||||
return { result, terminal, turnId: native.turnId, normalizedSessionId: native.normalizedSessionId, providerSessionId: "fixture-provider-session", driverKind: execution.session.driverKind, driverVersion: "fixture", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
|
||||
descriptor: async () => { throw new Error("Unexpected provider call"); },
|
||||
openSession: async () => { throw new Error("Unexpected provider call"); },
|
||||
} });
|
||||
await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
const [failedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(failedRun.status).toBe("failed");
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
return { runId, decision };
|
||||
}
|
||||
|
||||
const issueWakes = async (issueId: string) => (await db.select().from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.companyId, companyId))).filter(wake => wake.payload?.issueId === issueId);
|
||||
|
||||
it.each(["opencode_server_exited", "opencode_empty_response"] as const)("continues the task once after %s, then hands it to Retry", async (code) => {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Review a PR", status: "in_progress", workMode: "standard", assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
|
||||
const first = await failTurn(issueId, code);
|
||||
expect(first.decision.reasonCode).toBe("run_failed_partial_evidence_preserved");
|
||||
|
||||
const heartbeat = heartbeatService(db);
|
||||
expect(await heartbeat.dispatchPendingNativeStatusWakeups({ companyId })).toMatchObject({ dispatched: 1 });
|
||||
const wakes = await issueWakes(issueId);
|
||||
expect(wakes.filter(wake => wake.status === "skipped")).toEqual([]);
|
||||
const intent = wakes.find(wake => wake.requestedByActorId === "native-status-committer")!;
|
||||
expect(intent.payload).toMatchObject({ retryCause: "native_run_failed" });
|
||||
const [continuation] = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
|
||||
.filter(r => r.id !== first.runId && r.contextSnapshot?.issueId === issueId);
|
||||
expect(continuation?.contextSnapshot?.nativeStatusWakeIntentId).toBe(intent.id);
|
||||
expect(["queued", "running"]).toContain(continuation.status);
|
||||
// The continuation keeps the failed run as its conversation source.
|
||||
expect(await readNativeContinuationSource(db, { companyId, issueId, runId: continuation.id })).toBe(first.runId);
|
||||
// The fixture cannot launch a real runner; let the dispatched run settle.
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
const [task] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
expect(task.assigneeAgentId).toBe(agentId);
|
||||
|
||||
// The continuation's own turn fails the same way: no second automatic
|
||||
// wake, and a board blocker whose Retry restarts the work.
|
||||
await db.update(issues).set({ status: "in_progress", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, issueId));
|
||||
const before = (await issueWakes(issueId)).length;
|
||||
const second = await failTurn(issueId, code, { nativeStatusWakeIntentId: intent.id }, intent.id);
|
||||
expect(second.decision).toMatchObject({ reasonCode: "native_run_failed_continuation_exhausted", toStatus: "blocked" });
|
||||
await heartbeat.dispatchPendingNativeStatusWakeups({ companyId });
|
||||
expect((await issueWakes(issueId)).length).toBe(before + 1); // only the fixture dispatch receipt
|
||||
const [blocked] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
expect(blocked).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION } });
|
||||
// The visible Retry is accepted and starts a new run of the task.
|
||||
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session",
|
||||
idempotencyKey: randomUUID(), observedRunId: second.runId });
|
||||
expect(receipt).toMatchObject({ accepted: true, stale: false });
|
||||
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
|
||||
const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
|
||||
.filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId);
|
||||
expect(retried).toHaveLength(1);
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}, 120_000);
|
||||
|
||||
it("does not treat a copied retry intent id as a used continuation", async () => {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Review another PR", status: "in_progress", workMode: "standard", assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
|
||||
const first = await failTurn(issueId, "opencode_empty_response");
|
||||
expect(first.decision.reasonCode).toBe("run_failed_partial_evidence_preserved");
|
||||
const intent = (await issueWakes(issueId)).find(wake => wake.requestedByActorId === "native-status-committer")!;
|
||||
// A real intent id copied into a run that the dispatcher did not start.
|
||||
await db.update(issues).set({ status: "in_progress", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, issueId));
|
||||
const copied = await failTurn(issueId, "opencode_empty_response", { nativeStatusWakeIntentId: intent.id });
|
||||
expect(copied.decision.reasonCode).toBe("run_failed_partial_evidence_preserved");
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,36 @@
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, type Db, type heartbeatRuns } from "@paperclipai/db";
|
||||
import { NATIVE_RUN_FAILED_RETRY_CAUSE } from "./status-arbiter.js";
|
||||
|
||||
/**
|
||||
* True when this run was dispatched from the committer's `schedule_retry` wake
|
||||
* for an earlier failed turn. The intent id is set by the native-status wake
|
||||
* dispatcher, and the intent row itself proves the retry cause, so provider
|
||||
* output or a copied context cannot claim or hide the used continuation.
|
||||
*/
|
||||
export async function isFailedTurnContinuationRun(
|
||||
db: Pick<Db, "select">,
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "agentId" | "contextSnapshot" | "wakeupRequestId">,
|
||||
issueId: string,
|
||||
): Promise<boolean> {
|
||||
const intentId = run.contextSnapshot?.nativeStatusWakeIntentId;
|
||||
if (typeof intentId !== "string" || !/^[0-9a-f-]{36}$/i.test(intentId) || !run.wakeupRequestId) return false;
|
||||
// The run's own dispatch receipt must be the one the dispatcher created for
|
||||
// this intent, so a copied intent id on another run proves nothing.
|
||||
const [dispatch] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, run.companyId),
|
||||
eq(agentWakeupRequests.id, run.wakeupRequestId),
|
||||
eq(agentWakeupRequests.runId, run.id),
|
||||
eq(agentWakeupRequests.requestedByActorId, `native-status-wake-dispatch:${intentId}`),
|
||||
)).limit(1);
|
||||
if (!dispatch) return false;
|
||||
const [intent] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, run.companyId),
|
||||
eq(agentWakeupRequests.agentId, run.agentId),
|
||||
eq(agentWakeupRequests.id, intentId),
|
||||
eq(agentWakeupRequests.requestedByActorId, "native-status-committer"),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
||||
sql`${agentWakeupRequests.payload}->>'retryCause' = ${NATIVE_RUN_FAILED_RETRY_CAUSE}`,
|
||||
)).limit(1);
|
||||
return Boolean(intent);
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import { isNativePlanWaitResult, readNativePlanWait } from "./native-plan-wait.j
|
||||
import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js";
|
||||
import { settleExternalConversation } from "../slack-conversation-lifecycle.js";
|
||||
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
|
||||
import { isFailedTurnContinuationRun } from "./native-failed-turn-continuation.js";
|
||||
import { readPersistedNativeFailureEvidence } from "./native-provider-failure-evidence.js";
|
||||
import { explainFailedRun, NATIVE_RUN_FAILED_UNEXPLAINED } from "./native-run-failure-explanation.js";
|
||||
import { dismissAutomaticCompletionReviews } from "./automatic-completion-reviews.js";
|
||||
@@ -1655,6 +1656,10 @@ export async function finalizeNativeRun(input: {
|
||||
providerOverloaded: providerFailure?.errorCode === "native_provider_overloaded" && ownsProviderFailureDecision,
|
||||
providerFailureSuperseded: providerFailure?.errorCode === "native_provider_overloaded" && !ownsProviderFailureDecision,
|
||||
failureRetryCount: executionFailureRetryCount(run),
|
||||
// Only the task's current owner may turn its own failed continuation
|
||||
// into a blocker; the committer rechecks that under the issue lock.
|
||||
failedTurnContinuationUsed: terminalState === "failed" && ownsProviderFailureDecision && !reviewContext
|
||||
&& await isFailedTurnContinuationRun(input.db, run, authoritativeIssue.id),
|
||||
...(reviewContext ? { nativeReviewOutcome: nativeReview
|
||||
? nativeReview.interaction.status === "pending" ? "pending" as const : "resolved" as const
|
||||
: "stale" as const } : {}),
|
||||
@@ -1799,6 +1804,7 @@ export async function finalizeNativeRun(input: {
|
||||
? { agentId: run.agentId, reviewContext }
|
||||
: undefined,
|
||||
requireProviderFailureOwner: decision.reasonCode?.startsWith("native_provider_overloaded")
|
||||
|| decision.reasonCode === "native_run_failed_continuation_exhausted"
|
||||
? { agentId: run.agentId, reviewContext } : undefined,
|
||||
requireBoardResponseWaitSource:
|
||||
decision.reasonCode === "board_response_waiting" || repairBoardResponseWait
|
||||
|
||||
@@ -3,6 +3,11 @@ import { NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_CAPACITY_MAX_RETRIES, N
|
||||
|
||||
export const NATIVE_STATUS_ARBITER_POLICY_VERSION = "phase6-v13";
|
||||
|
||||
/** The automatic continuation after a failed turn is used once per chain. */
|
||||
export const NATIVE_RUN_FAILED_RETRY_CAUSE = "native_run_failed";
|
||||
export const NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION =
|
||||
"The agent's turn failed again after Paperclip continued it automatically. The saved work is kept. Check the failed run, then Retry the task.";
|
||||
|
||||
export type NativeAuthoritativeIssueStatus =
|
||||
| "backlog"
|
||||
| "todo"
|
||||
@@ -179,6 +184,11 @@ export function arbitrateNativeStatus(input: {
|
||||
providerOverloaded?: boolean;
|
||||
providerFailureSuperseded?: boolean;
|
||||
failureRetryCount?: number;
|
||||
/**
|
||||
* This run was itself the automatic continuation of an earlier failed turn.
|
||||
* Another failure needs a person's Retry, never another automatic wake.
|
||||
*/
|
||||
failedTurnContinuationUsed?: boolean;
|
||||
agentId: string;
|
||||
priorIssueStatus: NativeAuthoritativeIssueStatus;
|
||||
}): NativeStatusDecision {
|
||||
@@ -312,6 +322,18 @@ export function arbitrateNativeStatus(input: {
|
||||
effects: [{ kind: "release_run_resources" }],
|
||||
};
|
||||
}
|
||||
if (input.failedTurnContinuationUsed) {
|
||||
// The automatic continuation already ran and failed too. Keep the work
|
||||
// and hand the next attempt to a person instead of waking again.
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "blocked",
|
||||
toStatus: "blocked",
|
||||
reasonCode: "native_run_failed_continuation_exhausted",
|
||||
unblockDescriptor: { owner: "board", action: NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION },
|
||||
effects: [{ kind: "bind_blocker", owner: "board", action: NATIVE_RUN_FAILED_CONTINUATION_EXHAUSTED_ACTION }],
|
||||
};
|
||||
}
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "preserve",
|
||||
@@ -321,7 +343,7 @@ export function arbitrateNativeStatus(input: {
|
||||
effects: [
|
||||
{
|
||||
kind: "schedule_retry",
|
||||
cause: "native_run_failed",
|
||||
cause: NATIVE_RUN_FAILED_RETRY_CAUSE,
|
||||
summary:
|
||||
"Resume the persisted native run without opening a second provider session.",
|
||||
agentId: input.agentId,
|
||||
|
||||
@@ -1696,7 +1696,8 @@ export async function commitNativeStatusDecision(input: {
|
||||
if (reasonCode === "native_provider_model_rejected" && !input.requireModelRejectionOwner) {
|
||||
throw new Error("native_model_rejection_owner_required");
|
||||
}
|
||||
if (reasonCode.startsWith("native_provider_overloaded") && !input.requireProviderFailureOwner) {
|
||||
if ((reasonCode.startsWith("native_provider_overloaded") || reasonCode === "native_run_failed_continuation_exhausted")
|
||||
&& !input.requireProviderFailureOwner) {
|
||||
throw new Error("native_provider_failure_owner_required");
|
||||
}
|
||||
const publications: ActivityPublication[] = [];
|
||||
|
||||
Reference in new issue
Block a user