mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 01:50:13 +01:00
fix(native): keep accepted results when runner close cannot prove suspension (#19525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs persist an accepted result first. The Runner then closes and suspends its session. > - Under heavy host load, the Runner close can fail to prove a durable suspension after the result is accepted. > - The executor then kept its finalization lease. The heartbeat finalizer got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. > - The provider work was complete. Only Runner bookkeeping failed, so the run must not fail. > - This pull request hands the exact lease to finalization and completes the run from its accepted result. > - The benefit is that an accepted result stays accepted, independent of host load. ## Linked Issues or Issue Description **What happened?** A native ACPX Claude run asked a structured question. The control plane accepted its result (`Waiting for…`) and recorded `run.terminal` with `runTerminalState=succeeded`. The Runner close then took about 8.6 s. Its `runner.suspend` command ended `failed`, but the durable Runner state was `suspended` with a matching identity. The close threw `NativeSessionCloseUnrecoverableError`. The executor emitted `native.post_result_failure` (`category: runner_settlement_incomplete`, `resultAlreadyAccepted: true`) and rethrew `native_session_lease_lost`. Its finalization lease stayed held (its TTL is 20 minutes). `finalizeNativeRun` got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. No restart was in progress. Host load average was about 40. **Expected behavior** After the control plane accepts a result, finalization bookkeeping must not fail the run. The run settles from its accepted result. Repair of the Runner checkpoint stays with the existing next-run admission rules. **Steps to reproduce** Run `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts -t "accepted local result"` in `server/`. The test uses the real executor, the real control-plane result commit, and embedded PostgreSQL. The provider boundary accepts a result and then throws `NativeSessionCloseUnrecoverableError`. On the base revision, the finalization row keeps the executor lease (`phase: workspace_finalizing`) and the finalizer reports `lease_busy`. **Paperclip version or commit** Base `ff1d1dea5d`. The incident was on `819420a60a`. ## What Changed - `native-session-executor.ts`: after the rejected failure write proves an accepted result under the exact, unexpired lease, the executor releases that lease to finalization. It uses the same exact-authority predicates as the normal release. Before, the lease stayed held until its TTL expired. - `runtime-execution.ts`: for `runner_settlement_incomplete` on a local workspace (no workspace sync and no remote workspace authority), Paperclip records `workspace_finalize` as succeeded, finalizes the accepted result, and returns it to the ordinary completion path. - New `native-accepted-result-settlement.ts`: rebuilds the adapter result from the durable accepted result row, its terminal, the run's native session, and the accepted usage checkpoint. The result carries the closed `nativePostResultFailure` diagnostic. It contains no provider text. - Other post-result categories (for example `io_not_found`) keep their workspace-failure handling. They now also reach finalization at once, instead of after the lease TTL. - `doc/execution-semantics.md`: describes the settlement. ## Verification - Red, then green: `settles an accepted local result when runner close cannot prove suspension` and `keeps workspace failure handling for other accepted-result failures on a local workspace` in `server/src/services/heartbeat/runtime-execution.test.ts`. Both failed on the base revision with the executor lease still held. Both pass with this change. - Safety case `settles an accepted local result ... (workspace-check-failed)`: when the `workspace_finalize` check fails, the run keeps the existing failure path. Case `terminal-unreadable`: when the accepted result cannot be rebuilt, only a failed `workspace_finalize` record is written. - `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts`: all pass (some cases time out at 30 s only under host load above 40; they pass on a narrow rerun). This includes the 14 existing post-result diagnostic and authority cases (expired, replaced owner, generation, attempt, boot, pid, process start, stopped, lock busy, and others). - `pnpm --filter @paperclipai/server typecheck`: pass. - Live cell `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` (real ACPX Claude, local runner): 3 runs at host load average about 40. Runs 1 and 2 passed. Run 3 hit the fault naturally: the first run emitted `native.post_result_failure` (`runner_settlement_incomplete`) and now ended `run succeeded` (before this change it ended `failed/adapter_failed`). The cell then failed later, in the resume run after the answer: `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=target_not_sandbox_runner`. That is a separate follow-up (see Risks). ## Risks - A changed lease handoff. The executor releases the lease only after the rejected failure write proves an accepted result under the exact owner, attempt, controller generation, controller identity, and an unexpired lease. A stale or replaced owner does not release anything. A failed release is bounded and best effort. The old behavior then applies. - The run now completes as `succeeded` while the local Runner checkpoint is not proven suspended. The next run on the same session then meets the existing fence: the continuity-break retirement (`retireFinishedPerTurnRunnerdOwner`) accepts only sandbox Runners, so a local Runner is refused with `target_not_sandbox_runner` and its state root is quarantined. This PR does not change that fence. Extending the retirement to local Runners needs its own proof that the local Runner process stopped, and is a follow-up. - Settlement happens only when the accepted result and its terminal are rebuilt first, and the `workspace_finalize` record then succeeds. A failed branch or plugin workspace check keeps the failure path. - Synced and remote workspaces are not changed. They keep their copy-back retry path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19126 added the closed post-result diagnostic that this change uses. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
1206373a95
commit
07ddf9a5c2
5 files changed
+291
-8
No files matched your search
@@ -2328,6 +2328,17 @@ predecessor/successor receipts for every earlier owner. Unrelated, live, or
|
||||
incompletely linked owners remain ambiguous. The run then fails closed
|
||||
with `runner_harness_state_mismatch: backup_without_reusable_lease`.
|
||||
|
||||
A Runner close can also fail to prove suspension after the control plane
|
||||
accepted the run's result (`native.post_result_failure` with category
|
||||
`runner_settlement_incomplete`). The provider work is complete, so this is not
|
||||
a run failure. The executor first proves that it still holds the exact,
|
||||
unexpired finalization lease and that the result is accepted. It then releases
|
||||
that lease to finalization. A run on a local workspace has nothing to copy
|
||||
back. Paperclip finalizes its accepted result and completes the run from that
|
||||
result through the ordinary completion path. Load or a slow suspend command
|
||||
cannot turn an accepted result into `adapter_failed`. A run with a synced or
|
||||
remote workspace keeps its copy-back retry path.
|
||||
|
||||
A per-turn sandbox turn can finish through recovery without a Runner
|
||||
suspension receipt or failover backup on the controller. For example, its
|
||||
post-turn checkpoint copy can fail on a full sandbox disk. The controller
|
||||
|
||||
@@ -2,7 +2,7 @@ import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-c
|
||||
|
||||
import { createNativeUsageDelivery } from "../native-runtime/native-usage-delivery.js";
|
||||
import { flushBudgetEvaluationWork } from "../budgets.js";
|
||||
import type { PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
||||
import { NativeSessionCloseUnrecoverableError, type PrpEvent } from "../../vendor/paperclip-runner/index.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm, readFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
@@ -288,6 +288,118 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["settled", "workspace-check-failed", "terminal-unreadable"] as const)("settles an accepted local result when runner close cannot prove suspension (%s)", async scenario => {
|
||||
const { input } = await fixture(true);
|
||||
const issue = input.task.issueRef!;
|
||||
const [run] = await db.update(heartbeatRuns).set({ nativeIssueId: issue.id, contextSnapshot: { issueId: issue.id }, costAccountingPending: true }).where(eq(heartbeatRuns.id, input.run.id)).returning();
|
||||
input.run = run;
|
||||
const prepared = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issue.id));
|
||||
const completion = buildNativeCompletionContract(issue, { revision: contract.revision });
|
||||
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: issue.id, phase: "observed" });
|
||||
input.runtime.nativeExecution = {
|
||||
schema: "paperclip.native-execution-input.v1", interactionResponses: [], credentialBindings: [], binding: { companyId: run.companyId, agentId: run.agentId, issueId: issue.id, runId: run.id, executionWorkspaceId: run.id },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: completion },
|
||||
provider: { kind: "codex", model: "test-model" },
|
||||
task: { identifier: issue.id, title: issue.title, description: null, prompt: "Local runner settlement fault", workMode: "standard" },
|
||||
workspace: { cwd: home, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: prepared.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
} satisfies NativeExecutionInputV1;
|
||||
input.runtime.nativeRunnerInstanceId = prepared.runnerInstanceId;
|
||||
postResultProvider.execute.mockReset().mockImplementationOnce(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun({ identity: { companyId: run.companyId, issueId: issue.id, runId: run.id, agentId: run.agentId, sessionId: prepared.normalizedSessionId }, backendKind: "mock", sourceInstanceId: prepared.runnerInstanceId });
|
||||
await options.controlPlane.completeRun({ result: { ...CONTROL_PLANE_CONFORMANCE_RESULT, completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: completion.revision } },
|
||||
terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }, turnId: prepared.turnId });
|
||||
if (scenario === "terminal-unreadable") {
|
||||
await db.update(nativeRunResults).set({ resultJson: sql`${nativeRunResults.resultJson} - 'terminal'` }).where(eq(nativeRunResults.runId, run.id));
|
||||
}
|
||||
// The runner retired after the accepted result, but its suspend command
|
||||
// did not complete before the close deadline (for example, under load).
|
||||
throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: true }, true);
|
||||
});
|
||||
const actual = await vi.importActual<typeof import("../native-runtime/native-session-executor.js")>("../native-runtime/native-session-executor.js");
|
||||
vi.mocked(executePaperclipNativeSession).mockImplementationOnce(async options => actual.executePaperclipNativeSession({ ...options, useRunnerd: false,
|
||||
backend: { descriptor: async () => { throw new Error("Unexpected provider call"); }, openSession: async () => { throw new Error("Unexpected provider call"); } } }));
|
||||
|
||||
if (scenario === "terminal-unreadable") {
|
||||
// Without a rebuildable accepted result, no succeeded finalize row is written.
|
||||
await expect(executeHeartbeatRuntime(db, input)).rejects.toMatchObject({ message: "native_session_lease_lost" });
|
||||
const operations = await finalizations();
|
||||
expect(operations.map(operation => operation.status)).toEqual(["failed"]);
|
||||
return;
|
||||
}
|
||||
if (scenario === "workspace-check-failed") {
|
||||
// A failed workspace finalization check keeps the existing failure path.
|
||||
const recorder = input.workspace.workspaceOperationRecorder;
|
||||
const record = recorder.recordOperation.bind(recorder);
|
||||
vi.spyOn(recorder, "recordOperation").mockImplementation(async operation => {
|
||||
if (operation.phase === "workspace_finalize") throw new Error("workspace check failed");
|
||||
return record(operation);
|
||||
});
|
||||
await expect(executeHeartbeatRuntime(db, input)).rejects.toMatchObject({ message: "native_session_lease_lost", diagnostic: { category: "runner_settlement_incomplete" } });
|
||||
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
expect(after).toMatchObject({ phase: "retryable_failure", leaseOwner: null, failureCode: "workspace_finalization_failed" });
|
||||
return;
|
||||
}
|
||||
// Runner bookkeeping after an accepted result is not an adapter failure:
|
||||
// the result returns to ordinary completion like any successful turn.
|
||||
const execution = await executeHeartbeatRuntime(db, input);
|
||||
if (!execution.dispatched) throw new Error("expected a dispatched native execution");
|
||||
expect(execution.adapterResult).toMatchObject({ exitCode: 0, errorMessage: null, timedOut: false,
|
||||
resultJson: { nativePostResultFailure: { stage: "post_result_execution", category: "runner_settlement_incomplete", resultAlreadyAccepted: true } },
|
||||
nativeFinalization: { runId: run.id, terminal: { runTerminalState: "succeeded" } } });
|
||||
|
||||
// The executor handed over its lease, so the accepted result was finalized
|
||||
// now rather than after the lease TTL.
|
||||
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
expect(after).toMatchObject({ phase: "committed", leaseOwner: null, leaseExpiresAt: null });
|
||||
const saved = await read(run.id);
|
||||
expect(saved.status).toBe("running");
|
||||
expect(saved.errorCode).toBeNull();
|
||||
const [operation] = await finalizations();
|
||||
expect(operation.status).toBe("succeeded");
|
||||
expect(postResultProvider.execute).toHaveBeenCalledOnce();
|
||||
expect(input.services.enqueueWakeup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps workspace failure handling for other accepted-result failures on a local workspace", async () => {
|
||||
const { input } = await fixture(true);
|
||||
const issue = input.task.issueRef!;
|
||||
const [run] = await db.update(heartbeatRuns).set({ nativeIssueId: issue.id, contextSnapshot: { issueId: issue.id }, costAccountingPending: true }).where(eq(heartbeatRuns.id, input.run.id)).returning();
|
||||
input.run = run;
|
||||
const prepared = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issue.id));
|
||||
const completion = buildNativeCompletionContract(issue, { revision: contract.revision });
|
||||
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: issue.id, phase: "observed" });
|
||||
input.runtime.nativeExecution = {
|
||||
schema: "paperclip.native-execution-input.v1", interactionResponses: [], credentialBindings: [], binding: { companyId: run.companyId, agentId: run.agentId, issueId: issue.id, runId: run.id, executionWorkspaceId: run.id },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: completion },
|
||||
provider: { kind: "codex", model: "test-model" },
|
||||
task: { identifier: issue.id, title: issue.title, description: null, prompt: "Local post-result I/O fault", workMode: "standard" },
|
||||
workspace: { cwd: home, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: prepared.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
} satisfies NativeExecutionInputV1;
|
||||
input.runtime.nativeRunnerInstanceId = prepared.runnerInstanceId;
|
||||
postResultProvider.execute.mockReset().mockImplementationOnce(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun({ identity: { companyId: run.companyId, issueId: issue.id, runId: run.id, agentId: run.agentId, sessionId: prepared.normalizedSessionId }, backendKind: "mock", sourceInstanceId: prepared.runnerInstanceId });
|
||||
await options.controlPlane.completeRun({ result: { ...CONTROL_PLANE_CONFORMANCE_RESULT, completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: completion.revision } },
|
||||
terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done" }, turnId: prepared.turnId });
|
||||
await readFile(join(home, `missing-${run.id}`, "harmless-checkpoint"));
|
||||
throw new Error("fixture must fail with ENOENT");
|
||||
});
|
||||
const actual = await vi.importActual<typeof import("../native-runtime/native-session-executor.js")>("../native-runtime/native-session-executor.js");
|
||||
vi.mocked(executePaperclipNativeSession).mockImplementationOnce(async options => actual.executePaperclipNativeSession({ ...options, useRunnerd: false,
|
||||
backend: { descriptor: async () => { throw new Error("Unexpected provider call"); }, openSession: async () => { throw new Error("Unexpected provider call"); } } }));
|
||||
|
||||
await expect(executeHeartbeatRuntime(db, input)).rejects.toMatchObject({ message: "native_session_lease_lost", diagnostic: { category: "io_not_found" } });
|
||||
|
||||
// The executor hands its exact lease to finalization, which records the
|
||||
// workspace failure for repair instead of waiting out the lease TTL.
|
||||
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
expect(after).toMatchObject({ phase: "retryable_failure", leaseOwner: null, failureCode: "workspace_finalization_failed" });
|
||||
expect((await read(run.id)).status).toBe("running");
|
||||
});
|
||||
|
||||
const read = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!);
|
||||
const finalizations = () => db.select().from(workspaceOperations).where(eq(workspaceOperations.phase, "workspace_finalize"));
|
||||
function expectCleanup(input: HeartbeatRuntimeExecutionInput) {
|
||||
|
||||
@@ -2,6 +2,7 @@ import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancella
|
||||
import { readContextCompactionRequest } from "../context-compaction.js";
|
||||
import { remoteWorkspaceFailureCause } from "../remote-workspace-transport.js";
|
||||
import { readNativePostResultFailureDiagnostic } from "../native-runtime/native-post-result-failure.js";
|
||||
import { acceptedNativeAdapterResult } from "../native-runtime/native-accepted-result-settlement.js";
|
||||
import { NativeStartupCleanupPendingError } from "../native-runtime/native-startup-cleanup.js";
|
||||
import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js";
|
||||
import { describeFailedAttachment } from "../native-runtime/remote-warm-session-recovery.js";
|
||||
@@ -1160,14 +1161,39 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
// The control code can be a fencing label; the closed diagnostic says
|
||||
// what actually failed after an accepted result (no provider text).
|
||||
const postResultFailure = readNativePostResultFailureDiagnostic(adapterErr);
|
||||
// A runner that could not prove suspension after an accepted result left a
|
||||
// local workspace untouched: there is nothing to copy back. Settle the
|
||||
// accepted result instead of failing the run over runner bookkeeping.
|
||||
// Rebuild the accepted result before writing any finalize status: only a
|
||||
// proven, complete accepted result may record workspace finalization as
|
||||
// succeeded.
|
||||
let acceptedResult = nativeRuntimeResolution.kind === "native"
|
||||
&& postResultFailure?.category === "runner_settlement_incomplete"
|
||||
&& !nativeWorkspaceSync && !remoteWorkspaceAuthority
|
||||
? await (async () => {
|
||||
const resultId = await db.select({ resultId: nativeRunFinalizations.resultId }).from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, run.id)).limit(1).then(rows => rows[0]?.resultId ?? null);
|
||||
return resultId ? acceptedNativeAdapterResult(db, {
|
||||
runId: run.id, companyId: run.companyId, resultId,
|
||||
model: nativeExecution?.provider.model ?? null, driverKind: nativeExecution?.session.driverKind ?? "unknown",
|
||||
postResultFailure: postResultFailure!,
|
||||
}) : null;
|
||||
})().catch((readErr: unknown) => {
|
||||
logger.warn({ err: readErr, runId: run.id }, "accepted native result could not be read; keeping the failure path");
|
||||
return null;
|
||||
})
|
||||
: null;
|
||||
let settleAcceptedResult = acceptedResult !== null;
|
||||
try {
|
||||
await recordWorkspaceFinalize("failed", {
|
||||
errorMessage:
|
||||
adapterErr instanceof Error
|
||||
? adapterErr.message
|
||||
: String(adapterErr),
|
||||
...(postResultFailure ? { failureDiagnostic: postResultFailure } : {}),
|
||||
});
|
||||
await recordWorkspaceFinalize(settleAcceptedResult ? "succeeded" : "failed", settleAcceptedResult
|
||||
? { runnerSettlementDiagnostic: postResultFailure }
|
||||
: {
|
||||
errorMessage:
|
||||
adapterErr instanceof Error
|
||||
? adapterErr.message
|
||||
: String(adapterErr),
|
||||
...(postResultFailure ? { failureDiagnostic: postResultFailure } : {}),
|
||||
});
|
||||
} catch (recordErr) {
|
||||
logger.warn(
|
||||
{
|
||||
@@ -1178,6 +1204,9 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
"failed to record workspace_finalize=failed operation; dependents may remain gated",
|
||||
);
|
||||
}
|
||||
// Settle only when the workspace finalization record itself succeeded. A
|
||||
// failed branch or plugin workspace check keeps the failure path.
|
||||
if (adapterFinalizeOutcome !== "succeeded") { settleAcceptedResult = false; acceptedResult = null; }
|
||||
if (nativeRuntimeResolution.kind === "native") {
|
||||
const proposedResult = await db
|
||||
.select({ resultId: nativeRunFinalizations.resultId })
|
||||
@@ -1216,6 +1245,24 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
|
||||
: "workspace_sync_out_failed",
|
||||
);
|
||||
}
|
||||
if (settleAcceptedResult && acceptedResult) {
|
||||
// Finalize exactly as a returned result would, then let the ordinary
|
||||
// completion path settle the run, session, accounting and agent.
|
||||
await onLog("stderr", "[paperclip] runner settlement was incomplete after the result was accepted; finalizing the accepted result\n")
|
||||
.catch(() => undefined);
|
||||
try {
|
||||
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
|
||||
db, runId: run.id, workspaceFinalizeStatus: "succeeded",
|
||||
}));
|
||||
await dispatchPendingNativeStatusWakeups({ companyId: run.companyId });
|
||||
} catch (finalizeErr) {
|
||||
logger.warn(
|
||||
{ err: finalizeErr, runId: run.id },
|
||||
"native result persisted but finalization did not apply; the reconciliation loop will retry",
|
||||
);
|
||||
}
|
||||
return { dispatched: true, adapterResult: acceptedResult };
|
||||
}
|
||||
try {
|
||||
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
|
||||
db,
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { heartbeatRuns, nativeRunResults, type Db } from "@paperclipai/db";
|
||||
import { NATIVE_FINALIZATION_SCHEMA, type NativeFinalizationResult } from "@paperclipai/shared";
|
||||
import type { AdapterExecutionResult } from "../../adapters/index.js";
|
||||
import { readAcceptedRunUsageCheckpoint } from "../usage-receipts.js";
|
||||
import type { NativePostResultFailureDiagnostic } from "./native-post-result-failure.js";
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
}
|
||||
|
||||
const RUN_TERMINAL_STATES = new Set(["succeeded", "failed", "cancelled"]);
|
||||
|
||||
/**
|
||||
* Rebuild the execution result of an accepted native run whose runner could not
|
||||
* prove suspension when it closed. The provider turn, its result and terminal
|
||||
* state are durable; only runner bookkeeping failed. Ordinary completion can
|
||||
* then settle the run from the result instead of reporting an adapter failure.
|
||||
* Returns null unless the exact accepted result and its terminal are present.
|
||||
*/
|
||||
export async function acceptedNativeAdapterResult(db: Db, input: {
|
||||
runId: string;
|
||||
companyId: string;
|
||||
resultId: string;
|
||||
model: string | null;
|
||||
driverKind: string;
|
||||
postResultFailure: NativePostResultFailureDiagnostic;
|
||||
}): Promise<AdapterExecutionResult | null> {
|
||||
const [row] = await db.select({
|
||||
issueId: nativeRunResults.issueId,
|
||||
turnId: nativeRunResults.turnId,
|
||||
resultJson: nativeRunResults.resultJson,
|
||||
nativeSessionId: heartbeatRuns.nativeSessionId,
|
||||
runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
||||
}).from(nativeRunResults)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunResults.runId), eq(heartbeatRuns.companyId, nativeRunResults.companyId)))
|
||||
.where(and(
|
||||
eq(nativeRunResults.id, input.resultId),
|
||||
eq(nativeRunResults.runId, input.runId),
|
||||
eq(nativeRunResults.companyId, input.companyId),
|
||||
eq(nativeRunResults.schemaStatus, "accepted"),
|
||||
)).limit(1);
|
||||
if (!row) return null;
|
||||
const result = record(row.resultJson.result);
|
||||
const terminal = record(row.resultJson.terminal);
|
||||
if (terminal.schema !== "paperclip.prp.terminal.v1" || !RUN_TERMINAL_STATES.has(String(terminal.runTerminalState))) return null;
|
||||
const nativeTerminal = terminal as unknown as NativeFinalizationResult["terminal"];
|
||||
const succeeded = nativeTerminal.runTerminalState === "succeeded";
|
||||
const usage = await readAcceptedRunUsageCheckpoint(db, { companyId: input.companyId, runId: input.runId });
|
||||
const normalizedSessionId = row.nativeSessionId ?? null;
|
||||
return {
|
||||
exitCode: succeeded ? 0 : 1,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
errorMessage: succeeded ? null : `Native session ${nativeTerminal.runTerminalState}`,
|
||||
resultJson: {
|
||||
nativeResult: result,
|
||||
nativeTerminal: terminal,
|
||||
nativePostResultFailure: { ...input.postResultFailure },
|
||||
},
|
||||
summary: typeof result.summary === "string" ? result.summary : null,
|
||||
...(normalizedSessionId ? { sessionId: normalizedSessionId, sessionDisplayId: normalizedSessionId } : {}),
|
||||
model: input.model ?? undefined,
|
||||
...(usage ?? {}),
|
||||
usageComplete: usage?.complete ?? false,
|
||||
...(!usage ? { costStatus: "unpriced" as const } : {}),
|
||||
...(normalizedSessionId && row.runnerInstanceId ? {
|
||||
nativeFinalization: {
|
||||
schema: NATIVE_FINALIZATION_SCHEMA,
|
||||
runtimeMode: "native",
|
||||
runId: input.runId,
|
||||
issueId: row.issueId,
|
||||
companyId: input.companyId,
|
||||
result,
|
||||
terminal: nativeTerminal,
|
||||
turnId: row.turnId,
|
||||
sourceInstanceId: row.runnerInstanceId,
|
||||
normalizedSessionId,
|
||||
providerSessionId: null,
|
||||
driverKind: input.driverKind,
|
||||
driverVersion: "unknown",
|
||||
nativeEventCount: 0,
|
||||
highestContiguousSourceSeq: 0,
|
||||
workspaceFinalizeStatus: "succeeded",
|
||||
},
|
||||
} : {}),
|
||||
};
|
||||
}
|
||||
@@ -11249,6 +11249,31 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
// Emit after rollback so the diagnostic sink cannot deadlock on the
|
||||
// coordinator transaction. Reporting failure cannot change fencing.
|
||||
if (failure instanceof NativePostResultFailureError) {
|
||||
// The rejected write proved this exact, unexpired owner holds an
|
||||
// accepted result. Provider work is over and renewal is stopping, so
|
||||
// hand the lease to finalization now, as the success path does. A
|
||||
// retained lease would make the finalizer wait out its whole TTL.
|
||||
await stoppedLeaseRenewal;
|
||||
await boundedExecutionCleanup(async () => {
|
||||
await input.db.transaction(async tx => {
|
||||
await notifyNativeFinalizationWork(tx, input.execution.binding);
|
||||
await tx.update(nativeRunFinalizations)
|
||||
.set({ leaseOwner: null, leaseExpiresAt: null, recoveryState: null, recoveryRequestId: null, updatedAt: new Date() })
|
||||
.where(and(
|
||||
eq(nativeRunFinalizations.runId, input.execution.binding.runId),
|
||||
eq(nativeRunFinalizations.companyId, input.execution.binding.companyId),
|
||||
eq(nativeRunFinalizations.issueId, input.execution.binding.issueId),
|
||||
eq(nativeRunFinalizations.leaseOwner, leaseOwner),
|
||||
eq(nativeRunFinalizations.attempt, attempt),
|
||||
eq(nativeRunFinalizations.controllerGeneration, controllerGeneration),
|
||||
eq(nativeRunFinalizations.controllerBootId, controller.bootId),
|
||||
eq(nativeRunFinalizations.controllerPid, controller.pid),
|
||||
eq(nativeRunFinalizations.controllerProcessStartedAt, controller.processStartedAt),
|
||||
gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`),
|
||||
isNotNull(nativeRunFinalizations.resultId),
|
||||
));
|
||||
});
|
||||
});
|
||||
await boundedExecutionCleanup(async () => {
|
||||
await input.onEvent?.({ eventType: "native.post_result_failure", level: "warn",
|
||||
message: "Native execution failed after its result was accepted.",
|
||||
|
||||
Reference in new issue
Block a user