Files
PaperClipAI/server/src/services/heartbeat/runtime-selection.ts
T
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00

1183 lines
58 KiB
TypeScript

import { assertPaperclipRunnerBinaryConfiguration } from "../native-runtime/native-codex-runner.js";
import { readNativeWorkspaceMigration, migratedNativeExecutionCwd } from "../native-runtime/native-workspace-migration.js";
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
import { museRunnerBroker } from "../muse-runner-broker.js";
import { assertEphemeralNativeRuntime, readRemoteWorkspaceAuthority } from "../remote-workspaces.js";
import { measureRunTransaction } from "../execution-transaction-timing.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import {
ConfigurationIncompleteFailure,
type buildPaperclipWakePayload,
type createHeartbeatRunPreparation,
} from "./run-preparation.js";
import { dotRunnerBroker } from "../dot-runner-broker.js";
import { isAgentDirectoryCopy } from "../agent-directory-working-copies.js";
import { isConversation } from "../agent-conversations.js";
import { renewLegacyControllerLease } from "../legacy-controller-lease.js";
import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js";
import { resolveHeartbeatNativeLifecyclePolicy, type NativeLifecyclePolicy } from "./native-lifecycle.js";
import { resolveWarmProviderSelection } from "../native-runtime/native-warm-capability.js";
import {
getNativeReviewAssignment,
readNativeReviewAssignmentContext,
} from "../native-runtime/native-review-participant.js";
import { buildNativeReviewRequest } from "../native-runtime/native-review-prompt.js";
import type { CaseExecutionContext } from "../case-context.js";
import type { buildExecutionContinuation } from "../execution-continuation.js";
import type { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js";
import { createHash, randomUUID } from "node:crypto";
import { and, eq, inArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { agentHarnessType, CHAT_PROVIDERS } from "@paperclipai/shared";
import {
type agents,
completionContracts,
heartbeatRunEvents,
heartbeatRuns,
nativeRunFinalizations,
} from "@paperclipai/db";
import { getStartupTracer } from "../../instrumentation.js";
import { documentService } from "../documents.js";
import { readPortableTaskReseed } from "../plugin-task-ownership.js";
import type { PluginTaskDescription } from "../environment-task-runtime.js";
import { managedAgentProfileService } from "../managed-agent-profiles.js";
import { remoteAgentProfileService } from "../remote-agent-profiles.js";
import {
buildNativeExecutionInput,
buildNativeExecutionWithCheckpoint,
buildNativeRuntimeContext,
ensureNativeCompletionContract,
findNativeSessionResumeRun,
isUnusedNativeSessionBootstrap,
isUnusedLegacyNativeRetryReplacement,
materializeNativeInteractionResponses,
nativeCompletionRequestsWithSources,
nativeCompletionSource,
nativeImmediateObjectiveSource,
nativeToolContractFingerprintForTarget,
prepareNativeSessionBootstrapPersistence,
prepareNativeWorkspaceSync,
type NativeRestartRecoveryClaim,
rebindNativeSessionCheckpoint,
type resolveHeartbeatNativeRuntimeMode,
} from "../native-runtime/index.js";
import {
assertAgentCoreProfileRecoveryBinding,
assertManagedProfileRecoveryBinding,
projectPaperclipRunnerTaskConfig,
resolvePaperclipRunnerNativeProviderInput,
} from "../native-runtime/provider-profile.js";
import { readRemoteCodexModelCliVersion } from "../native-runtime/codex-model-fallback.js";
import {
describeRunnerdNativeSessionBackend,
parseNativeExecutionInput,
type NativeExecutionInput,
} from "../../vendor/paperclip-runner/index.js";
import { PROVIDER_TRACE_MAX_BYTES } from "../provider-trace-store.js";
import type { AdapterRuntimeEvent } from "../../adapters/index.js";
import { parseObject } from "../../adapters/utils.js";
import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "../native-runtime/external-chat-question-response.js";
import {
materializeExternalChatQuestionResponseInput,
} from "../native-runtime/external-chat-question-response-input.js";
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js";
import { NATIVE_PREPARATION_FENCING_VERSION } from "../native-runtime/native-active-runs.js";
import type { issueService } from "../issues.js";
import { buildPlanReviewContext } from "../plan-review-context.js";
import { normalizePaperclipWakePayload, selectPaperclipTaskMarkdown } from "@paperclipai/adapter-utils/server-utils";
import type { ProviderResourceDisposition } from "../environment-runtime.js";
import { stageNativeRuntimeSkillAssets, bindStagedNativeRuntimeSkillAssets } from "../native-runtime/runtime-skill-preservation.js";
type Run = typeof heartbeatRuns.$inferSelect;
type Agent = typeof agents.$inferSelect;
type IssueContext = Awaited<ReturnType<ReturnType<typeof createHeartbeatRunPreparation>["getIssueExecutionContext"]>> | null;
type IssueRef = Parameters<typeof ensureNativeCompletionContract>[0]["issue"] & Parameters<typeof buildNativeExecutionInput>[0]["issue"];
export interface NativeLifecycleTelemetry {
provider: string;
harness: string;
lifecycleMode: "per_turn" | "warm";
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
}
export interface HeartbeatRuntimeSelectionInput {
run: Run;
agent: Agent;
nativeRuntimeResolution: ReturnType<typeof resolveHeartbeatNativeRuntimeMode>;
task: {
issueRef: IssueRef | null;
issueContext: IssueContext;
context: Record<string, unknown>;
executionContinuation: Awaited<ReturnType<typeof buildExecutionContinuation>> | null;
safeWakeComments: Parameters<typeof nativeCompletionRequestsWithSources>[0];
safeWakeCommentContext: { id: string; body: string } | null;
paperclipWakePayload: Awaited<ReturnType<typeof buildPaperclipWakePayload>>;
acceptedPlanContinuationWake: boolean;
};
session: {
persistedNativeExecutionInput: NativeExecutionInput | null;
persistedRunnerProfile: Record<string, unknown>;
taskResumeSessionId: string | null;
isFailedChatRunRetry: boolean;
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
};
workspace: {
persistedExecutionWorkspace: { id: string; metadata?: Record<string, unknown> | null } | null;
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
activeEnvironmentLease: (Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> & {
pluginTaskDescription?: PluginTaskDescription;
}) | null;
isDotRun: boolean;
projectRepositoryPaths: string[];
onCheckpoint?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onCheckpoint"];
onPhase?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onPhase"];
};
config: {
nativeLifecyclePolicy?: NativeLifecyclePolicy | null;
runtimeConfig: Record<string, unknown>;
runtimeSkillEntries: Parameters<typeof buildNativeRuntimeContext>[0]["runtimeSkillEntries"];
runScopedMentionedSkillKeys: string[];
instructionCopy: Awaited<ReturnType<ReturnType<typeof agentInstructionWorkingCopyService>["prepare"]>>;
agentIdentity: { keyId: string } | undefined;
issueAssigneeOverrides: { adapterConfig: Record<string, unknown> | null } | null;
resolvedConfig: Record<string, unknown>;
managedAiRuntime: Awaited<ReturnType<typeof prepareManagedAiRuntime>> | undefined;
};
recovery: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise<void> };
trace: { providerTraceRequested: boolean; providerTraceCapture: { metadata: { id: string } } | null };
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
runtimeEnv: Record<string, string | undefined>;
beforeNativeRuntimeSelection?: (runId: string) => Promise<void>;
onAdapterEvent: (event: AdapterRuntimeEvent) => Promise<void>;
// Teardown must observe these updates even when a later preparation step throws.
onNativeLifecycleSelected: (telemetry: NativeLifecycleTelemetry) => void;
onProviderResourceDisposition: (disposition: ProviderResourceDisposition | undefined) => void;
onNativeOwnershipHeld: () => void;
stopControllerLease: () => void;
}
export type HeartbeatRuntimeSelectionResult = { selected: false } | {
selected: true;
nativeExecution: NativeExecutionInput | null;
nativeRunnerInstanceId: string | null;
pluginTaskDescription?: PluginTaskDescription;
toolRefreshOnResume?: boolean;
getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>>;
};
export interface NativeSandboxLifecycle {
runnerProcess: "per_turn" | "warm";
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
failoverBackup: "verified";
}
/** The acquired sandbox lease is reusable: configured, and supported by the provider. */
function sandboxTargetHasReusableLease(target: {
reusableLeaseConfigured?: boolean;
effectiveCapabilities?: { reusableLeases: boolean } | null;
}): boolean {
return target.reusableLeaseConfigured === true && target.effectiveCapabilities?.reusableLeases === true;
}
export function resolveReusableSandboxLifecycle(input: {
lifecyclePolicy:
| { mode: "per_turn"; idleTimeoutMs: null }
| { mode: "warm"; idleTimeoutMs: number };
target: {
kind: "local" | "remote";
transport?: string;
reusableLeaseConfigured?: boolean;
effectiveCapabilities?: { reusableLeases: boolean } | null;
} | null;
}): NativeSandboxLifecycle | null {
if (input.target?.kind !== "remote" || input.target.transport !== "sandbox") {
return null;
}
const reusableLease = sandboxTargetHasReusableLease(input.target);
if (input.lifecyclePolicy.mode === "warm" && !reusableLease) {
throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease");
}
return {
runnerProcess: input.lifecyclePolicy.mode,
sandboxResource:
input.lifecyclePolicy.mode === "warm"
? "keep_running"
: reusableLease
? "stop_and_reuse"
: "destroy_after_turn",
failoverBackup: "verified",
};
}
export function resolveNativeSandboxLifecycle(input: {
adapterType: string;
lifecyclePolicy:
| { mode: "per_turn"; idleTimeoutMs: null }
| { mode: "warm"; idleTimeoutMs: number };
target: {
kind: "local" | "remote";
transport?: string;
reusableLeaseConfigured?: boolean;
effectiveCapabilities?: { reusableLeases: boolean } | null;
} | null;
}): NativeSandboxLifecycle | null {
if (
input.adapterType !== "paperclip_runner" ||
input.target?.kind !== "remote" ||
input.target.transport !== "sandbox"
)
return null;
return resolveReusableSandboxLifecycle(input);
}
export async function postNativeModelFallbackWarning(input: {
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
onEvent: (event: AdapterRuntimeEvent) => Promise<void>;
issueId: string;
runId: string;
requestedModel: string | null;
effectiveModel: string | null;
codexCliVersion: string;
}): Promise<void> {
const message = `Using ${input.effectiveModel} because the sandbox's Codex ${input.codexCliVersion} does not support ${input.requestedModel}. Work will continue with the compatible model. Update the sandbox's Codex CLI to use the requested model.`;
await input.onEvent({
eventType: "runner.model_fallback",
stream: "system",
level: "warn",
message,
payload: {
requestedModel: input.requestedModel,
effectiveModel: input.effectiveModel,
codexCliVersion: input.codexCliVersion,
},
});
await input.issuesSvc.addComment(input.issueId, message, { runId: input.runId }, {
authorType: "system",
presentation: {
kind: "system_notice",
tone: "warning",
title: `Using ${input.effectiveModel}`,
density: "compact",
detailsDefaultOpen: false,
},
});
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
function resolveRuntimeLifecyclePolicy(input: HeartbeatRuntimeSelectionInput): NativeLifecyclePolicy | null {
if (input.nativeRuntimeResolution.kind !== "native") return null;
const target = input.workspace.executionTarget;
const sandboxTarget = target?.kind === "remote" && target.transport === "sandbox" ? target : null;
const backend = input.nativeRuntimeResolution.profile.backend;
const rawAcpxAgent = parseObject(input.agent.adapterConfig).acpxAgent;
// The environment view resolves the harness through the runtime's resolver
// (ACPX without an agent is Claude); use the same harness here so the lease
// and the runner decisions agree.
const resolved = backend === "acpx_runtime" && rawAcpxAgent === undefined
? resolveWarmProviderSelection(input.agent.adapterConfig, input.agent.companyId) : null;
return resolveHeartbeatNativeLifecyclePolicy({
adapterType: input.agent.adapterType, backend,
acpxAgent: resolved?.backend === backend ? resolved.acpxAgent : rawAcpxAgent,
agentMode: parseObject(input.agent.adapterConfig).lifecycleMode, agentIdleTimeoutMs: parseObject(input.agent.adapterConfig).idleTimeoutMs,
environmentPolicy: sandboxTarget ? sandboxTarget.runnerLifecyclePolicy : null,
persistedPolicy: input.session.persistedNativeExecutionInput?.session.lifecyclePolicy,
managedAiSelected: Boolean(input.config.managedAiRuntime), managedAiAttribution: input.config.managedAiRuntime?.attribution,
taskOwned: Boolean(sandboxTarget?.runnerTask),
localTarget: target?.kind !== "remote",
remoteSandboxTarget: sandboxTarget && !sandboxTarget.runnerTask
? { reusableLease: sandboxTargetHasReusableLease(sandboxTarget), idleTimeoutMs: sandboxTarget.runnerIdleTimeoutMs } : undefined,
});
}
export async function prepareAndSelectHeartbeatRuntime(
db: Db,
input: HeartbeatRuntimeSelectionInput,
preparation: {
prepareInstructions: (policy: NativeLifecyclePolicy | null) => Promise<Pick<HeartbeatRuntimeSelectionInput["config"], "runtimeConfig" | "instructionCopy">>;
measure: <T>(phase: "instructions" | "runtime_selection", work: () => Promise<T>) => Promise<T>;
},
): Promise<HeartbeatRuntimeSelectionResult> {
const nativeLifecyclePolicy = resolveRuntimeLifecyclePolicy(input);
const prepared = await preparation.measure("instructions", () => preparation.prepareInstructions(nativeLifecyclePolicy));
return preparation.measure("runtime_selection", () => selectHeartbeatRuntime(db, {
...input,
config: { ...input.config, ...prepared, nativeLifecyclePolicy },
}));
}
export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSelectionInput): Promise<HeartbeatRuntimeSelectionResult> {
const { run, agent, nativeRuntimeResolution, issuesSvc, runtimeEnv, beforeNativeRuntimeSelection, onAdapterEvent,
onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input;
const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext,
paperclipWakePayload, acceptedPlanContinuationWake } = input.task;
const { persistedNativeExecutionInput, persistedRunnerProfile, taskResumeSessionId, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace;
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
const runOptions = input.recovery;
const { providerTraceRequested, providerTraceCapture } = input.trace;
if (!isDotRun) assertEphemeralNativeRuntime(nativeRuntimeResolution.kind === "native");
let nativeExecution: NativeExecutionInput | null = null;
let toolRefreshOnResume: boolean | undefined;
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
let nativeRunnerInstanceId: string | null = null;
let pluginTaskDescription = activeEnvironmentLease?.pluginTaskDescription;
let nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>> = null;
let providerResourceDispositionForRun: ProviderResourceDisposition | undefined;
if (nativeRuntimeResolution.kind === "native") {
if (!issueRef) {
throw new Error("native_runtime_ineligible: issue is required");
}
if (run.runtimeMode !== "native" && agent.adapterType === "paperclip_runner"
&& executionTarget?.kind !== "remote"
&& nativeRuntimeResolution.profile.backend !== "cursor_cloud_api"
&& nativeRuntimeResolution.profile.backend !== "claude_managed_agents_api"
&& nativeRuntimeResolution.profile.backend !== "aws_agentcore_harness_api") {
assertPaperclipRunnerBinaryConfiguration();
}
const nativeExecutionWorkspaceId =
persistedNativeExecutionInput?.provider.kind === "cursor_cloud"
// Older Cursor runs keep their admitted binding identity. Their task
// files and Computer are not part of the recovered API controller.
? persistedNativeExecutionInput.binding.executionWorkspaceId
: persistedExecutionWorkspace?.id ?? run.id;
const nativeReviewContext = readNativeReviewAssignmentContext(context);
const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, {
companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id,
contextSnapshot: nativeReviewContext,
})) : null;
if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available");
const nativeReviewRequest = nativeReview
? buildNativeReviewRequest({
title: nativeReview.interaction.title,
summary: nativeReview.interaction.summary,
payload: nativeReview.interaction.payload,
})
: null;
const persistedContract = run.completionContractId
? await measureRunPhase("runtime_selection.contract_read", () => db
.select()
.from(completionContracts)
.where(
and(
eq(completionContracts.id, run.completionContractId!),
eq(completionContracts.companyId, agent.companyId),
eq(completionContracts.issueId, issueRef.id),
),
)
.limit(1)
.then((rows) => rows[0] ?? null))
: null;
// Only a server-verified human resolution may supply a current answer
// reference. Tool/agent results and generated summaries stay evidence.
const currentHumanResponseId = !nativeReviewRequest
? executionContinuation?.humanResponses?.find(
(response) => response.id === executionContinuation.trigger.interactionId,
)?.id
: undefined;
const immediateCompletion = (() => {
if (nativeReviewRequest) return { requests: [nativeReviewRequest], sources: [null] };
const { requests, sources } = nativeCompletionRequestsWithSources(
safeWakeComments.length > 0
? safeWakeComments
: safeWakeCommentContext?.body
? [safeWakeCommentContext]
: [],
{
requiredFullWakeCommentCount:
paperclipWakePayload?.fallbackFetchNeeded === true &&
CHAT_PROVIDERS.some(
(provider) =>
provider ===
paperclipWakePayload.externalChatProvider,
) &&
Array.isArray(paperclipWakePayload.commentIds)
? paperclipWakePayload.commentIds.length
: undefined,
},
);
// Preserve every admitted pending chat request while also
// retaining newer user direction materialized by recovery.
// A file-only wake must not inherit an old task objective.
const latestComment =
executionContinuation?.messages.findLast(
(message) =>
message.authorType === "user" &&
!message.createdByRunId &&
!message.deleted &&
message.body.trim().length > 0,
);
const latestRequest = latestComment?.body;
if (
latestRequest &&
!requests.some(
(request) => request === latestRequest.trim(),
)
) {
requests.push(latestRequest.trim());
sources.push(nativeCompletionSource("comment", latestComment!.id, latestRequest));
}
return { requests: requests.length > 0 ? requests : undefined, sources };
})();
// Rebuilding a default contract is not a change in user direction.
// In particular, an upgraded checkpoint may have an intentionally
// authored contract and no continuation envelope yet.
const completionContract =
persistedContract && persistedNativeExecutionInput
? {
row: persistedContract,
contract: persistedContract.contractJson as never,
}
: await measureRunPhase("runtime_selection.contract_ensure", () => ensureNativeCompletionContract({
db,
companyId: agent.companyId,
issue: issueRef,
actorId: agent.id,
immediateRequest:
nativeReviewRequest ?? (currentHumanResponseId
? null
: executionContinuation?.objective ?? safeWakeCommentContext?.body ?? null),
// The ordinary initial objective is selected by the server-owned
// continuation envelope. Carry its explicit description source
// through the singular-request compatibility path; do not infer
// provenance for review requests, answers, or wake fallbacks.
immediateRequestSource: nativeImmediateObjectiveSource({
issueId: issueRef.id,
objectiveSource: executionContinuation?.objectiveSource,
excluded: Boolean(nativeReviewRequest || currentHumanResponseId),
}),
humanResponseId: currentHumanResponseId,
immediateRequests: immediateCompletion.requests,
immediateRequestSources: immediateCompletion.sources,
}));
const portableLease = activeEnvironmentLease?.lease;
const runnerTask = executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" &&
executionTarget.leaseId === portableLease?.id ? executionTarget.runnerTask : undefined;
const portable = portableLease && runnerTask ? await readPortableTaskReseed(db, {
companyId: agent.companyId, runId: run.id, lease: portableLease, description: pluginTaskDescription,
runnerTask,
}) : null;
if (portable) pluginTaskDescription = portable.description;
const reseed = portable?.reseed;
if (reseed && run.nativeSessionId && (!reseed.previousSessionId || run.nativeSessionId === reseed.previousSessionId ||
persistedNativeExecutionInput?.session.normalizedSessionId !== run.nativeSessionId)) {
throw new Error("native_portable_reseed_session_already_attached");
}
const reseedPortableSession = Boolean(reseed);
// Compatibility for native retry rows created before same-run restart
// recovery existed. Only an entirely unused replacement row may
// inherit its source checkpoint; any process/provider evidence on the
// replacement makes the ownership ambiguous and therefore ineligible.
const legacyRetrySource =
run.retryOfRunId && !isFailedChatRunRetry
? await measureRunPhase("runtime_selection.resume", () => db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
runnerInstanceId: heartbeatRuns.runnerInstanceId,
nativeSessionId: heartbeatRuns.nativeSessionId,
runnerProfileJson: heartbeatRuns.runnerProfileJson,
runtimeMode: heartbeatRuns.runtimeMode,
status: heartbeatRuns.status,
})
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, run.retryOfRunId!),
eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.agentId, agent.id),
),
)
.limit(1)
.then((rows) => rows[0] ?? null))
: null;
const nativeBootstrapHasProviderEvidence =
legacyRetrySource ||
run.nativeSessionId ||
persistedNativeExecutionInput
? await measureRunPhase("runtime_selection.resume", () => db
.select({ id: heartbeatRunEvents.id })
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.runId, run.id),
inArray(heartbeatRunEvents.eventType, [
"harness.ready",
"session.started",
"session.resumed",
"session.updated",
"turn.started",
"provider.event",
"provider.rpc_result",
]),
),
)
.limit(1)
.then((rows) => rows.length > 0))
: false;
const compatibleLegacyRetrySource =
!managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({
replacement: run,
source: legacyRetrySource,
hasProviderEvents: nativeBootstrapHasProviderEvidence,
})
? legacyRetrySource
: null;
const legacyRetrySessionId =
compatibleLegacyRetrySource?.nativeSessionId;
const resumableTaskSessionId = reseedPortableSession || isDotRun ? null : taskResumeSessionId ?? legacyRetrySessionId ?? null;
const requestedNativeSessionId =
run.nativeSessionId ?? resumableTaskSessionId;
// A task-session lastRunId can lag a failed turn or point at an older
// normalized session. Find the newest exact-session authority instead.
// Rows that already acquired provider authority are progress barriers,
// even when they do not contain a usable checkpoint.
const previousNativeRun =
requestedNativeSessionId &&
isUnusedNativeSessionBootstrap(
run,
nativeBootstrapHasProviderEvidence,
)
? await measureRunPhase("runtime_selection.resume", () => findNativeSessionResumeRun(db, {
companyId: agent.companyId,
agentId: agent.id,
issueId: issueRef.id,
normalizedSessionId: requestedNativeSessionId,
currentRunId: run.id,
beforeCreatedAt: run.createdAt,
}))
: null;
nativeRunnerInstanceId =
previousNativeRun?.runnerInstanceId &&
previousNativeRun.nativeSessionId ===
(run.nativeSessionId ?? resumableTaskSessionId)
? previousNativeRun.runnerInstanceId
: (run.runnerInstanceId ?? randomUUID());
let nativeSessionId =
run.nativeSessionId ?? resumableTaskSessionId ?? randomUUID();
let nativeResumeCheckpoint: ReturnType<
typeof rebindNativeSessionCheckpoint
> = null;
const effectiveLifecyclePolicy = input.config.nativeLifecyclePolicy ?? resolveRuntimeLifecyclePolicy(input)!;
if (
effectiveLifecyclePolicy.mode === "warm" &&
executionTarget?.kind === "remote" &&
executionTarget.transport === "sandbox" &&
(executionTarget.reusableLeaseConfigured !== true ||
executionTarget.effectiveCapabilities?.reusableLeases !== true)
) {
throw new Error("runner_warm_environment_requires_reusable_lease");
}
const persistedProfile = persistedRunnerProfile;
if (persistedNativeExecutionInput) {
nativeExecution = persistedNativeExecutionInput;
toolRefreshOnResume = persistedProfile.nativeToolRefreshOnResume === true;
if (
nativeExecution.binding.companyId !== agent.companyId ||
nativeExecution.binding.runId !== run.id ||
nativeExecution.binding.issueId !== issueRef.id ||
nativeExecution.binding.agentId !== agent.id ||
nativeExecution.binding.executionWorkspaceId !==
nativeExecutionWorkspaceId ||
nativeExecution.completionContract.id !==
completionContract.row.id ||
nativeExecution.completionContract.sha256 !==
completionContract.row.canonicalSha256
)
throw new Error(
"native_execution_input_persisted_binding_mismatch",
);
if ("runtimeContext" in nativeExecution) {
const savedVersions = parseObject(run.contextSnapshot?.skillVersions);
const accessibleKeys = new Set(runtimeSkillEntries.filter(entry => entry.sourceStatus !== "missing").map(entry => entry.key));
if (nativeExecution.runtimeContext.skills.some(skill =>
Object.prototype.hasOwnProperty.call(savedVersions, skill.key) && !accessibleKeys.has(skill.key))) {
throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", {
configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" },
});
}
}
// Recover only the originally admitted request. A stored idle
// checkpoint can precede an already-started provider turn, so even
// apparent idleness is not authority to rewrite its contract.
// Newer user direction retains its separate durable wakeup cause.
// A failed pre-bootstrap attempt may have persisted its immutable
// input before discovering that lastRunId no longer names this
// session. Restore only an exactly compatible prior checkpoint;
// never rewrite the admitted input or skip current provider work.
if (
previousNativeRun &&
isUnusedNativeSessionBootstrap(
run,
nativeBootstrapHasProviderEvidence,
)
) {
nativeResumeCheckpoint = rebindNativeSessionCheckpoint({
previousRun: previousNativeRun,
currentExecution: nativeExecution,
executionTargetKind: executionTarget?.kind ?? "local",
});
}
if (nativeExecution.provider.kind === "claude_managed") {
const recoveryProfileId = nativeExecution.provider.managedProfile.profileId;
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(
db,
).requireQualified(
agent.companyId,
recoveryProfileId,
));
assertManagedProfileRecoveryBinding({
adapterConfig: agent.adapterConfig,
snapshot: nativeExecution.provider.managedProfile,
stored: recoveryProfile,
});
}
if (nativeExecution.provider.kind === "aws_agentcore") {
const recoveryProfileId = nativeExecution.provider.agentCoreProfile.profileId;
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(
db,
).requireQualified(
agent.companyId,
recoveryProfileId,
"aws_bedrock_agentcore_harness",
));
assertAgentCoreProfileRecoveryBinding({
snapshot: nativeExecution.provider.agentCoreProfile,
stored: recoveryProfile,
});
}
} else {
const interactionId = readNonEmptyString(context.interactionId);
const interactionResponses = context[
EXTERNAL_CHAT_QUESTION_RESPONSE_KEY
]
? await measureRunPhase("runtime_selection.interactions", () => materializeExternalChatQuestionResponseInput({
db,
binding: {
companyId: agent.companyId,
issueId: issueRef.id,
runId: run.id,
agentId: agent.id,
},
contextSnapshot: context,
}))
: await measureRunPhase("runtime_selection.interactions", () => materializeNativeInteractionResponses({
db,
companyId: agent.companyId,
issueId: issueRef.id,
runId: run.id,
agentId: agent.id,
interactionIds: Array.isArray(context.interactionIds)
? [
...new Set([
...(interactionId ? [interactionId] : []),
...context.interactionIds.filter(
(id): id is string => typeof id === "string",
),
]),
]
: interactionId
? [interactionId]
: [],
}));
const runnerAdapterConfig = parseObject(agent.adapterConfig);
const managedProfile =
nativeRuntimeResolution.profile.backend ===
"claude_managed_agents_api"
? await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(db).requireQualified(
agent.companyId,
readNonEmptyString(runnerAdapterConfig.managedProfileId) ??
"",
))
: null;
const agentCoreProfile =
nativeRuntimeResolution.profile.backend ===
"aws_agentcore_harness_api"
? await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(db).requireQualified(
agent.companyId,
readNonEmptyString(
runnerAdapterConfig.agentCoreProfileId,
) ?? "",
"aws_bedrock_agentcore_harness",
))
: null;
if (managedProfile) {
const rawApiKeyBinding = parseObject(
runnerAdapterConfig.env,
).ANTHROPIC_API_KEY;
const boundSecretId =
typeof rawApiKeyBinding === "object" &&
rawApiKeyBinding !== null
? readNonEmptyString(
(rawApiKeyBinding as Record<string, unknown>).secretId,
)
: null;
if (boundSecretId !== managedProfile.apiKeySecretId) {
throw new ConfigurationIncompleteFailure(
"configuration incomplete: Claude Managed profile API key is not bound at env.ANTHROPIC_API_KEY",
{
configurationIncomplete: {
reason: "managed_agent_profile_secret_binding_mismatch",
companyId: agent.companyId,
agentId: agent.id,
profileId: managedProfile.id,
requiredEnvKeys: ["ANTHROPIC_API_KEY"],
},
},
);
}
}
const executionMode =
issueRef.workMode === "planning" && !isConversation(issueContext) && !acceptedPlanContinuationWake
? ("plan" as const)
: ("default" as const);
const pinnedPlan =
executionMode === "plan"
? await documentService(db).getIssueDocumentByKey(
issueRef.id,
"plan",
)
: null;
const pinnedReviewContext =
executionMode === "plan"
? await buildPlanReviewContext({
db,
companyId: agent.companyId,
issueId: issueRef.id,
issueWorkMode: issueRef.workMode,
interactionId: readNonEmptyString(context.interactionId),
})
: null;
const pinnedPlanMarkdown = pinnedPlan?.body ?? "";
const museBinding = nativeRuntimeResolution.profile.backend === "muse_external"
? await museRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig).museBindingId ?? "")) : undefined;
const dotBinding = ["openai_dot_mcp", "grok_bot_mcp"].includes(nativeRuntimeResolution.profile.backend)
? await dotRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig)[nativeRuntimeResolution.profile.backend === "grok_bot_mcp" ? "externalBindingId" : "dotBindingId"] ?? "")) : undefined;
const nativeRuntimeContext = await measureRunPhase("runtime_selection.runtime_context", () => buildNativeRuntimeContext({
db,
agent,
runId: run.id,
runtimeConfig,
runtimeSkillEntries,
instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined,
}));
getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff;
const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "cursor_cloud_api"
|| nativeRuntimeResolution.profile.backend === "codex_app_server"
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime" && ["claude_local", "grok_local"].includes(agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig))))
|| nativeRuntimeResolution.profile.backend === "opencode_server"
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime"
&& parseObject(agent.adapterConfig).acpxAgent === "copilot")
? projectPaperclipRunnerTaskConfig(
nativeRuntimeResolution.profile.backend,
agent.adapterConfig,
issueAssigneeOverrides?.adapterConfig,
managedAiRuntime ? readNonEmptyString(resolvedConfig.model) ?? undefined : undefined,
)
: agent.adapterConfig;
const requestedNativeProvider = resolvePaperclipRunnerNativeProviderInput({
companyId: agent.companyId,
backend: nativeRuntimeResolution.profile.backend,
adapterConfig: nativeProviderConfig, managedProfile, agentCoreProfile, dotBinding, museBinding,
});
const codexCliVersion = agent.adapterType === "paperclip_runner" && requestedNativeProvider.provider === "codex"
? await measureRunPhase("runtime_selection.codex_cli_version", () => readRemoteCodexModelCliVersion({
model: requestedNativeProvider.model,
target: executionTarget,
remoteCodexPath: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH,
remoteCodexNpmSpec: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC,
})) : null;
const nativeProvider = codexCliVersion
? resolvePaperclipRunnerNativeProviderInput({
companyId: agent.companyId,
backend: nativeRuntimeResolution.profile.backend,
adapterConfig: nativeProviderConfig, codexCliVersion, managedProfile, agentCoreProfile, dotBinding, museBinding,
}) : requestedNativeProvider;
if (nativeProvider.model !== requestedNativeProvider.model) {
await postNativeModelFallbackWarning({
issuesSvc, onEvent: onAdapterEvent, issueId: issueRef.id, runId: run.id,
requestedModel: requestedNativeProvider.model, effectiveModel: nativeProvider.model,
codexCliVersion: codexCliVersion!,
});
}
const workspaceMigration = persistedExecutionWorkspace?.metadata?.nativeWorkspaceMigration
? await readNativeWorkspaceMigration(db, { companyId: agent.companyId, id: nativeExecutionWorkspaceId }) : null;
const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) =>
buildNativeExecutionInput({
agentKeyId: agentIdentity?.keyId,
companyId: agent.companyId,
runId: run.id,
issue: nativeReviewRequest ? { ...issueRef, title: `Review: ${issueRef.title}`, description: nativeReviewRequest } : issueRef,
taskPrompt: [
nativeReviewRequest ?? readNonEmptyString(
selectPaperclipTaskMarkdown(context, {
resumedSession: false,
includeCommunicationGuidance: false,
}),
) ??
`# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`,
!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) && projectRepositoryPaths.length > 0
? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}`
: null,
].filter(Boolean).join("\n\n"),
initialCommunicationGuidance: nativeReviewRequest ? null : readNonEmptyString(context.paperclipTaskCommunicationGuidance),
wakePayload: context.paperclipWake,
caseContext: nativeReviewRequest ? null : context.paperclipCase as CaseExecutionContext | undefined,
turnContext: context.paperclipTurnContext,
githubInstructionSkillKeys: runScopedMentionedSkillKeys,
resumedSession,
previousTurn: (() => {
if (!previousNativeRun || nativeReviewRequest) return null;
try {
const previousInput = parseNativeExecutionInput(parseObject(previousNativeRun.runnerProfileJson).nativeExecutionInput);
const previousTask = previousInput.task;
const previousWake = normalizePaperclipWakePayload(parseObject(previousNativeRun.contextSnapshot).paperclipWake);
const githubReviewConfigurationKeys = previousWake?.githubReviewConfigurations.map((config) => config.key);
if (paperclipWakePayload?.externalChatProvider || previousWake?.externalChatProvider) {
// External native inputs use a neutral task title. Compare
// the saved canonical brief so old provider text is not
// repeated as a change, while genuine edits still arrive.
const savedIssue = parseObject(parseObject(previousNativeRun.contextSnapshot).paperclipIssue);
if (savedIssue.id !== issueRef.id || typeof savedIssue.title !== "string" ||
(savedIssue.description !== null && typeof savedIssue.description !== "string")) return null;
return { runId: previousNativeRun.id, task: { title: savedIssue.title, description: savedIssue.description },
githubReviewConfigurationKeys, nativeExecutionInput: previousInput,
};
}
return { runId: previousNativeRun.id, task: previousTask, githubReviewConfigurationKeys, nativeExecutionInput: previousInput };
} catch {
// An invalid prior snapshot must use the fresh bootstrap.
return null;
}
})(),
conversationMode: context.conversationMode === true,
agentId: agent.id,
workspace: {
// Projectless paperclip_runner tasks still have a resolved local cwd. Bind that
// transient workspace to the run id so the native input remains durable and replayable
// without fabricating a project-scoped execution_workspaces row.
id: nativeExecutionWorkspaceId,
cwd: migratedNativeExecutionCwd(workspaceMigration, { agentId: agent.id, issueId: issueRef.id, sessionId: normalizedSessionId }) ?? executionWorkspace.cwd,
repoUrl: executionWorkspace.repoUrl,
repoRef: executionWorkspace.repoRef,
branchName: executionWorkspace.branchName,
},
normalizedSessionId,
executionMode,
planningContext:
executionMode === "plan"
? {
documentId: pinnedPlan?.id ?? null,
baseRevisionId:
pinnedPlan?.latestRevisionId ?? null,
baseRevisionNumber:
pinnedPlan?.latestRevisionNumber ?? 0,
markdown: pinnedPlanMarkdown,
sha256: createHash("sha256")
.update(pinnedPlanMarkdown)
.digest("hex"),
reviewContext: pinnedReviewContext
? (structuredClone(
pinnedReviewContext,
) as unknown as Record<string, unknown>)
: {},
}
: null,
...nativeProvider,
lifecyclePolicy: effectiveLifecyclePolicy,
interactionResponses,
completionContract: {
id: completionContract.row.id,
sha256: completionContract.row.canonicalSha256,
schemaVersion: completionContract.row.schemaVersion,
contract: completionContract.contract,
sources: "sources" in completionContract ? completionContract.sources : undefined,
},
runtimeContext: nativeRuntimeContext,
});
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
})));
toolRefreshOnResume = backendDescriptor.capabilities.toolRefreshOnResume === true;
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
previousRun: previousNativeRun,
normalizedSessionId: nativeSessionId,
executionTargetKind: executionTarget?.kind ?? "local",
toolRefreshOnResume,
refreshTools: context.refreshTools === true,
buildExecution,
});
nativeExecution = nativeExecutionWithCheckpoint.execution;
nativeResumeCheckpoint = nativeExecutionWithCheckpoint.checkpoint;
if (
nativeSessionId !==
nativeExecutionWithCheckpoint.normalizedSessionId
) {
nativeRunnerInstanceId = randomUUID();
}
nativeSessionId = nativeExecutionWithCheckpoint.normalizedSessionId;
}
if (nativeResumeCheckpoint && nativeExecution.provider.kind === "acpx"
&& nativeExecution.provider.agent === "antigravity") {
nativeResumeCheckpoint = { ...nativeResumeCheckpoint, providerRecoveryPolicy: "same_session_only" };
}
const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({
adapterType: agent.adapterType,
lifecyclePolicy: nativeExecution.session.lifecyclePolicy,
target: executionTarget,
});
if (nativeSandboxLifecycle) {
onNativeLifecycleSelected({
provider: nativeExecution.provider.kind,
harness: nativeExecution.session.driverKind,
lifecycleMode: nativeExecution.session.lifecyclePolicy.mode,
sandboxResource: nativeSandboxLifecycle.sandboxResource,
});
const selectedLifecycleSpan = getStartupTracer(
"paperclip.environment-lifecycle",
).startSpan("sandbox.lifecycle.selected", {
attributes: {
"paperclip.native.span.provider": nativeExecution.provider.kind,
"paperclip.native.span.harness":
nativeExecution.session.driverKind,
"paperclip.native.span.lifecycle_mode":
nativeExecution.session.lifecyclePolicy.mode,
"paperclip.native.span.sandbox_resource":
nativeSandboxLifecycle.sandboxResource,
"paperclip.native.span.outcome": "selected",
"paperclip.native.span.bytes_transferred": 0,
},
});
selectedLifecycleSpan.end();
}
providerResourceDispositionForRun =
nativeSandboxLifecycle?.sandboxResource === "keep_running"
? "keep_running"
: nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse"
? "stop_and_retain"
: nativeSandboxLifecycle?.sandboxResource ===
"destroy_after_turn"
? "destroy"
: undefined;
onProviderResourceDisposition(providerResourceDispositionForRun);
const stagedRuntimeSkills = await measureRunPhase("runtime_selection.skill_assets", () => stageNativeRuntimeSkillAssets(db, {
companyId: agent.companyId, agentId: agent.id, runId: run.id, issueId: issueRef.id, execution: nativeExecution!,
}));
await beforeNativeRuntimeSelection?.(run.id);
const nativeSelected = await measureRunTransaction(db, "runtime_selection", async (tx) => {
const lockedRun = await measureRunPhase("runtime_selection.row_lock", () => tx
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, run.id))
.for("update")
.limit(1)
.then((rows) => rows[0] ?? null));
if (!lockedRun) throw new Error("native_runtime_run_missing");
// Cancellation and runtime selection serialize on this row. A
// stopped preparation must never create a new native coordinator.
const stopRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
&& runOptions.nativeRestartRecovery.runId === lockedRun.id
&& runOptions.nativeRestartRecovery.leaseOwner === runOptions.nativeLeaseOwner
&& await readClaimedNativeStopRecovery(tx as unknown as Db, lockedRun, runOptions.nativeLeaseOwner);
if (lockedRun.status !== "running" || (lockedRun.resultJson?.startupCancellation && !stopRecovery)) return false;
if (lockedRun.runtimeMode === "legacy" && lockedRun.controllerBootId &&
!(await renewLegacyControllerLease(tx as unknown as Db, lockedRun))) {
onNativeOwnershipHeld();
return false;
}
if (
lockedRun.runtimeModeResolvedAt &&
lockedRun.runtimeMode !== "native"
) {
throw new Error("native_runtime_mode_conflict");
}
const lockedProfile = parseObject(lockedRun.runnerProfileJson);
const runtimeSkillPreservation = await bindStagedNativeRuntimeSkillAssets(tx as unknown as Db, {
companyId: lockedRun.companyId, agentId: lockedRun.agentId, runId: lockedRun.id, issueId: issueRef.id,
execution: (lockedProfile.nativeExecutionInput ?? nativeExecution) as NativeExecutionInput, staged: stagedRuntimeSkills,
});
const persistedNativeSessionId =
await measureRunPhase("runtime_selection.bootstrap", () => prepareNativeSessionBootstrapPersistence(tx, {
run: lockedRun,
selectedSessionId: nativeSessionId,
execution: nativeExecution!,
restoringCheckpoint: nativeResumeCheckpoint !== null,
}));
await measureRunPhase("runtime_selection.run_write", () => tx
.update(heartbeatRuns)
.set({
runtimeMode: "native",
legacyProcessSettlement: null,
controllerBootId: null,
controllerLeaseExpiresAt: null,
executionStage: null,
runtimeModeResolverVersion:
lockedRun.runtimeModeResolverVersion ??
nativeRuntimeResolution.resolverVersion,
runtimeModeReason:
lockedRun.runtimeModeReason ?? nativeRuntimeResolution.reason,
runtimeModeResolvedAt:
lockedRun.runtimeModeResolvedAt ?? new Date(),
runnerProfileJson: {
...nativeRuntimeResolution.profile,
...lockedProfile,
...(runtimeSkillPreservation ? { nativeRuntimeSkillPreservation: runtimeSkillPreservation } : {}),
...(nativeExecution?.provider.kind === "cursor_cloud" ? { executionPlacement: resolveProviderApiPlacement({ agent, run }) } : {}),
nativePreparationFencingVersion: !lockedRun.runnerInstanceId && !lockedRun.nativeSessionId
? NATIVE_PREPARATION_FENCING_VERSION
: lockedProfile.nativePreparationFencingVersion,
...(lockedProfile.nativeExecutionInput
? {}
: { recoveryEventInventoryVersion: 1 }),
...(providerTraceRequested
? {
providerTrace: {
mode: "raw",
traceId: providerTraceCapture?.metadata.id ?? null,
maxBytes: PROVIDER_TRACE_MAX_BYTES,
},
}
: {}),
nativeExecutionInput:
lockedProfile.nativeExecutionInput ?? nativeExecution,
nativeToolRefreshOnResume: lockedProfile.nativeExecutionInput
? lockedProfile.nativeToolRefreshOnResume === true : toolRefreshOnResume === true,
...(typeof lockedProfile.inheritedProcessIdentityRunId === "string"
? { inheritedProcessIdentityRunId: lockedProfile.inheritedProcessIdentityRunId }
: lockedRun.processPid == null && lockedRun.processGroupId == null && lockedRun.processStartedAt == null
&& previousNativeRun?.nativeSessionId === nativeSessionId
&& (previousNativeRun.processPid != null || previousNativeRun.processGroupId != null)
? { inheritedProcessIdentityRunId: previousNativeRun.id }
: {}),
nativeResourceDisposition: providerResourceDispositionForRun,
nativeToolContractFingerprint:
nativeToolContractFingerprintForTarget(
executionTarget?.kind ?? "local",
nativeExecution!.provider.kind,
),
...(lockedProfile.sessionCheckpoint != null
? { sessionCheckpoint: lockedProfile.sessionCheckpoint }
: nativeResumeCheckpoint
? {
sessionCheckpoint:
nativeResumeCheckpoint as unknown as Record<
string,
unknown
>,
}
: {}),
},
runnerInstanceId:
previousNativeRun?.runnerInstanceId &&
persistedNativeSessionId === previousNativeRun.nativeSessionId
? previousNativeRun.runnerInstanceId
: lockedRun.nativeSessionId !== persistedNativeSessionId
? nativeRunnerInstanceId
: (lockedRun.runnerInstanceId ?? nativeRunnerInstanceId),
nativeSessionId: persistedNativeSessionId,
processPid:
lockedRun.processPid ??
(previousNativeRun?.nativeSessionId === nativeSessionId
? previousNativeRun.processPid
: null),
processGroupId:
lockedRun.processGroupId ??
(previousNativeRun?.nativeSessionId === nativeSessionId
? previousNativeRun.processGroupId
: null),
processStartedAt:
lockedRun.processStartedAt ??
(previousNativeRun?.nativeSessionId === nativeSessionId
? previousNativeRun.processStartedAt
: null),
nativeIssueId: lockedRun.nativeIssueId ?? issueRef.id,
driverKind:
lockedRun.driverKind ??
nativeExecution?.session.driverKind ??
"codex_app_server",
driverVersion: lockedRun.driverVersion ?? "phase6-v1",
completionContractId:
lockedRun.completionContractId ?? completionContract.row.id,
completionContractSha256:
lockedRun.completionContractSha256 ??
completionContract.row.canonicalSha256,
nativePhase: lockedRun.nativePhase ?? "observed",
nativePhaseUpdatedAt:
lockedRun.nativePhaseUpdatedAt ?? new Date(),
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, run.id)));
await measureRunPhase("runtime_selection.finalization_write", () => tx
.insert(nativeRunFinalizations)
.values({
runId: run.id,
companyId: agent.companyId,
issueId: issueRef.id,
phase: "observed",
})
.onConflictDoNothing());
return true;
});
if (!nativeSelected) return { selected: false };
stopControllerLease();
if (nativeExecution?.provider.kind !== "cursor_cloud" && !activeEnvironmentLease) throw new Error("native_environment_lease_unavailable");
nativeWorkspaceSync = nativeExecution?.provider.kind === "cursor_cloud" || readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata) ? null
: await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({
db,
runId: run.id,
companyId: agent.companyId,
workspaceId: nativeExecutionWorkspaceId,
workspaceLocalDir: executionWorkspace.cwd,
target: executionTarget,
lease: activeEnvironmentLease!.lease,
restartRecovery: runOptions.nativeRestartRecovery,
sameRunRecovery: Boolean(runOptions.nativeLeaseOwner),
resourceDisposition: providerResourceDispositionForRun,
workspaceGitIndexRefresh: nativeExecution?.provider.kind === "opencode",
onCheckpoint: input.workspace.onCheckpoint,
onPhase: input.workspace.onPhase,
onEvent: onAdapterEvent,
}));
} else {
const legacyWarmLifecycle =
executionTarget?.kind === "remote" &&
executionTarget.transport === "sandbox" &&
executionTarget.runnerLifecyclePolicy?.mode === "warm"
? resolveReusableSandboxLifecycle({
lifecyclePolicy: executionTarget.runnerLifecyclePolicy,
target: executionTarget,
})
: null;
if (legacyWarmLifecycle?.sandboxResource === "keep_running") {
providerResourceDispositionForRun = "keep_running";
onProviderResourceDisposition(providerResourceDispositionForRun);
}
await measureRunPhase("runtime_selection.run_write", () => db
.update(heartbeatRuns)
.set({
runtimeMode: "legacy",
runtimeModeResolverVersion:
nativeRuntimeResolution.resolverVersion,
runtimeModeReason: nativeRuntimeResolution.reason,
runtimeModeResolvedAt: run.runtimeModeResolvedAt ?? new Date(),
// Preserve server-owned admission and dispatch evidence on this
// row; never copy another run's execution profile.
runnerProfileJson: sql`(case when ${heartbeatRuns.runnerProfileJson} ? ${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}
then jsonb_build_object(${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}::text, ${heartbeatRuns.runnerProfileJson}->${CHAT_CONTROL_RECOVERY_ADMISSION_KEY})
else '{}'::jsonb end)
|| (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch'
then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch')
else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, run.id)));
}
return { selected: true, nativeExecution, nativeRunnerInstanceId, pluginTaskDescription, toolRefreshOnResume, getNativeFreshSessionHandoff, nativeWorkspaceSync };
}