Files
PaperClipAI/server/src/services/heartbeat/runtime-selection.ts
T
DottaandPaperclip 1f55073f96 fix(cursor-cloud): support API-only controllers on managed stacks (#18310)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - Cursor Cloud hosts execution behind its API.
> - Product setup still expected a Paperclip Computer.
> - Managed-only tenants therefore rejected valid setup.
> - This admits the validated API-owned controller.
> - Authority, Stop proof, budgets and cleanup remain enforced.

## Linked Issues or Issue Description

Refs #18068 and #18069; both dependencies have merged.

**What happened?**

Managed-only setup still requested a Computer. Live tests exposed
missing Cursor Stop proof and duplicate replay.

**Expected behavior**

Use Cursor’s API controller. Verify provider/controller termination and
replay each log identity once.

**Steps to reproduce**

Enable native Runner/managed-only. Configure Cursor model/repository/key
without a Computer, then Test, finish setup, run and Stop.

## What Changed

- Derive API-only placement from validated native Cursor configuration;
reject placement/command/workspace/home overrides and keep
Kubernetes-only execution closed.
- Skip Computers/checkout/workspace realization. Keep scoped tools,
completion, private staging, budgets, leases and recovery. Persist
placement; reject contradictions.
- Align Cloud cards, Test, Save and Finish setup; hold unready drafts.
Legacy keeps its Computer, credential and policy checks.
- Verify Stop from exact API ownership, audited intent,
provider-terminal proof, no Computer/process ownership and controller
retirement. Recheck immediate/deferred retirement; wake existing
recovery on durable evidence.
- Clear warnings when the actual stopped target later has authoritative
confirmation. Keep missing/other-run negatives and the existing
predicate.
- Publish safe status refresh after full-proof ACK. Retain bounded
native/physical cancellation history; add no poller or weaker proof
path.

## Verification

Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public
branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile
summary is 5/5; the fresh audit found zero unresolved threads and
complete pagination. The controlled workspace retry and required summary
passed. All 59 current-head contexts are terminal and clean. Prior
green-36 evidence keeps its identity.

[Current
CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731)
first failed workspaces-a 1/2: the CLI company-import/export beforeAll
30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI
files/4,714 tests and 285 other CLI tests passed (one skip). The cause
remains UNKNOWN. Child output records startup recovery complete, but
buffered CI prefixes do not establish its relation to the waiter
deadline; runtime-file/PID/last-health state was not captured. Both
unchanged local probes were RED. The second observed dependency
preflight still pending at the deadline; observer overhead was not
quantified. Neither proves the CI cause. After all functional jobs
finished, only the queued metadata summary was cancelled; the other 48
records stayed unchanged. Attempt 1 retains 46 successes, the original
failure, one skip and the cancelled summary. One same-head debug retry
was admitted at 18:42 UTC on November 3. Its workspace job passed at
19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33
files/286 tests, including company import/export. Test caching was
disabled. The new `ci / verify` summary passed; workflow attempt 2
completed SUCCESS. Required `ci / e2e` also passed. The final check at
19:53 UTC on November 3 found all 59 current-head contexts terminal and
clean: zero pending and zero failing. The 47 inherited completed/skipped
rows are not new executions. No code, assertion or timeout changed;
success does not explain the original failure or erase the local REDs.

CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents
`2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add
low-memory build/Rust task-memory. Scripts cohere; later matrix changes
only labels, with no newer debug-cache steps. Static audit found no
blocker. Supported API arguments omit the memory flag: no default
monitor/instruction changes. Flagged external launchers can change
Cursor instructions/digests and remain unqualified.

The merge added 41 master-identical paths, retained the green-36 inbox
hunk and found UI fixtures matching upstream. Startup/backup, recovery
projection, sleep diagnostics and binary backup change production.
Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL
templates unchanged, with sleep fences/single-flight accounting intact.
Exact precommit tree: 416/416 across eight server files, binary backup
1/1, server types PASS; repeated scopes are not summed.

The deferred-retirement P1 reproduced three real-DB/owned-process
failures; missing proof stayed unverified. Recovery passed 47 process
cases/33 proof negatives. Warning repair binds actual targets to
same-run proof. Validation passed 729 scoped UI cases, a separate
359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21
activity/privacy), types/tokens. Proof validators/predicate are
unchanged; no new poller or increased timeout.

The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures,
derived verify failure and one skip. Both shards shared three failures:
two successful mocks omitted the account guard’s required userId. The
two-line fixture repair changed no guard, assertion or timeout. Three
failures reproduced; 54 access/auth tests then passed, including
account-change/denial negatives, plus UI types/tokens.

The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9
failed, its verify aggregate failed and one selection job skipped. The
shard passed 575/576 cases. Its aborted-snapshot case expected nine
computes but saw eight: a final HTTP 200 coalesced while seven ordinary
reads had not proved the aborted compute settled. Earlier CI and the
unmodified local baseline passed with a miss; no local RED is claimed.
The repair awaits the snapshot, allows exactly one extra GET if
coalesced, then requires a miss and retains nine computes/all HTTP429
assertions. No production code, hook, loop, sleep or timeout changed.
Six inbox and seven cache/admission cases passed locally, plus server
types; 30 unrelated cases were filtered. Green 36 passed 576/576 without
taking the extra-GET branch; local runs also missed that schedule.
Current CI event merge 1ea9 now passed 576/576 and naturally exercised
the former failure: coalesced (count 2, 1ms), then exactly one fresh
miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions
intact. No instrumentation or retry was added.

Earlier compatibility passed scoped remote/inbox, routine, API-placement
and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path
adoption restored fleet routing without workflow overrides or added
skips. Each result retains its source identity.

Live qualification remains actual `9af1a5de42a2`: unchanged Artifact
grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three
provider test resources were deleted canonically with GET404;
post-delete download returned the exact 27 bytes. Histories stayed
stable (Artifact 121 and Stop 67 canonical identities), with 21 company
runs, zero active work and no accounting/cancellation/reservation debt
or integrity findings. Saved settings reopened; no-prompt Test/Linux
admission passed with `claude-sonnet-4-5`. No new Save is claimed.

The first Stop sampling case stays FAILED on a 10.373-second gap.
Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap
4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180
seconds. Initial grading rejected locator spelling before functional
checks; metadata changed only after same-file/inode/byte proof, with raw
evidence/oracles unchanged. Original Question/Plan/reconnect and
historical deleted-artifact results keep their identities.

Git trees verified 11 history-only conflicts and six earlier mainline
paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500
other mode/blob leaves match. Runner runtime implementations and six
observer/driver pins match 9af on that branch only; inherited
`drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src,
CI/latest-master runtime or image equality is claimed. Later static
audits through `8ab89f41e499` found no concrete API-only blocker. That
base adds completed-history sleep proofs and an MCP cleanup migration;
protected Stop/ACK owners are unchanged. These later changes are outside
this CI and have no live qualification here. Host duplicate-card and
later recovery/UI repairs retain separate proof, not deployed 9af
identity. Direct-key coverage requires absent aiConnection; API-only
placement skips inherited remote-workspace disk-full paths, and
config-read gates match their parent.

Other retained failures: stale fixture 132/133; E8 48/135 with 87
reset-hook failures; Mac 17 SQL failures/skips;
title/startup/custody/Stop/deletion unknown causes and bounded
recoveries; superseded warning/50 duplicate identities; pre-repair
ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121
shutdown tests with a failing broken-unsubscribe control; stable WSP
wrapper passed 128 after physical teardown failure. No full local-suite
pass is inferred. All paid attempts remain recorded; vendor dollar cost
is UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. Explicit values
are submitted and behavior checked live; unpriced provider dollar cost
stays UNKNOWN.

The placement exception applies only to validated native Cursor API
execution. Other local work remains denied under managed-only policy.
Missing/stale Stop or cleanup proof stays pending; damaged archives
remain failed. Explicit Legacy keeps its existing policy owners.

## Model Used

OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated
review. API ID/context window unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; retained
failures are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 13:55:39 -06:00

1121 lines
53 KiB
TypeScript

import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
import { museRunnerBroker } from "../muse-runner-broker.js";
import { assertEphemeralNativeLifecycle, readRemoteWorkspaceAuthority } from "../remote-workspaces.js";
import { measureRunTransaction } from "../execution-transaction-timing.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import {
ConfigurationIncompleteFailure,
type buildPaperclipWakePayload,
type createHeartbeatRunPreparation,
} from "./run-preparation.js";
import { dotRunnerBroker } from "../dot-runner-broker.js";
import { isAgentDirectoryCopy } from "../agent-directory-working-copies.js";
import { isConversation } from "../agent-conversations.js";
import { renewLegacyControllerLease } from "../legacy-controller-lease.js";
import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js";
import { supportsManagedNativeWarmSession } from "../native-runtime/native-warm-capability.js";
import {
getNativeReviewAssignment,
readNativeReviewAssignmentContext,
} from "../native-runtime/native-review-participant.js";
import { buildNativeReviewRequest } from "../native-runtime/native-review-prompt.js";
import type { CaseExecutionContext } from "../case-context.js";
import type { buildExecutionContinuation } from "../execution-continuation.js";
import type { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js";
import { createHash, randomUUID } from "node:crypto";
import { and, eq, inArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { agentHarnessType, CHAT_PROVIDERS } from "@paperclipai/shared";
import {
type agents,
completionContracts,
heartbeatRunEvents,
heartbeatRuns,
nativeRunFinalizations,
} from "@paperclipai/db";
import { getStartupTracer } from "../../instrumentation.js";
import { documentService } from "../documents.js";
import { managedAgentProfileService } from "../managed-agent-profiles.js";
import { remoteAgentProfileService } from "../remote-agent-profiles.js";
import {
buildNativeExecutionInput,
buildNativeExecutionWithCheckpoint,
buildNativeRuntimeContext,
ensureNativeCompletionContract,
findNativeSessionResumeRun,
isNativeSessionId,
isUnusedNativeSessionBootstrap,
isUnusedLegacyNativeRetryReplacement,
materializeNativeInteractionResponses,
nativeCompletionRequestsWithSources,
nativeCompletionSource,
nativeImmediateObjectiveSource,
nativeToolContractFingerprintForTarget,
prepareNativeSessionBootstrapPersistence,
prepareNativeWorkspaceSync,
type NativeRestartRecoveryClaim,
rebindNativeSessionCheckpoint,
type resolveHeartbeatNativeRuntimeMode,
} from "../native-runtime/index.js";
import {
assertAgentCoreProfileRecoveryBinding,
assertManagedProfileRecoveryBinding,
projectPaperclipRunnerTaskConfig,
resolvePaperclipRunnerNativeProviderInput,
} from "../native-runtime/provider-profile.js";
import { readRemoteCodexModelCliVersion } from "../native-runtime/codex-model-fallback.js";
import {
describeRunnerdNativeSessionBackend,
parseNativeExecutionInput,
type NativeExecutionInput,
} from "../../vendor/paperclip-runner/index.js";
import { PROVIDER_TRACE_MAX_BYTES } from "../provider-trace-store.js";
import type { AdapterRuntimeEvent } from "../../adapters/index.js";
import { parseObject } from "../../adapters/utils.js";
import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "../native-runtime/external-chat-question-response.js";
import {
materializeExternalChatQuestionResponseInput,
} from "../native-runtime/external-chat-question-response-input.js";
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js";
import { NATIVE_PREPARATION_FENCING_VERSION } from "../native-runtime/native-active-runs.js";
import type { issueService } from "../issues.js";
import { buildPlanReviewContext } from "../plan-review-context.js";
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
import { normalizePaperclipWakePayload, selectPaperclipTaskMarkdown } from "@paperclipai/adapter-utils/server-utils";
import type { ProviderResourceDisposition } from "../environment-runtime.js";
type Run = typeof heartbeatRuns.$inferSelect;
type Agent = typeof agents.$inferSelect;
type IssueContext = Awaited<ReturnType<ReturnType<typeof createHeartbeatRunPreparation>["getIssueExecutionContext"]>> | null;
type IssueRef = Parameters<typeof ensureNativeCompletionContract>[0]["issue"] & Parameters<typeof buildNativeExecutionInput>[0]["issue"];
export interface NativeLifecycleTelemetry {
provider: string;
harness: string;
lifecycleMode: "per_turn" | "warm";
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
}
export interface HeartbeatRuntimeSelectionInput {
run: Run;
agent: Agent;
nativeRuntimeResolution: ReturnType<typeof resolveHeartbeatNativeRuntimeMode>;
task: {
issueRef: IssueRef | null;
issueContext: IssueContext;
context: Record<string, unknown>;
executionContinuation: Awaited<ReturnType<typeof buildExecutionContinuation>> | null;
safeWakeComments: Parameters<typeof nativeCompletionRequestsWithSources>[0];
safeWakeCommentContext: { id: string; body: string } | null;
paperclipWakePayload: Awaited<ReturnType<typeof buildPaperclipWakePayload>>;
acceptedPlanContinuationWake: boolean;
};
session: {
persistedNativeExecutionInput: NativeExecutionInput | null;
persistedRunnerProfile: Record<string, unknown>;
taskSessionCredentialCompatible: boolean;
taskSessionDecodedParams: Record<string, unknown> | null;
taskSessionForRun: { lastRunId: string | null } | null;
isFailedChatRunRetry: boolean;
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
};
workspace: {
persistedExecutionWorkspace: { id: string; metadata?: Record<string, unknown> | null } | null;
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
activeEnvironmentLease: Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> | null;
isDotRun: boolean;
projectRepositoryPaths: string[];
onCheckpoint?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onCheckpoint"];
onPhase?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onPhase"];
};
config: {
runtimeConfig: Record<string, unknown>;
runtimeSkillEntries: Parameters<typeof buildNativeRuntimeContext>[0]["runtimeSkillEntries"];
runScopedMentionedSkillKeys: string[];
instructionCopy: Awaited<ReturnType<ReturnType<typeof agentInstructionWorkingCopyService>["prepare"]>>;
agentIdentity: { keyId: string } | undefined;
issueAssigneeOverrides: { adapterConfig: Record<string, unknown> | null } | null;
resolvedConfig: Record<string, unknown>;
managedAiRuntime: Awaited<ReturnType<typeof prepareManagedAiRuntime>> | undefined;
};
recovery: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise<void> };
trace: { providerTraceRequested: boolean; providerTraceCapture: { metadata: { id: string } } | null };
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
runtimeEnv: Record<string, string | undefined>;
beforeNativeRuntimeSelection?: (runId: string) => Promise<void>;
onAdapterEvent: (event: AdapterRuntimeEvent) => Promise<void>;
// Teardown must observe these updates even when a later preparation step throws.
onNativeLifecycleSelected: (telemetry: NativeLifecycleTelemetry) => void;
onProviderResourceDisposition: (disposition: ProviderResourceDisposition | undefined) => void;
onNativeOwnershipHeld: () => void;
stopControllerLease: () => void;
}
export type HeartbeatRuntimeSelectionResult = { selected: false } | {
selected: true;
nativeExecution: NativeExecutionInput | null;
nativeRunnerInstanceId: string | null;
getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>>;
};
export interface NativeSandboxLifecycle {
runnerProcess: "per_turn" | "warm";
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
failoverBackup: "verified";
}
export function resolveReusableSandboxLifecycle(input: {
lifecyclePolicy:
| { mode: "per_turn"; idleTimeoutMs: null }
| { mode: "warm"; idleTimeoutMs: number };
target: {
kind: "local" | "remote";
transport?: string;
reusableLeaseConfigured?: boolean;
effectiveCapabilities?: { reusableLeases: boolean } | null;
} | null;
}): NativeSandboxLifecycle | null {
if (input.target?.kind !== "remote" || input.target.transport !== "sandbox") {
return null;
}
const reusableLease =
input.target.reusableLeaseConfigured === true &&
input.target.effectiveCapabilities?.reusableLeases === true;
if (input.lifecyclePolicy.mode === "warm" && !reusableLease) {
throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease");
}
return {
runnerProcess: input.lifecyclePolicy.mode,
sandboxResource:
input.lifecyclePolicy.mode === "warm"
? "keep_running"
: reusableLease
? "stop_and_reuse"
: "destroy_after_turn",
failoverBackup: "verified",
};
}
export function resolveNativeSandboxLifecycle(input: {
adapterType: string;
lifecyclePolicy:
| { mode: "per_turn"; idleTimeoutMs: null }
| { mode: "warm"; idleTimeoutMs: number };
target: {
kind: "local" | "remote";
transport?: string;
reusableLeaseConfigured?: boolean;
effectiveCapabilities?: { reusableLeases: boolean } | null;
} | null;
}): NativeSandboxLifecycle | null {
if (
input.adapterType !== "paperclip_runner" ||
input.target?.kind !== "remote" ||
input.target.transport !== "sandbox"
)
return null;
return resolveReusableSandboxLifecycle(input);
}
export async function postNativeModelFallbackWarning(input: {
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
onEvent: (event: AdapterRuntimeEvent) => Promise<void>;
issueId: string;
runId: string;
requestedModel: string | null;
effectiveModel: string | null;
codexCliVersion: string;
}): Promise<void> {
const message = `Using ${input.effectiveModel} because the sandbox's Codex ${input.codexCliVersion} does not support ${input.requestedModel}. Work will continue with the compatible model. Update the sandbox's Codex CLI to use the requested model.`;
await input.onEvent({
eventType: "runner.model_fallback",
stream: "system",
level: "warn",
message,
payload: {
requestedModel: input.requestedModel,
effectiveModel: input.effectiveModel,
codexCliVersion: input.codexCliVersion,
},
});
await input.issuesSvc.addComment(input.issueId, message, { runId: input.runId }, {
authorType: "system",
presentation: {
kind: "system_notice",
tone: "warning",
title: `Using ${input.effectiveModel}`,
density: "compact",
detailsDefaultOpen: false,
},
});
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
/** Prepare and persist the selected runtime before the executor dispatches provider work. */
export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSelectionInput): Promise<HeartbeatRuntimeSelectionResult> {
const { run, agent, nativeRuntimeResolution, issuesSvc, runtimeEnv, beforeNativeRuntimeSelection, onAdapterEvent,
onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input;
const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext,
paperclipWakePayload, acceptedPlanContinuationWake } = input.task;
const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams,
taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace;
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
const runOptions = input.recovery;
const { providerTraceRequested, providerTraceCapture } = input.trace;
if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined);
let nativeExecution: NativeExecutionInput | null = null;
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
let nativeRunnerInstanceId: string | null = null;
let nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>> = null;
let providerResourceDispositionForRun: ProviderResourceDisposition | undefined;
if (nativeRuntimeResolution.kind === "native") {
if (!issueRef) {
throw new Error("native_runtime_ineligible: issue is required");
}
const nativeExecutionWorkspaceId =
persistedNativeExecutionInput?.provider.kind === "cursor_cloud"
// Older Cursor runs keep their admitted binding identity. Their task
// files and Computer are not part of the recovered API controller.
? persistedNativeExecutionInput.binding.executionWorkspaceId
: persistedExecutionWorkspace?.id ?? run.id;
const nativeReviewContext = readNativeReviewAssignmentContext(context);
const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, {
companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id,
contextSnapshot: nativeReviewContext,
})) : null;
if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available");
const nativeReviewRequest = nativeReview
? buildNativeReviewRequest({
title: nativeReview.interaction.title,
summary: nativeReview.interaction.summary,
payload: nativeReview.interaction.payload,
})
: null;
const persistedContract = run.completionContractId
? await measureRunPhase("runtime_selection.contract_read", () => db
.select()
.from(completionContracts)
.where(
and(
eq(completionContracts.id, run.completionContractId!),
eq(completionContracts.companyId, agent.companyId),
eq(completionContracts.issueId, issueRef.id),
),
)
.limit(1)
.then((rows) => rows[0] ?? null))
: null;
// Only a server-verified human resolution may supply a current answer
// reference. Tool/agent results and generated summaries stay evidence.
const currentHumanResponseId = !nativeReviewRequest
? executionContinuation?.humanResponses?.find(
(response) => response.id === executionContinuation.trigger.interactionId,
)?.id
: undefined;
const immediateCompletion = (() => {
if (nativeReviewRequest) return { requests: [nativeReviewRequest], sources: [null] };
const { requests, sources } = nativeCompletionRequestsWithSources(
safeWakeComments.length > 0
? safeWakeComments
: safeWakeCommentContext?.body
? [safeWakeCommentContext]
: [],
{
requiredFullWakeCommentCount:
paperclipWakePayload?.fallbackFetchNeeded === true &&
CHAT_PROVIDERS.some(
(provider) =>
provider ===
paperclipWakePayload.externalChatProvider,
) &&
Array.isArray(paperclipWakePayload.commentIds)
? paperclipWakePayload.commentIds.length
: undefined,
},
);
// Preserve every admitted pending chat request while also
// retaining newer user direction materialized by recovery.
// A file-only wake must not inherit an old task objective.
const latestComment =
executionContinuation?.messages.findLast(
(message) =>
message.authorType === "user" &&
!message.createdByRunId &&
!message.deleted &&
message.body.trim().length > 0,
);
const latestRequest = latestComment?.body;
if (
latestRequest &&
!requests.some(
(request) => request === latestRequest.trim(),
)
) {
requests.push(latestRequest.trim());
sources.push(nativeCompletionSource("comment", latestComment!.id, latestRequest));
}
return { requests: requests.length > 0 ? requests : undefined, sources };
})();
// Rebuilding a default contract is not a change in user direction.
// In particular, an upgraded checkpoint may have an intentionally
// authored contract and no continuation envelope yet.
const completionContract =
persistedContract && persistedNativeExecutionInput
? {
row: persistedContract,
contract: persistedContract.contractJson as never,
}
: await measureRunPhase("runtime_selection.contract_ensure", () => ensureNativeCompletionContract({
db,
companyId: agent.companyId,
issue: issueRef,
actorId: agent.id,
immediateRequest:
nativeReviewRequest ?? (currentHumanResponseId
? null
: executionContinuation?.objective ?? safeWakeCommentContext?.body ?? null),
// The ordinary initial objective is selected by the server-owned
// continuation envelope. Carry its explicit description source
// through the singular-request compatibility path; do not infer
// provenance for review requests, answers, or wake fallbacks.
immediateRequestSource: nativeImmediateObjectiveSource({
issueId: issueRef.id,
objectiveSource: executionContinuation?.objectiveSource,
excluded: Boolean(nativeReviewRequest || currentHumanResponseId),
}),
humanResponseId: currentHumanResponseId,
immediateRequests: immediateCompletion.requests,
immediateRequestSources: immediateCompletion.sources,
}));
const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString(
taskSessionDecodedParams?.sessionId,
);
// Compatibility for native retry rows created before same-run restart
// recovery existed. Only an entirely unused replacement row may
// inherit its source checkpoint; any process/provider evidence on the
// replacement makes the ownership ambiguous and therefore ineligible.
const legacyRetrySource =
run.retryOfRunId && !isFailedChatRunRetry
? await measureRunPhase("runtime_selection.resume", () => db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
runnerInstanceId: heartbeatRuns.runnerInstanceId,
nativeSessionId: heartbeatRuns.nativeSessionId,
runnerProfileJson: heartbeatRuns.runnerProfileJson,
runtimeMode: heartbeatRuns.runtimeMode,
status: heartbeatRuns.status,
})
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.id, run.retryOfRunId!),
eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.agentId, agent.id),
),
)
.limit(1)
.then((rows) => rows[0] ?? null))
: null;
const nativeBootstrapHasProviderEvidence =
legacyRetrySource ||
run.nativeSessionId ||
persistedNativeExecutionInput
? await measureRunPhase("runtime_selection.resume", () => db
.select({ id: heartbeatRunEvents.id })
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.runId, run.id),
inArray(heartbeatRunEvents.eventType, [
"harness.ready",
"session.started",
"session.resumed",
"session.updated",
"turn.started",
"provider.event",
"provider.rpc_result",
]),
),
)
.limit(1)
.then((rows) => rows.length > 0))
: false;
const compatibleLegacyRetrySource =
!managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({
replacement: run,
source: legacyRetrySource,
hasProviderEvents: nativeBootstrapHasProviderEvidence,
})
? legacyRetrySource
: null;
const legacyRetrySessionId =
compatibleLegacyRetrySource?.nativeSessionId;
const taskResumeRunId =
taskSessionForRun?.lastRunId &&
taskSessionForRun.lastRunId !== run.id &&
isNativeSessionId(taskNativeSessionId)
? taskSessionForRun.lastRunId
: null;
const resumableTaskSessionId = isDotRun ? null : taskResumeRunId
? taskNativeSessionId
: (legacyRetrySessionId ?? null);
const requestedNativeSessionId =
run.nativeSessionId ?? resumableTaskSessionId;
// A task-session lastRunId can lag a failed turn or point at an older
// normalized session. Find the newest exact-session authority instead.
// Rows that already acquired provider authority are progress barriers,
// even when they do not contain a usable checkpoint.
const previousNativeRun =
requestedNativeSessionId &&
isUnusedNativeSessionBootstrap(
run,
nativeBootstrapHasProviderEvidence,
)
? await measureRunPhase("runtime_selection.resume", () => findNativeSessionResumeRun(db, {
companyId: agent.companyId,
agentId: agent.id,
issueId: issueRef.id,
normalizedSessionId: requestedNativeSessionId,
currentRunId: run.id,
beforeCreatedAt: run.createdAt,
}))
: null;
nativeRunnerInstanceId =
previousNativeRun?.runnerInstanceId &&
previousNativeRun.nativeSessionId ===
(run.nativeSessionId ?? resumableTaskSessionId)
? previousNativeRun.runnerInstanceId
: (run.runnerInstanceId ?? randomUUID());
let nativeSessionId =
run.nativeSessionId ?? resumableTaskSessionId ?? randomUUID();
let nativeResumeCheckpoint: ReturnType<
typeof rebindNativeSessionCheckpoint
> = null;
const agentLifecyclePolicy =
parseObject(agent.adapterConfig).lifecycleMode === "warm"
? {
mode: "warm" as const,
idleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs(
parseObject(agent.adapterConfig).idleTimeoutMs,
),
}
: { mode: "per_turn" as const, idleTimeoutMs: null };
const environmentLifecyclePolicy =
executionTarget?.kind === "remote" &&
executionTarget.transport === "sandbox"
? (executionTarget.runnerLifecyclePolicy ?? null)
: null;
// Task providers own credential lifetime; process-backed managed homes retain their provider-specific cleanup policy.
const supportsManagedWarmSession = agent.adapterType === "paperclip_runner" &&
((executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" && !!executionTarget.runnerTask) ||
supportsManagedNativeWarmSession({ ...nativeRuntimeResolution.profile, acpxAgent: parseObject(agent.adapterConfig).acpxAgent }, managedAiRuntime?.attribution));
const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ??
(["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession
? { mode: "per_turn" as const, idleTimeoutMs: null }
: environmentLifecyclePolicy ?? agentLifecyclePolicy);
if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend)) {
assertEphemeralNativeLifecycle(effectiveLifecyclePolicy.mode);
}
if (
effectiveLifecyclePolicy.mode === "warm" &&
executionTarget?.kind === "remote" &&
executionTarget.transport === "sandbox" &&
(executionTarget.reusableLeaseConfigured !== true ||
executionTarget.effectiveCapabilities?.reusableLeases !== true)
) {
throw new Error("runner_warm_environment_requires_reusable_lease");
}
const persistedProfile = persistedRunnerProfile;
if (persistedNativeExecutionInput) {
nativeExecution = persistedNativeExecutionInput;
if (
nativeExecution.binding.companyId !== agent.companyId ||
nativeExecution.binding.runId !== run.id ||
nativeExecution.binding.issueId !== issueRef.id ||
nativeExecution.binding.agentId !== agent.id ||
nativeExecution.binding.executionWorkspaceId !==
nativeExecutionWorkspaceId ||
nativeExecution.completionContract.id !==
completionContract.row.id ||
nativeExecution.completionContract.sha256 !==
completionContract.row.canonicalSha256
)
throw new Error(
"native_execution_input_persisted_binding_mismatch",
);
if ("runtimeContext" in nativeExecution) {
const savedVersions = parseObject(run.contextSnapshot?.skillVersions);
const accessibleKeys = new Set(runtimeSkillEntries.filter(entry => entry.sourceStatus !== "missing").map(entry => entry.key));
if (nativeExecution.runtimeContext.skills.some(skill =>
Object.prototype.hasOwnProperty.call(savedVersions, skill.key) && !accessibleKeys.has(skill.key))) {
throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", {
configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" },
});
}
}
// Recover only the originally admitted request. A stored idle
// checkpoint can precede an already-started provider turn, so even
// apparent idleness is not authority to rewrite its contract.
// Newer user direction retains its separate durable wakeup cause.
// A failed pre-bootstrap attempt may have persisted its immutable
// input before discovering that lastRunId no longer names this
// session. Restore only an exactly compatible prior checkpoint;
// never rewrite the admitted input or skip current provider work.
if (
previousNativeRun &&
isUnusedNativeSessionBootstrap(
run,
nativeBootstrapHasProviderEvidence,
)
) {
nativeResumeCheckpoint = rebindNativeSessionCheckpoint({
previousRun: previousNativeRun,
currentExecution: nativeExecution,
executionTargetKind: executionTarget?.kind ?? "local",
});
}
if (nativeExecution.provider.kind === "claude_managed") {
const recoveryProfileId = nativeExecution.provider.managedProfile.profileId;
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(
db,
).requireQualified(
agent.companyId,
recoveryProfileId,
));
assertManagedProfileRecoveryBinding({
adapterConfig: agent.adapterConfig,
snapshot: nativeExecution.provider.managedProfile,
stored: recoveryProfile,
});
}
if (nativeExecution.provider.kind === "aws_agentcore") {
const recoveryProfileId = nativeExecution.provider.agentCoreProfile.profileId;
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(
db,
).requireQualified(
agent.companyId,
recoveryProfileId,
"aws_bedrock_agentcore_harness",
));
assertAgentCoreProfileRecoveryBinding({
snapshot: nativeExecution.provider.agentCoreProfile,
stored: recoveryProfile,
});
}
} else {
const interactionId = readNonEmptyString(context.interactionId);
const interactionResponses = context[
EXTERNAL_CHAT_QUESTION_RESPONSE_KEY
]
? await measureRunPhase("runtime_selection.interactions", () => materializeExternalChatQuestionResponseInput({
db,
binding: {
companyId: agent.companyId,
issueId: issueRef.id,
runId: run.id,
agentId: agent.id,
},
contextSnapshot: context,
}))
: await measureRunPhase("runtime_selection.interactions", () => materializeNativeInteractionResponses({
db,
companyId: agent.companyId,
issueId: issueRef.id,
runId: run.id,
agentId: agent.id,
interactionIds: Array.isArray(context.interactionIds)
? [
...new Set([
...(interactionId ? [interactionId] : []),
...context.interactionIds.filter(
(id): id is string => typeof id === "string",
),
]),
]
: interactionId
? [interactionId]
: [],
}));
const runnerAdapterConfig = parseObject(agent.adapterConfig);
const managedProfile =
nativeRuntimeResolution.profile.backend ===
"claude_managed_agents_api"
? await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(db).requireQualified(
agent.companyId,
readNonEmptyString(runnerAdapterConfig.managedProfileId) ??
"",
))
: null;
const agentCoreProfile =
nativeRuntimeResolution.profile.backend ===
"aws_agentcore_harness_api"
? await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(db).requireQualified(
agent.companyId,
readNonEmptyString(
runnerAdapterConfig.agentCoreProfileId,
) ?? "",
"aws_bedrock_agentcore_harness",
))
: null;
if (managedProfile) {
const rawApiKeyBinding = parseObject(
runnerAdapterConfig.env,
).ANTHROPIC_API_KEY;
const boundSecretId =
typeof rawApiKeyBinding === "object" &&
rawApiKeyBinding !== null
? readNonEmptyString(
(rawApiKeyBinding as Record<string, unknown>).secretId,
)
: null;
if (boundSecretId !== managedProfile.apiKeySecretId) {
throw new ConfigurationIncompleteFailure(
"configuration incomplete: Claude Managed profile API key is not bound at env.ANTHROPIC_API_KEY",
{
configurationIncomplete: {
reason: "managed_agent_profile_secret_binding_mismatch",
companyId: agent.companyId,
agentId: agent.id,
profileId: managedProfile.id,
requiredEnvKeys: ["ANTHROPIC_API_KEY"],
},
},
);
}
}
const executionMode =
issueRef.workMode === "planning" && !isConversation(issueContext) && !acceptedPlanContinuationWake
? ("plan" as const)
: ("default" as const);
const pinnedPlan =
executionMode === "plan"
? await documentService(db).getIssueDocumentByKey(
issueRef.id,
"plan",
)
: null;
const pinnedReviewContext =
executionMode === "plan"
? await buildPlanReviewContext({
db,
companyId: agent.companyId,
issueId: issueRef.id,
issueWorkMode: issueRef.workMode,
interactionId: readNonEmptyString(context.interactionId),
})
: null;
const pinnedPlanMarkdown = pinnedPlan?.body ?? "";
const museBinding = nativeRuntimeResolution.profile.backend === "muse_external"
? await museRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig).museBindingId ?? "")) : undefined;
const dotBinding = ["openai_dot_mcp", "grok_bot_mcp"].includes(nativeRuntimeResolution.profile.backend)
? await dotRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig)[nativeRuntimeResolution.profile.backend === "grok_bot_mcp" ? "externalBindingId" : "dotBindingId"] ?? "")) : undefined;
const nativeRuntimeContext = await buildNativeRuntimeContext({
db,
agent,
runId: run.id,
runtimeConfig,
runtimeSkillEntries,
instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined,
});
getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff;
const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "cursor_cloud_api"
|| nativeRuntimeResolution.profile.backend === "codex_app_server"
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime" && ["claude_local", "grok_local"].includes(agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig))))
|| nativeRuntimeResolution.profile.backend === "opencode_server"
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime"
&& parseObject(agent.adapterConfig).acpxAgent === "copilot")
? projectPaperclipRunnerTaskConfig(
nativeRuntimeResolution.profile.backend,
agent.adapterConfig,
issueAssigneeOverrides?.adapterConfig,
managedAiRuntime ? readNonEmptyString(resolvedConfig.model) ?? undefined : undefined,
)
: agent.adapterConfig;
const requestedNativeProvider = resolvePaperclipRunnerNativeProviderInput({
companyId: agent.companyId,
backend: nativeRuntimeResolution.profile.backend,
adapterConfig: nativeProviderConfig, managedProfile, agentCoreProfile, dotBinding, museBinding,
});
const codexCliVersion = agent.adapterType === "paperclip_runner" && requestedNativeProvider.provider === "codex"
? await readRemoteCodexModelCliVersion({
model: requestedNativeProvider.model,
target: executionTarget,
remoteCodexPath: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH,
remoteCodexNpmSpec: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC,
}) : null;
const nativeProvider = codexCliVersion
? resolvePaperclipRunnerNativeProviderInput({
companyId: agent.companyId,
backend: nativeRuntimeResolution.profile.backend,
adapterConfig: nativeProviderConfig, codexCliVersion, managedProfile, agentCoreProfile, dotBinding, museBinding,
}) : requestedNativeProvider;
if (nativeProvider.model !== requestedNativeProvider.model) {
await postNativeModelFallbackWarning({
issuesSvc, onEvent: onAdapterEvent, issueId: issueRef.id, runId: run.id,
requestedModel: requestedNativeProvider.model, effectiveModel: nativeProvider.model,
codexCliVersion: codexCliVersion!,
});
}
const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) =>
buildNativeExecutionInput({
agentKeyId: agentIdentity?.keyId,
companyId: agent.companyId,
runId: run.id,
issue: nativeReviewRequest ? { ...issueRef, title: `Review: ${issueRef.title}`, description: nativeReviewRequest } : issueRef,
taskPrompt: [
nativeReviewRequest ?? readNonEmptyString(
selectPaperclipTaskMarkdown(context, {
resumedSession: false,
includeCommunicationGuidance: false,
}),
) ??
`# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`,
!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) && projectRepositoryPaths.length > 0
? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}`
: null,
].filter(Boolean).join("\n\n"),
initialCommunicationGuidance: nativeReviewRequest ? null : readNonEmptyString(context.paperclipTaskCommunicationGuidance),
wakePayload: context.paperclipWake,
caseContext: nativeReviewRequest ? null : context.paperclipCase as CaseExecutionContext | undefined,
turnContext: context.paperclipTurnContext,
githubInstructionSkillKeys: runScopedMentionedSkillKeys,
resumedSession,
previousTurn: (() => {
if (!previousNativeRun || nativeReviewRequest) return null;
try {
const previousInput = parseNativeExecutionInput(parseObject(previousNativeRun.runnerProfileJson).nativeExecutionInput);
const previousTask = previousInput.task;
const previousWake = normalizePaperclipWakePayload(parseObject(previousNativeRun.contextSnapshot).paperclipWake);
const githubReviewConfigurationKeys = previousWake?.githubReviewConfigurations.map((config) => config.key);
if (paperclipWakePayload?.externalChatProvider || previousWake?.externalChatProvider) {
// External native inputs use a neutral task title. Compare
// the saved canonical brief so old provider text is not
// repeated as a change, while genuine edits still arrive.
const savedIssue = parseObject(parseObject(previousNativeRun.contextSnapshot).paperclipIssue);
if (savedIssue.id !== issueRef.id || typeof savedIssue.title !== "string" ||
(savedIssue.description !== null && typeof savedIssue.description !== "string")) return null;
return { runId: previousNativeRun.id, task: { title: savedIssue.title, description: savedIssue.description },
githubReviewConfigurationKeys, nativeExecutionInput: previousInput,
};
}
return { runId: previousNativeRun.id, task: previousTask, githubReviewConfigurationKeys, nativeExecutionInput: previousInput };
} catch {
// An invalid prior snapshot must use the fresh bootstrap.
return null;
}
})(),
conversationMode: context.conversationMode === true,
agentId: agent.id,
workspace: {
// Projectless paperclip_runner tasks still have a resolved local cwd. Bind that
// transient workspace to the run id so the native input remains durable and replayable
// without fabricating a project-scoped execution_workspaces row.
id: nativeExecutionWorkspaceId,
cwd: executionWorkspace.cwd,
repoUrl: executionWorkspace.repoUrl,
repoRef: executionWorkspace.repoRef,
branchName: executionWorkspace.branchName,
},
normalizedSessionId,
executionMode,
planningContext:
executionMode === "plan"
? {
documentId: pinnedPlan?.id ?? null,
baseRevisionId:
pinnedPlan?.latestRevisionId ?? null,
baseRevisionNumber:
pinnedPlan?.latestRevisionNumber ?? 0,
markdown: pinnedPlanMarkdown,
sha256: createHash("sha256")
.update(pinnedPlanMarkdown)
.digest("hex"),
reviewContext: pinnedReviewContext
? (structuredClone(
pinnedReviewContext,
) as unknown as Record<string, unknown>)
: {},
}
: null,
...nativeProvider,
lifecyclePolicy: effectiveLifecyclePolicy,
interactionResponses,
completionContract: {
id: completionContract.row.id,
sha256: completionContract.row.canonicalSha256,
schemaVersion: completionContract.row.schemaVersion,
contract: completionContract.contract,
sources: "sources" in completionContract ? completionContract.sources : undefined,
},
runtimeContext: nativeRuntimeContext,
});
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
})));
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
previousRun: previousNativeRun,
normalizedSessionId: nativeSessionId,
executionTargetKind: executionTarget?.kind ?? "local",
toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true,
refreshTools: context.refreshTools === true,
buildExecution,
});
nativeExecution = nativeExecutionWithCheckpoint.execution;
nativeResumeCheckpoint = nativeExecutionWithCheckpoint.checkpoint;
if (
nativeSessionId !==
nativeExecutionWithCheckpoint.normalizedSessionId
) {
nativeRunnerInstanceId = randomUUID();
}
nativeSessionId = nativeExecutionWithCheckpoint.normalizedSessionId;
}
if (nativeResumeCheckpoint && nativeExecution.provider.kind === "acpx"
&& nativeExecution.provider.agent === "antigravity") {
nativeResumeCheckpoint = { ...nativeResumeCheckpoint, providerRecoveryPolicy: "same_session_only" };
}
const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({
adapterType: agent.adapterType,
lifecyclePolicy: nativeExecution.session.lifecyclePolicy,
target: executionTarget,
});
if (nativeSandboxLifecycle) {
onNativeLifecycleSelected({
provider: nativeExecution.provider.kind,
harness: nativeExecution.session.driverKind,
lifecycleMode: nativeExecution.session.lifecyclePolicy.mode,
sandboxResource: nativeSandboxLifecycle.sandboxResource,
});
const selectedLifecycleSpan = getStartupTracer(
"paperclip.environment-lifecycle",
).startSpan("sandbox.lifecycle.selected", {
attributes: {
"paperclip.native.span.provider": nativeExecution.provider.kind,
"paperclip.native.span.harness":
nativeExecution.session.driverKind,
"paperclip.native.span.lifecycle_mode":
nativeExecution.session.lifecyclePolicy.mode,
"paperclip.native.span.sandbox_resource":
nativeSandboxLifecycle.sandboxResource,
"paperclip.native.span.outcome": "selected",
"paperclip.native.span.bytes_transferred": 0,
},
});
selectedLifecycleSpan.end();
}
providerResourceDispositionForRun =
nativeSandboxLifecycle?.sandboxResource === "keep_running"
? "keep_running"
: nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse"
? "stop_and_retain"
: nativeSandboxLifecycle?.sandboxResource ===
"destroy_after_turn"
? "destroy"
: undefined;
onProviderResourceDisposition(providerResourceDispositionForRun);
await beforeNativeRuntimeSelection?.(run.id);
const nativeSelected = await measureRunTransaction(db, "runtime_selection", async (tx) => {
const lockedRun = await measureRunPhase("runtime_selection.row_lock", () => tx
.select()
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, run.id))
.for("update")
.limit(1)
.then((rows) => rows[0] ?? null));
if (!lockedRun) throw new Error("native_runtime_run_missing");
// Cancellation and runtime selection serialize on this row. A
// stopped preparation must never create a new native coordinator.
const stopRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
&& runOptions.nativeRestartRecovery.runId === lockedRun.id
&& runOptions.nativeRestartRecovery.leaseOwner === runOptions.nativeLeaseOwner
&& await readClaimedNativeStopRecovery(tx as unknown as Db, lockedRun, runOptions.nativeLeaseOwner);
if (lockedRun.status !== "running" || (lockedRun.resultJson?.startupCancellation && !stopRecovery)) return false;
if (lockedRun.runtimeMode === "legacy" && lockedRun.controllerBootId &&
!(await renewLegacyControllerLease(tx as unknown as Db, lockedRun))) {
onNativeOwnershipHeld();
return false;
}
if (
lockedRun.runtimeModeResolvedAt &&
lockedRun.runtimeMode !== "native"
) {
throw new Error("native_runtime_mode_conflict");
}
const lockedProfile = parseObject(lockedRun.runnerProfileJson);
const persistedNativeSessionId =
await measureRunPhase("runtime_selection.bootstrap", () => prepareNativeSessionBootstrapPersistence(tx, {
run: lockedRun,
selectedSessionId: nativeSessionId,
execution: nativeExecution!,
restoringCheckpoint: nativeResumeCheckpoint !== null,
}));
await measureRunPhase("runtime_selection.run_write", () => tx
.update(heartbeatRuns)
.set({
runtimeMode: "native",
controllerBootId: null,
controllerLeaseExpiresAt: null,
executionStage: null,
runtimeModeResolverVersion:
lockedRun.runtimeModeResolverVersion ??
nativeRuntimeResolution.resolverVersion,
runtimeModeReason:
lockedRun.runtimeModeReason ?? nativeRuntimeResolution.reason,
runtimeModeResolvedAt:
lockedRun.runtimeModeResolvedAt ?? new Date(),
runnerProfileJson: {
...nativeRuntimeResolution.profile,
...lockedProfile,
...(nativeExecution?.provider.kind === "cursor_cloud" ? { executionPlacement: resolveProviderApiPlacement({ agent, run }) } : {}),
nativePreparationFencingVersion: !lockedRun.runnerInstanceId && !lockedRun.nativeSessionId
? NATIVE_PREPARATION_FENCING_VERSION
: lockedProfile.nativePreparationFencingVersion,
...(lockedProfile.nativeExecutionInput
? {}
: { recoveryEventInventoryVersion: 1 }),
...(providerTraceRequested
? {
providerTrace: {
mode: "raw",
traceId: providerTraceCapture?.metadata.id ?? null,
maxBytes: PROVIDER_TRACE_MAX_BYTES,
},
}
: {}),
nativeExecutionInput:
lockedProfile.nativeExecutionInput ?? nativeExecution,
nativeResourceDisposition: providerResourceDispositionForRun,
nativeToolContractFingerprint:
nativeToolContractFingerprintForTarget(
executionTarget?.kind ?? "local",
nativeExecution!.provider.kind,
),
...(lockedProfile.sessionCheckpoint != null
? { sessionCheckpoint: lockedProfile.sessionCheckpoint }
: nativeResumeCheckpoint
? {
sessionCheckpoint:
nativeResumeCheckpoint as unknown as Record<
string,
unknown
>,
}
: {}),
},
runnerInstanceId:
previousNativeRun?.runnerInstanceId &&
persistedNativeSessionId === previousNativeRun.nativeSessionId
? previousNativeRun.runnerInstanceId
: lockedRun.nativeSessionId !== persistedNativeSessionId
? nativeRunnerInstanceId
: (lockedRun.runnerInstanceId ?? nativeRunnerInstanceId),
nativeSessionId: persistedNativeSessionId,
processPid:
lockedRun.processPid ??
(previousNativeRun?.nativeSessionId === nativeSessionId
? previousNativeRun.processPid
: null),
processGroupId:
lockedRun.processGroupId ??
(previousNativeRun?.nativeSessionId === nativeSessionId
? previousNativeRun.processGroupId
: null),
processStartedAt:
lockedRun.processStartedAt ??
(previousNativeRun?.nativeSessionId === nativeSessionId
? previousNativeRun.processStartedAt
: null),
nativeIssueId: lockedRun.nativeIssueId ?? issueRef.id,
driverKind:
lockedRun.driverKind ??
nativeExecution?.session.driverKind ??
"codex_app_server",
driverVersion: lockedRun.driverVersion ?? "phase6-v1",
completionContractId:
lockedRun.completionContractId ?? completionContract.row.id,
completionContractSha256:
lockedRun.completionContractSha256 ??
completionContract.row.canonicalSha256,
nativePhase: lockedRun.nativePhase ?? "observed",
nativePhaseUpdatedAt:
lockedRun.nativePhaseUpdatedAt ?? new Date(),
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, run.id)));
await measureRunPhase("runtime_selection.finalization_write", () => tx
.insert(nativeRunFinalizations)
.values({
runId: run.id,
companyId: agent.companyId,
issueId: issueRef.id,
phase: "observed",
})
.onConflictDoNothing());
return true;
});
if (!nativeSelected) return { selected: false };
stopControllerLease();
if (nativeExecution?.provider.kind !== "cursor_cloud" && !activeEnvironmentLease) throw new Error("native_environment_lease_unavailable");
nativeWorkspaceSync = nativeExecution?.provider.kind === "cursor_cloud" || readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata) ? null
: await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({
db,
runId: run.id,
companyId: agent.companyId,
workspaceId: nativeExecutionWorkspaceId,
workspaceLocalDir: executionWorkspace.cwd,
target: executionTarget,
lease: activeEnvironmentLease!.lease,
restartRecovery: runOptions.nativeRestartRecovery,
sameRunRecovery: Boolean(runOptions.nativeLeaseOwner),
resourceDisposition: providerResourceDispositionForRun,
workspaceGitIndexRefresh: nativeExecution?.provider.kind === "opencode",
onCheckpoint: input.workspace.onCheckpoint,
onPhase: input.workspace.onPhase,
onEvent: onAdapterEvent,
}));
} else {
const legacyWarmLifecycle =
executionTarget?.kind === "remote" &&
executionTarget.transport === "sandbox" &&
executionTarget.runnerLifecyclePolicy?.mode === "warm"
? resolveReusableSandboxLifecycle({
lifecyclePolicy: executionTarget.runnerLifecyclePolicy,
target: executionTarget,
})
: null;
if (legacyWarmLifecycle?.sandboxResource === "keep_running") {
providerResourceDispositionForRun = "keep_running";
onProviderResourceDisposition(providerResourceDispositionForRun);
}
await measureRunPhase("runtime_selection.run_write", () => db
.update(heartbeatRuns)
.set({
runtimeMode: "legacy",
runtimeModeResolverVersion:
nativeRuntimeResolution.resolverVersion,
runtimeModeReason: nativeRuntimeResolution.reason,
runtimeModeResolvedAt: run.runtimeModeResolvedAt ?? new Date(),
// Preserve server-owned admission and dispatch evidence on this
// row; never copy another run's execution profile.
runnerProfileJson: sql`(case when ${heartbeatRuns.runnerProfileJson} ? ${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}
then jsonb_build_object(${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}::text, ${heartbeatRuns.runnerProfileJson}->${CHAT_CONTROL_RECOVERY_ADMISSION_KEY})
else '{}'::jsonb end)
|| (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch'
then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch')
else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, run.id)));
}
return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync };
}