mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-05 12:29:37 +01:00
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1121 lines
53 KiB
TypeScript
1121 lines
53 KiB
TypeScript
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
|
|
import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
|
|
import { museRunnerBroker } from "../muse-runner-broker.js";
|
|
import { assertEphemeralNativeLifecycle, readRemoteWorkspaceAuthority } from "../remote-workspaces.js";
|
|
import { measureRunTransaction } from "../execution-transaction-timing.js";
|
|
import { measureRunPhase } from "../execution-preparation-timing.js";
|
|
import {
|
|
ConfigurationIncompleteFailure,
|
|
type buildPaperclipWakePayload,
|
|
type createHeartbeatRunPreparation,
|
|
} from "./run-preparation.js";
|
|
import { dotRunnerBroker } from "../dot-runner-broker.js";
|
|
import { isAgentDirectoryCopy } from "../agent-directory-working-copies.js";
|
|
import { isConversation } from "../agent-conversations.js";
|
|
import { renewLegacyControllerLease } from "../legacy-controller-lease.js";
|
|
import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js";
|
|
import { supportsManagedNativeWarmSession } from "../native-runtime/native-warm-capability.js";
|
|
import {
|
|
getNativeReviewAssignment,
|
|
readNativeReviewAssignmentContext,
|
|
} from "../native-runtime/native-review-participant.js";
|
|
import { buildNativeReviewRequest } from "../native-runtime/native-review-prompt.js";
|
|
import type { CaseExecutionContext } from "../case-context.js";
|
|
import type { buildExecutionContinuation } from "../execution-continuation.js";
|
|
import type { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { agentHarnessType, CHAT_PROVIDERS } from "@paperclipai/shared";
|
|
import {
|
|
type agents,
|
|
completionContracts,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
nativeRunFinalizations,
|
|
} from "@paperclipai/db";
|
|
import { getStartupTracer } from "../../instrumentation.js";
|
|
import { documentService } from "../documents.js";
|
|
import { managedAgentProfileService } from "../managed-agent-profiles.js";
|
|
import { remoteAgentProfileService } from "../remote-agent-profiles.js";
|
|
import {
|
|
buildNativeExecutionInput,
|
|
buildNativeExecutionWithCheckpoint,
|
|
buildNativeRuntimeContext,
|
|
ensureNativeCompletionContract,
|
|
findNativeSessionResumeRun,
|
|
isNativeSessionId,
|
|
isUnusedNativeSessionBootstrap,
|
|
isUnusedLegacyNativeRetryReplacement,
|
|
materializeNativeInteractionResponses,
|
|
nativeCompletionRequestsWithSources,
|
|
nativeCompletionSource,
|
|
nativeImmediateObjectiveSource,
|
|
nativeToolContractFingerprintForTarget,
|
|
prepareNativeSessionBootstrapPersistence,
|
|
prepareNativeWorkspaceSync,
|
|
type NativeRestartRecoveryClaim,
|
|
rebindNativeSessionCheckpoint,
|
|
type resolveHeartbeatNativeRuntimeMode,
|
|
} from "../native-runtime/index.js";
|
|
import {
|
|
assertAgentCoreProfileRecoveryBinding,
|
|
assertManagedProfileRecoveryBinding,
|
|
projectPaperclipRunnerTaskConfig,
|
|
resolvePaperclipRunnerNativeProviderInput,
|
|
} from "../native-runtime/provider-profile.js";
|
|
import { readRemoteCodexModelCliVersion } from "../native-runtime/codex-model-fallback.js";
|
|
import {
|
|
describeRunnerdNativeSessionBackend,
|
|
parseNativeExecutionInput,
|
|
type NativeExecutionInput,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import { PROVIDER_TRACE_MAX_BYTES } from "../provider-trace-store.js";
|
|
import type { AdapterRuntimeEvent } from "../../adapters/index.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "../native-runtime/external-chat-question-response.js";
|
|
import {
|
|
materializeExternalChatQuestionResponseInput,
|
|
} from "../native-runtime/external-chat-question-response-input.js";
|
|
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js";
|
|
import { NATIVE_PREPARATION_FENCING_VERSION } from "../native-runtime/native-active-runs.js";
|
|
import type { issueService } from "../issues.js";
|
|
import { buildPlanReviewContext } from "../plan-review-context.js";
|
|
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
|
|
import { normalizePaperclipWakePayload, selectPaperclipTaskMarkdown } from "@paperclipai/adapter-utils/server-utils";
|
|
import type { ProviderResourceDisposition } from "../environment-runtime.js";
|
|
|
|
type Run = typeof heartbeatRuns.$inferSelect;
|
|
type Agent = typeof agents.$inferSelect;
|
|
type IssueContext = Awaited<ReturnType<ReturnType<typeof createHeartbeatRunPreparation>["getIssueExecutionContext"]>> | null;
|
|
type IssueRef = Parameters<typeof ensureNativeCompletionContract>[0]["issue"] & Parameters<typeof buildNativeExecutionInput>[0]["issue"];
|
|
|
|
export interface NativeLifecycleTelemetry {
|
|
provider: string;
|
|
harness: string;
|
|
lifecycleMode: "per_turn" | "warm";
|
|
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
|
}
|
|
|
|
export interface HeartbeatRuntimeSelectionInput {
|
|
run: Run;
|
|
agent: Agent;
|
|
nativeRuntimeResolution: ReturnType<typeof resolveHeartbeatNativeRuntimeMode>;
|
|
task: {
|
|
issueRef: IssueRef | null;
|
|
issueContext: IssueContext;
|
|
context: Record<string, unknown>;
|
|
executionContinuation: Awaited<ReturnType<typeof buildExecutionContinuation>> | null;
|
|
safeWakeComments: Parameters<typeof nativeCompletionRequestsWithSources>[0];
|
|
safeWakeCommentContext: { id: string; body: string } | null;
|
|
paperclipWakePayload: Awaited<ReturnType<typeof buildPaperclipWakePayload>>;
|
|
acceptedPlanContinuationWake: boolean;
|
|
};
|
|
session: {
|
|
persistedNativeExecutionInput: NativeExecutionInput | null;
|
|
persistedRunnerProfile: Record<string, unknown>;
|
|
taskSessionCredentialCompatible: boolean;
|
|
taskSessionDecodedParams: Record<string, unknown> | null;
|
|
taskSessionForRun: { lastRunId: string | null } | null;
|
|
isFailedChatRunRetry: boolean;
|
|
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
|
};
|
|
workspace: {
|
|
persistedExecutionWorkspace: { id: string; metadata?: Record<string, unknown> | null } | null;
|
|
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
|
|
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
|
|
activeEnvironmentLease: Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease"> | null;
|
|
isDotRun: boolean;
|
|
projectRepositoryPaths: string[];
|
|
onCheckpoint?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onCheckpoint"];
|
|
onPhase?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onPhase"];
|
|
};
|
|
config: {
|
|
runtimeConfig: Record<string, unknown>;
|
|
runtimeSkillEntries: Parameters<typeof buildNativeRuntimeContext>[0]["runtimeSkillEntries"];
|
|
runScopedMentionedSkillKeys: string[];
|
|
instructionCopy: Awaited<ReturnType<ReturnType<typeof agentInstructionWorkingCopyService>["prepare"]>>;
|
|
agentIdentity: { keyId: string } | undefined;
|
|
issueAssigneeOverrides: { adapterConfig: Record<string, unknown> | null } | null;
|
|
resolvedConfig: Record<string, unknown>;
|
|
managedAiRuntime: Awaited<ReturnType<typeof prepareManagedAiRuntime>> | undefined;
|
|
};
|
|
recovery: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim; onNativeExecutionLeaseAcquired?: () => Promise<void> };
|
|
trace: { providerTraceRequested: boolean; providerTraceCapture: { metadata: { id: string } } | null };
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
|
|
runtimeEnv: Record<string, string | undefined>;
|
|
beforeNativeRuntimeSelection?: (runId: string) => Promise<void>;
|
|
onAdapterEvent: (event: AdapterRuntimeEvent) => Promise<void>;
|
|
// Teardown must observe these updates even when a later preparation step throws.
|
|
onNativeLifecycleSelected: (telemetry: NativeLifecycleTelemetry) => void;
|
|
onProviderResourceDisposition: (disposition: ProviderResourceDisposition | undefined) => void;
|
|
onNativeOwnershipHeld: () => void;
|
|
stopControllerLease: () => void;
|
|
}
|
|
|
|
export type HeartbeatRuntimeSelectionResult = { selected: false } | {
|
|
selected: true;
|
|
nativeExecution: NativeExecutionInput | null;
|
|
nativeRunnerInstanceId: string | null;
|
|
getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
|
nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>>;
|
|
};
|
|
|
|
export interface NativeSandboxLifecycle {
|
|
runnerProcess: "per_turn" | "warm";
|
|
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
|
failoverBackup: "verified";
|
|
}
|
|
|
|
export function resolveReusableSandboxLifecycle(input: {
|
|
lifecyclePolicy:
|
|
| { mode: "per_turn"; idleTimeoutMs: null }
|
|
| { mode: "warm"; idleTimeoutMs: number };
|
|
target: {
|
|
kind: "local" | "remote";
|
|
transport?: string;
|
|
reusableLeaseConfigured?: boolean;
|
|
effectiveCapabilities?: { reusableLeases: boolean } | null;
|
|
} | null;
|
|
}): NativeSandboxLifecycle | null {
|
|
if (input.target?.kind !== "remote" || input.target.transport !== "sandbox") {
|
|
return null;
|
|
}
|
|
const reusableLease =
|
|
input.target.reusableLeaseConfigured === true &&
|
|
input.target.effectiveCapabilities?.reusableLeases === true;
|
|
if (input.lifecyclePolicy.mode === "warm" && !reusableLease) {
|
|
throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease");
|
|
}
|
|
return {
|
|
runnerProcess: input.lifecyclePolicy.mode,
|
|
sandboxResource:
|
|
input.lifecyclePolicy.mode === "warm"
|
|
? "keep_running"
|
|
: reusableLease
|
|
? "stop_and_reuse"
|
|
: "destroy_after_turn",
|
|
failoverBackup: "verified",
|
|
};
|
|
}
|
|
|
|
export function resolveNativeSandboxLifecycle(input: {
|
|
adapterType: string;
|
|
lifecyclePolicy:
|
|
| { mode: "per_turn"; idleTimeoutMs: null }
|
|
| { mode: "warm"; idleTimeoutMs: number };
|
|
target: {
|
|
kind: "local" | "remote";
|
|
transport?: string;
|
|
reusableLeaseConfigured?: boolean;
|
|
effectiveCapabilities?: { reusableLeases: boolean } | null;
|
|
} | null;
|
|
}): NativeSandboxLifecycle | null {
|
|
if (
|
|
input.adapterType !== "paperclip_runner" ||
|
|
input.target?.kind !== "remote" ||
|
|
input.target.transport !== "sandbox"
|
|
)
|
|
return null;
|
|
return resolveReusableSandboxLifecycle(input);
|
|
}
|
|
|
|
export async function postNativeModelFallbackWarning(input: {
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
|
|
onEvent: (event: AdapterRuntimeEvent) => Promise<void>;
|
|
issueId: string;
|
|
runId: string;
|
|
requestedModel: string | null;
|
|
effectiveModel: string | null;
|
|
codexCliVersion: string;
|
|
}): Promise<void> {
|
|
const message = `Using ${input.effectiveModel} because the sandbox's Codex ${input.codexCliVersion} does not support ${input.requestedModel}. Work will continue with the compatible model. Update the sandbox's Codex CLI to use the requested model.`;
|
|
await input.onEvent({
|
|
eventType: "runner.model_fallback",
|
|
stream: "system",
|
|
level: "warn",
|
|
message,
|
|
payload: {
|
|
requestedModel: input.requestedModel,
|
|
effectiveModel: input.effectiveModel,
|
|
codexCliVersion: input.codexCliVersion,
|
|
},
|
|
});
|
|
await input.issuesSvc.addComment(input.issueId, message, { runId: input.runId }, {
|
|
authorType: "system",
|
|
presentation: {
|
|
kind: "system_notice",
|
|
tone: "warning",
|
|
title: `Using ${input.effectiveModel}`,
|
|
density: "compact",
|
|
detailsDefaultOpen: false,
|
|
},
|
|
});
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
/** Prepare and persist the selected runtime before the executor dispatches provider work. */
|
|
export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSelectionInput): Promise<HeartbeatRuntimeSelectionResult> {
|
|
const { run, agent, nativeRuntimeResolution, issuesSvc, runtimeEnv, beforeNativeRuntimeSelection, onAdapterEvent,
|
|
onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input;
|
|
const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext,
|
|
paperclipWakePayload, acceptedPlanContinuationWake } = input.task;
|
|
const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams,
|
|
taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
|
|
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace;
|
|
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
|
|
const runOptions = input.recovery;
|
|
const { providerTraceRequested, providerTraceCapture } = input.trace;
|
|
if (nativeRuntimeResolution.kind !== "native" && !isDotRun) assertEphemeralNativeLifecycle(undefined);
|
|
let nativeExecution: NativeExecutionInput | null = null;
|
|
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
|
let nativeRunnerInstanceId: string | null = null;
|
|
let nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>> = null;
|
|
let providerResourceDispositionForRun: ProviderResourceDisposition | undefined;
|
|
if (nativeRuntimeResolution.kind === "native") {
|
|
if (!issueRef) {
|
|
throw new Error("native_runtime_ineligible: issue is required");
|
|
}
|
|
const nativeExecutionWorkspaceId =
|
|
persistedNativeExecutionInput?.provider.kind === "cursor_cloud"
|
|
// Older Cursor runs keep their admitted binding identity. Their task
|
|
// files and Computer are not part of the recovered API controller.
|
|
? persistedNativeExecutionInput.binding.executionWorkspaceId
|
|
: persistedExecutionWorkspace?.id ?? run.id;
|
|
const nativeReviewContext = readNativeReviewAssignmentContext(context);
|
|
const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, {
|
|
companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id,
|
|
contextSnapshot: nativeReviewContext,
|
|
})) : null;
|
|
if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available");
|
|
const nativeReviewRequest = nativeReview
|
|
? buildNativeReviewRequest({
|
|
title: nativeReview.interaction.title,
|
|
summary: nativeReview.interaction.summary,
|
|
payload: nativeReview.interaction.payload,
|
|
})
|
|
: null;
|
|
const persistedContract = run.completionContractId
|
|
? await measureRunPhase("runtime_selection.contract_read", () => db
|
|
.select()
|
|
.from(completionContracts)
|
|
.where(
|
|
and(
|
|
eq(completionContracts.id, run.completionContractId!),
|
|
eq(completionContracts.companyId, agent.companyId),
|
|
eq(completionContracts.issueId, issueRef.id),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null))
|
|
: null;
|
|
// Only a server-verified human resolution may supply a current answer
|
|
// reference. Tool/agent results and generated summaries stay evidence.
|
|
const currentHumanResponseId = !nativeReviewRequest
|
|
? executionContinuation?.humanResponses?.find(
|
|
(response) => response.id === executionContinuation.trigger.interactionId,
|
|
)?.id
|
|
: undefined;
|
|
const immediateCompletion = (() => {
|
|
if (nativeReviewRequest) return { requests: [nativeReviewRequest], sources: [null] };
|
|
const { requests, sources } = nativeCompletionRequestsWithSources(
|
|
safeWakeComments.length > 0
|
|
? safeWakeComments
|
|
: safeWakeCommentContext?.body
|
|
? [safeWakeCommentContext]
|
|
: [],
|
|
{
|
|
requiredFullWakeCommentCount:
|
|
paperclipWakePayload?.fallbackFetchNeeded === true &&
|
|
CHAT_PROVIDERS.some(
|
|
(provider) =>
|
|
provider ===
|
|
paperclipWakePayload.externalChatProvider,
|
|
) &&
|
|
Array.isArray(paperclipWakePayload.commentIds)
|
|
? paperclipWakePayload.commentIds.length
|
|
: undefined,
|
|
},
|
|
);
|
|
// Preserve every admitted pending chat request while also
|
|
// retaining newer user direction materialized by recovery.
|
|
// A file-only wake must not inherit an old task objective.
|
|
const latestComment =
|
|
executionContinuation?.messages.findLast(
|
|
(message) =>
|
|
message.authorType === "user" &&
|
|
!message.createdByRunId &&
|
|
!message.deleted &&
|
|
message.body.trim().length > 0,
|
|
);
|
|
const latestRequest = latestComment?.body;
|
|
if (
|
|
latestRequest &&
|
|
!requests.some(
|
|
(request) => request === latestRequest.trim(),
|
|
)
|
|
) {
|
|
requests.push(latestRequest.trim());
|
|
sources.push(nativeCompletionSource("comment", latestComment!.id, latestRequest));
|
|
}
|
|
return { requests: requests.length > 0 ? requests : undefined, sources };
|
|
})();
|
|
// Rebuilding a default contract is not a change in user direction.
|
|
// In particular, an upgraded checkpoint may have an intentionally
|
|
// authored contract and no continuation envelope yet.
|
|
const completionContract =
|
|
persistedContract && persistedNativeExecutionInput
|
|
? {
|
|
row: persistedContract,
|
|
contract: persistedContract.contractJson as never,
|
|
}
|
|
: await measureRunPhase("runtime_selection.contract_ensure", () => ensureNativeCompletionContract({
|
|
db,
|
|
companyId: agent.companyId,
|
|
issue: issueRef,
|
|
actorId: agent.id,
|
|
immediateRequest:
|
|
nativeReviewRequest ?? (currentHumanResponseId
|
|
? null
|
|
: executionContinuation?.objective ?? safeWakeCommentContext?.body ?? null),
|
|
// The ordinary initial objective is selected by the server-owned
|
|
// continuation envelope. Carry its explicit description source
|
|
// through the singular-request compatibility path; do not infer
|
|
// provenance for review requests, answers, or wake fallbacks.
|
|
immediateRequestSource: nativeImmediateObjectiveSource({
|
|
issueId: issueRef.id,
|
|
objectiveSource: executionContinuation?.objectiveSource,
|
|
excluded: Boolean(nativeReviewRequest || currentHumanResponseId),
|
|
}),
|
|
humanResponseId: currentHumanResponseId,
|
|
immediateRequests: immediateCompletion.requests,
|
|
immediateRequestSources: immediateCompletion.sources,
|
|
}));
|
|
const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString(
|
|
taskSessionDecodedParams?.sessionId,
|
|
);
|
|
// Compatibility for native retry rows created before same-run restart
|
|
// recovery existed. Only an entirely unused replacement row may
|
|
// inherit its source checkpoint; any process/provider evidence on the
|
|
// replacement makes the ownership ambiguous and therefore ineligible.
|
|
const legacyRetrySource =
|
|
run.retryOfRunId && !isFailedChatRunRetry
|
|
? await measureRunPhase("runtime_selection.resume", () => db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId,
|
|
runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
status: heartbeatRuns.status,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.retryOfRunId!),
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agent.id),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null))
|
|
: null;
|
|
const nativeBootstrapHasProviderEvidence =
|
|
legacyRetrySource ||
|
|
run.nativeSessionId ||
|
|
persistedNativeExecutionInput
|
|
? await measureRunPhase("runtime_selection.resume", () => db
|
|
.select({ id: heartbeatRunEvents.id })
|
|
.from(heartbeatRunEvents)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRunEvents.runId, run.id),
|
|
inArray(heartbeatRunEvents.eventType, [
|
|
"harness.ready",
|
|
"session.started",
|
|
"session.resumed",
|
|
"session.updated",
|
|
"turn.started",
|
|
"provider.event",
|
|
"provider.rpc_result",
|
|
]),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows.length > 0))
|
|
: false;
|
|
const compatibleLegacyRetrySource =
|
|
!managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({
|
|
replacement: run,
|
|
source: legacyRetrySource,
|
|
hasProviderEvents: nativeBootstrapHasProviderEvidence,
|
|
})
|
|
? legacyRetrySource
|
|
: null;
|
|
const legacyRetrySessionId =
|
|
compatibleLegacyRetrySource?.nativeSessionId;
|
|
const taskResumeRunId =
|
|
taskSessionForRun?.lastRunId &&
|
|
taskSessionForRun.lastRunId !== run.id &&
|
|
isNativeSessionId(taskNativeSessionId)
|
|
? taskSessionForRun.lastRunId
|
|
: null;
|
|
const resumableTaskSessionId = isDotRun ? null : taskResumeRunId
|
|
? taskNativeSessionId
|
|
: (legacyRetrySessionId ?? null);
|
|
const requestedNativeSessionId =
|
|
run.nativeSessionId ?? resumableTaskSessionId;
|
|
// A task-session lastRunId can lag a failed turn or point at an older
|
|
// normalized session. Find the newest exact-session authority instead.
|
|
// Rows that already acquired provider authority are progress barriers,
|
|
// even when they do not contain a usable checkpoint.
|
|
const previousNativeRun =
|
|
requestedNativeSessionId &&
|
|
isUnusedNativeSessionBootstrap(
|
|
run,
|
|
nativeBootstrapHasProviderEvidence,
|
|
)
|
|
? await measureRunPhase("runtime_selection.resume", () => findNativeSessionResumeRun(db, {
|
|
companyId: agent.companyId,
|
|
agentId: agent.id,
|
|
issueId: issueRef.id,
|
|
normalizedSessionId: requestedNativeSessionId,
|
|
currentRunId: run.id,
|
|
beforeCreatedAt: run.createdAt,
|
|
}))
|
|
: null;
|
|
nativeRunnerInstanceId =
|
|
previousNativeRun?.runnerInstanceId &&
|
|
previousNativeRun.nativeSessionId ===
|
|
(run.nativeSessionId ?? resumableTaskSessionId)
|
|
? previousNativeRun.runnerInstanceId
|
|
: (run.runnerInstanceId ?? randomUUID());
|
|
let nativeSessionId =
|
|
run.nativeSessionId ?? resumableTaskSessionId ?? randomUUID();
|
|
let nativeResumeCheckpoint: ReturnType<
|
|
typeof rebindNativeSessionCheckpoint
|
|
> = null;
|
|
const agentLifecyclePolicy =
|
|
parseObject(agent.adapterConfig).lifecycleMode === "warm"
|
|
? {
|
|
mode: "warm" as const,
|
|
idleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs(
|
|
parseObject(agent.adapterConfig).idleTimeoutMs,
|
|
),
|
|
}
|
|
: { mode: "per_turn" as const, idleTimeoutMs: null };
|
|
const environmentLifecyclePolicy =
|
|
executionTarget?.kind === "remote" &&
|
|
executionTarget.transport === "sandbox"
|
|
? (executionTarget.runnerLifecyclePolicy ?? null)
|
|
: null;
|
|
// Task providers own credential lifetime; process-backed managed homes retain their provider-specific cleanup policy.
|
|
const supportsManagedWarmSession = agent.adapterType === "paperclip_runner" &&
|
|
((executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" && !!executionTarget.runnerTask) ||
|
|
supportsManagedNativeWarmSession({ ...nativeRuntimeResolution.profile, acpxAgent: parseObject(agent.adapterConfig).acpxAgent }, managedAiRuntime?.attribution));
|
|
const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ??
|
|
(["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession
|
|
? { mode: "per_turn" as const, idleTimeoutMs: null }
|
|
: environmentLifecyclePolicy ?? agentLifecyclePolicy);
|
|
if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend)) {
|
|
assertEphemeralNativeLifecycle(effectiveLifecyclePolicy.mode);
|
|
}
|
|
if (
|
|
effectiveLifecyclePolicy.mode === "warm" &&
|
|
executionTarget?.kind === "remote" &&
|
|
executionTarget.transport === "sandbox" &&
|
|
(executionTarget.reusableLeaseConfigured !== true ||
|
|
executionTarget.effectiveCapabilities?.reusableLeases !== true)
|
|
) {
|
|
throw new Error("runner_warm_environment_requires_reusable_lease");
|
|
}
|
|
const persistedProfile = persistedRunnerProfile;
|
|
if (persistedNativeExecutionInput) {
|
|
nativeExecution = persistedNativeExecutionInput;
|
|
if (
|
|
nativeExecution.binding.companyId !== agent.companyId ||
|
|
nativeExecution.binding.runId !== run.id ||
|
|
nativeExecution.binding.issueId !== issueRef.id ||
|
|
nativeExecution.binding.agentId !== agent.id ||
|
|
nativeExecution.binding.executionWorkspaceId !==
|
|
nativeExecutionWorkspaceId ||
|
|
nativeExecution.completionContract.id !==
|
|
completionContract.row.id ||
|
|
nativeExecution.completionContract.sha256 !==
|
|
completionContract.row.canonicalSha256
|
|
)
|
|
throw new Error(
|
|
"native_execution_input_persisted_binding_mismatch",
|
|
);
|
|
if ("runtimeContext" in nativeExecution) {
|
|
const savedVersions = parseObject(run.contextSnapshot?.skillVersions);
|
|
const accessibleKeys = new Set(runtimeSkillEntries.filter(entry => entry.sourceStatus !== "missing").map(entry => entry.key));
|
|
if (nativeExecution.runtimeContext.skills.some(skill =>
|
|
Object.prototype.hasOwnProperty.call(savedVersions, skill.key) && !accessibleKeys.has(skill.key))) {
|
|
throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", {
|
|
configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" },
|
|
});
|
|
}
|
|
}
|
|
// Recover only the originally admitted request. A stored idle
|
|
// checkpoint can precede an already-started provider turn, so even
|
|
// apparent idleness is not authority to rewrite its contract.
|
|
// Newer user direction retains its separate durable wakeup cause.
|
|
// A failed pre-bootstrap attempt may have persisted its immutable
|
|
// input before discovering that lastRunId no longer names this
|
|
// session. Restore only an exactly compatible prior checkpoint;
|
|
// never rewrite the admitted input or skip current provider work.
|
|
if (
|
|
previousNativeRun &&
|
|
isUnusedNativeSessionBootstrap(
|
|
run,
|
|
nativeBootstrapHasProviderEvidence,
|
|
)
|
|
) {
|
|
nativeResumeCheckpoint = rebindNativeSessionCheckpoint({
|
|
previousRun: previousNativeRun,
|
|
currentExecution: nativeExecution,
|
|
executionTargetKind: executionTarget?.kind ?? "local",
|
|
});
|
|
}
|
|
if (nativeExecution.provider.kind === "claude_managed") {
|
|
const recoveryProfileId = nativeExecution.provider.managedProfile.profileId;
|
|
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(
|
|
db,
|
|
).requireQualified(
|
|
agent.companyId,
|
|
recoveryProfileId,
|
|
));
|
|
assertManagedProfileRecoveryBinding({
|
|
adapterConfig: agent.adapterConfig,
|
|
snapshot: nativeExecution.provider.managedProfile,
|
|
stored: recoveryProfile,
|
|
});
|
|
}
|
|
if (nativeExecution.provider.kind === "aws_agentcore") {
|
|
const recoveryProfileId = nativeExecution.provider.agentCoreProfile.profileId;
|
|
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(
|
|
db,
|
|
).requireQualified(
|
|
agent.companyId,
|
|
recoveryProfileId,
|
|
"aws_bedrock_agentcore_harness",
|
|
));
|
|
assertAgentCoreProfileRecoveryBinding({
|
|
snapshot: nativeExecution.provider.agentCoreProfile,
|
|
stored: recoveryProfile,
|
|
});
|
|
}
|
|
} else {
|
|
const interactionId = readNonEmptyString(context.interactionId);
|
|
const interactionResponses = context[
|
|
EXTERNAL_CHAT_QUESTION_RESPONSE_KEY
|
|
]
|
|
? await measureRunPhase("runtime_selection.interactions", () => materializeExternalChatQuestionResponseInput({
|
|
db,
|
|
binding: {
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
runId: run.id,
|
|
agentId: agent.id,
|
|
},
|
|
contextSnapshot: context,
|
|
}))
|
|
: await measureRunPhase("runtime_selection.interactions", () => materializeNativeInteractionResponses({
|
|
db,
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
runId: run.id,
|
|
agentId: agent.id,
|
|
interactionIds: Array.isArray(context.interactionIds)
|
|
? [
|
|
...new Set([
|
|
...(interactionId ? [interactionId] : []),
|
|
...context.interactionIds.filter(
|
|
(id): id is string => typeof id === "string",
|
|
),
|
|
]),
|
|
]
|
|
: interactionId
|
|
? [interactionId]
|
|
: [],
|
|
}));
|
|
const runnerAdapterConfig = parseObject(agent.adapterConfig);
|
|
const managedProfile =
|
|
nativeRuntimeResolution.profile.backend ===
|
|
"claude_managed_agents_api"
|
|
? await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(db).requireQualified(
|
|
agent.companyId,
|
|
readNonEmptyString(runnerAdapterConfig.managedProfileId) ??
|
|
"",
|
|
))
|
|
: null;
|
|
const agentCoreProfile =
|
|
nativeRuntimeResolution.profile.backend ===
|
|
"aws_agentcore_harness_api"
|
|
? await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(db).requireQualified(
|
|
agent.companyId,
|
|
readNonEmptyString(
|
|
runnerAdapterConfig.agentCoreProfileId,
|
|
) ?? "",
|
|
"aws_bedrock_agentcore_harness",
|
|
))
|
|
: null;
|
|
if (managedProfile) {
|
|
const rawApiKeyBinding = parseObject(
|
|
runnerAdapterConfig.env,
|
|
).ANTHROPIC_API_KEY;
|
|
const boundSecretId =
|
|
typeof rawApiKeyBinding === "object" &&
|
|
rawApiKeyBinding !== null
|
|
? readNonEmptyString(
|
|
(rawApiKeyBinding as Record<string, unknown>).secretId,
|
|
)
|
|
: null;
|
|
if (boundSecretId !== managedProfile.apiKeySecretId) {
|
|
throw new ConfigurationIncompleteFailure(
|
|
"configuration incomplete: Claude Managed profile API key is not bound at env.ANTHROPIC_API_KEY",
|
|
{
|
|
configurationIncomplete: {
|
|
reason: "managed_agent_profile_secret_binding_mismatch",
|
|
companyId: agent.companyId,
|
|
agentId: agent.id,
|
|
profileId: managedProfile.id,
|
|
requiredEnvKeys: ["ANTHROPIC_API_KEY"],
|
|
},
|
|
},
|
|
);
|
|
}
|
|
}
|
|
const executionMode =
|
|
issueRef.workMode === "planning" && !isConversation(issueContext) && !acceptedPlanContinuationWake
|
|
? ("plan" as const)
|
|
: ("default" as const);
|
|
const pinnedPlan =
|
|
executionMode === "plan"
|
|
? await documentService(db).getIssueDocumentByKey(
|
|
issueRef.id,
|
|
"plan",
|
|
)
|
|
: null;
|
|
const pinnedReviewContext =
|
|
executionMode === "plan"
|
|
? await buildPlanReviewContext({
|
|
db,
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
issueWorkMode: issueRef.workMode,
|
|
interactionId: readNonEmptyString(context.interactionId),
|
|
})
|
|
: null;
|
|
const pinnedPlanMarkdown = pinnedPlan?.body ?? "";
|
|
const museBinding = nativeRuntimeResolution.profile.backend === "muse_external"
|
|
? await museRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig).museBindingId ?? "")) : undefined;
|
|
const dotBinding = ["openai_dot_mcp", "grok_bot_mcp"].includes(nativeRuntimeResolution.profile.backend)
|
|
? await dotRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig)[nativeRuntimeResolution.profile.backend === "grok_bot_mcp" ? "externalBindingId" : "dotBindingId"] ?? "")) : undefined;
|
|
const nativeRuntimeContext = await buildNativeRuntimeContext({
|
|
db,
|
|
agent,
|
|
runId: run.id,
|
|
runtimeConfig,
|
|
runtimeSkillEntries,
|
|
instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined,
|
|
});
|
|
getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff;
|
|
const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "cursor_cloud_api"
|
|
|| nativeRuntimeResolution.profile.backend === "codex_app_server"
|
|
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime" && ["claude_local", "grok_local"].includes(agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig))))
|
|
|| nativeRuntimeResolution.profile.backend === "opencode_server"
|
|
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime"
|
|
&& parseObject(agent.adapterConfig).acpxAgent === "copilot")
|
|
? projectPaperclipRunnerTaskConfig(
|
|
nativeRuntimeResolution.profile.backend,
|
|
agent.adapterConfig,
|
|
issueAssigneeOverrides?.adapterConfig,
|
|
managedAiRuntime ? readNonEmptyString(resolvedConfig.model) ?? undefined : undefined,
|
|
)
|
|
: agent.adapterConfig;
|
|
const requestedNativeProvider = resolvePaperclipRunnerNativeProviderInput({
|
|
companyId: agent.companyId,
|
|
backend: nativeRuntimeResolution.profile.backend,
|
|
adapterConfig: nativeProviderConfig, managedProfile, agentCoreProfile, dotBinding, museBinding,
|
|
});
|
|
const codexCliVersion = agent.adapterType === "paperclip_runner" && requestedNativeProvider.provider === "codex"
|
|
? await readRemoteCodexModelCliVersion({
|
|
model: requestedNativeProvider.model,
|
|
target: executionTarget,
|
|
remoteCodexPath: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH,
|
|
remoteCodexNpmSpec: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC,
|
|
}) : null;
|
|
const nativeProvider = codexCliVersion
|
|
? resolvePaperclipRunnerNativeProviderInput({
|
|
companyId: agent.companyId,
|
|
backend: nativeRuntimeResolution.profile.backend,
|
|
adapterConfig: nativeProviderConfig, codexCliVersion, managedProfile, agentCoreProfile, dotBinding, museBinding,
|
|
}) : requestedNativeProvider;
|
|
if (nativeProvider.model !== requestedNativeProvider.model) {
|
|
await postNativeModelFallbackWarning({
|
|
issuesSvc, onEvent: onAdapterEvent, issueId: issueRef.id, runId: run.id,
|
|
requestedModel: requestedNativeProvider.model, effectiveModel: nativeProvider.model,
|
|
codexCliVersion: codexCliVersion!,
|
|
});
|
|
}
|
|
const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) =>
|
|
buildNativeExecutionInput({
|
|
agentKeyId: agentIdentity?.keyId,
|
|
companyId: agent.companyId,
|
|
runId: run.id,
|
|
issue: nativeReviewRequest ? { ...issueRef, title: `Review: ${issueRef.title}`, description: nativeReviewRequest } : issueRef,
|
|
taskPrompt: [
|
|
nativeReviewRequest ?? readNonEmptyString(
|
|
selectPaperclipTaskMarkdown(context, {
|
|
resumedSession: false,
|
|
includeCommunicationGuidance: false,
|
|
}),
|
|
) ??
|
|
`# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`,
|
|
!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) && projectRepositoryPaths.length > 0
|
|
? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}`
|
|
: null,
|
|
].filter(Boolean).join("\n\n"),
|
|
initialCommunicationGuidance: nativeReviewRequest ? null : readNonEmptyString(context.paperclipTaskCommunicationGuidance),
|
|
wakePayload: context.paperclipWake,
|
|
caseContext: nativeReviewRequest ? null : context.paperclipCase as CaseExecutionContext | undefined,
|
|
turnContext: context.paperclipTurnContext,
|
|
githubInstructionSkillKeys: runScopedMentionedSkillKeys,
|
|
resumedSession,
|
|
previousTurn: (() => {
|
|
if (!previousNativeRun || nativeReviewRequest) return null;
|
|
try {
|
|
const previousInput = parseNativeExecutionInput(parseObject(previousNativeRun.runnerProfileJson).nativeExecutionInput);
|
|
const previousTask = previousInput.task;
|
|
const previousWake = normalizePaperclipWakePayload(parseObject(previousNativeRun.contextSnapshot).paperclipWake);
|
|
const githubReviewConfigurationKeys = previousWake?.githubReviewConfigurations.map((config) => config.key);
|
|
if (paperclipWakePayload?.externalChatProvider || previousWake?.externalChatProvider) {
|
|
// External native inputs use a neutral task title. Compare
|
|
// the saved canonical brief so old provider text is not
|
|
// repeated as a change, while genuine edits still arrive.
|
|
const savedIssue = parseObject(parseObject(previousNativeRun.contextSnapshot).paperclipIssue);
|
|
if (savedIssue.id !== issueRef.id || typeof savedIssue.title !== "string" ||
|
|
(savedIssue.description !== null && typeof savedIssue.description !== "string")) return null;
|
|
return { runId: previousNativeRun.id, task: { title: savedIssue.title, description: savedIssue.description },
|
|
githubReviewConfigurationKeys, nativeExecutionInput: previousInput,
|
|
};
|
|
}
|
|
return { runId: previousNativeRun.id, task: previousTask, githubReviewConfigurationKeys, nativeExecutionInput: previousInput };
|
|
} catch {
|
|
// An invalid prior snapshot must use the fresh bootstrap.
|
|
return null;
|
|
}
|
|
})(),
|
|
conversationMode: context.conversationMode === true,
|
|
agentId: agent.id,
|
|
workspace: {
|
|
// Projectless paperclip_runner tasks still have a resolved local cwd. Bind that
|
|
// transient workspace to the run id so the native input remains durable and replayable
|
|
// without fabricating a project-scoped execution_workspaces row.
|
|
id: nativeExecutionWorkspaceId,
|
|
cwd: executionWorkspace.cwd,
|
|
repoUrl: executionWorkspace.repoUrl,
|
|
repoRef: executionWorkspace.repoRef,
|
|
branchName: executionWorkspace.branchName,
|
|
},
|
|
normalizedSessionId,
|
|
executionMode,
|
|
planningContext:
|
|
executionMode === "plan"
|
|
? {
|
|
documentId: pinnedPlan?.id ?? null,
|
|
baseRevisionId:
|
|
pinnedPlan?.latestRevisionId ?? null,
|
|
baseRevisionNumber:
|
|
pinnedPlan?.latestRevisionNumber ?? 0,
|
|
markdown: pinnedPlanMarkdown,
|
|
sha256: createHash("sha256")
|
|
.update(pinnedPlanMarkdown)
|
|
.digest("hex"),
|
|
reviewContext: pinnedReviewContext
|
|
? (structuredClone(
|
|
pinnedReviewContext,
|
|
) as unknown as Record<string, unknown>)
|
|
: {},
|
|
}
|
|
: null,
|
|
...nativeProvider,
|
|
lifecyclePolicy: effectiveLifecyclePolicy,
|
|
interactionResponses,
|
|
completionContract: {
|
|
id: completionContract.row.id,
|
|
sha256: completionContract.row.canonicalSha256,
|
|
schemaVersion: completionContract.row.schemaVersion,
|
|
contract: completionContract.contract,
|
|
sources: "sources" in completionContract ? completionContract.sources : undefined,
|
|
},
|
|
runtimeContext: nativeRuntimeContext,
|
|
});
|
|
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
|
|
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
|
|
})));
|
|
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
|
|
previousRun: previousNativeRun,
|
|
normalizedSessionId: nativeSessionId,
|
|
executionTargetKind: executionTarget?.kind ?? "local",
|
|
toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true,
|
|
refreshTools: context.refreshTools === true,
|
|
buildExecution,
|
|
});
|
|
nativeExecution = nativeExecutionWithCheckpoint.execution;
|
|
nativeResumeCheckpoint = nativeExecutionWithCheckpoint.checkpoint;
|
|
if (
|
|
nativeSessionId !==
|
|
nativeExecutionWithCheckpoint.normalizedSessionId
|
|
) {
|
|
nativeRunnerInstanceId = randomUUID();
|
|
}
|
|
nativeSessionId = nativeExecutionWithCheckpoint.normalizedSessionId;
|
|
}
|
|
if (nativeResumeCheckpoint && nativeExecution.provider.kind === "acpx"
|
|
&& nativeExecution.provider.agent === "antigravity") {
|
|
nativeResumeCheckpoint = { ...nativeResumeCheckpoint, providerRecoveryPolicy: "same_session_only" };
|
|
}
|
|
const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({
|
|
adapterType: agent.adapterType,
|
|
lifecyclePolicy: nativeExecution.session.lifecyclePolicy,
|
|
target: executionTarget,
|
|
});
|
|
if (nativeSandboxLifecycle) {
|
|
onNativeLifecycleSelected({
|
|
provider: nativeExecution.provider.kind,
|
|
harness: nativeExecution.session.driverKind,
|
|
lifecycleMode: nativeExecution.session.lifecyclePolicy.mode,
|
|
sandboxResource: nativeSandboxLifecycle.sandboxResource,
|
|
});
|
|
const selectedLifecycleSpan = getStartupTracer(
|
|
"paperclip.environment-lifecycle",
|
|
).startSpan("sandbox.lifecycle.selected", {
|
|
attributes: {
|
|
"paperclip.native.span.provider": nativeExecution.provider.kind,
|
|
"paperclip.native.span.harness":
|
|
nativeExecution.session.driverKind,
|
|
"paperclip.native.span.lifecycle_mode":
|
|
nativeExecution.session.lifecyclePolicy.mode,
|
|
"paperclip.native.span.sandbox_resource":
|
|
nativeSandboxLifecycle.sandboxResource,
|
|
"paperclip.native.span.outcome": "selected",
|
|
"paperclip.native.span.bytes_transferred": 0,
|
|
},
|
|
});
|
|
selectedLifecycleSpan.end();
|
|
}
|
|
providerResourceDispositionForRun =
|
|
nativeSandboxLifecycle?.sandboxResource === "keep_running"
|
|
? "keep_running"
|
|
: nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse"
|
|
? "stop_and_retain"
|
|
: nativeSandboxLifecycle?.sandboxResource ===
|
|
"destroy_after_turn"
|
|
? "destroy"
|
|
: undefined;
|
|
onProviderResourceDisposition(providerResourceDispositionForRun);
|
|
await beforeNativeRuntimeSelection?.(run.id);
|
|
const nativeSelected = await measureRunTransaction(db, "runtime_selection", async (tx) => {
|
|
const lockedRun = await measureRunPhase("runtime_selection.row_lock", () => tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, run.id))
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null));
|
|
if (!lockedRun) throw new Error("native_runtime_run_missing");
|
|
// Cancellation and runtime selection serialize on this row. A
|
|
// stopped preparation must never create a new native coordinator.
|
|
const stopRecovery = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
|
|
&& runOptions.nativeRestartRecovery.runId === lockedRun.id
|
|
&& runOptions.nativeRestartRecovery.leaseOwner === runOptions.nativeLeaseOwner
|
|
&& await readClaimedNativeStopRecovery(tx as unknown as Db, lockedRun, runOptions.nativeLeaseOwner);
|
|
if (lockedRun.status !== "running" || (lockedRun.resultJson?.startupCancellation && !stopRecovery)) return false;
|
|
if (lockedRun.runtimeMode === "legacy" && lockedRun.controllerBootId &&
|
|
!(await renewLegacyControllerLease(tx as unknown as Db, lockedRun))) {
|
|
onNativeOwnershipHeld();
|
|
return false;
|
|
}
|
|
if (
|
|
lockedRun.runtimeModeResolvedAt &&
|
|
lockedRun.runtimeMode !== "native"
|
|
) {
|
|
throw new Error("native_runtime_mode_conflict");
|
|
}
|
|
const lockedProfile = parseObject(lockedRun.runnerProfileJson);
|
|
const persistedNativeSessionId =
|
|
await measureRunPhase("runtime_selection.bootstrap", () => prepareNativeSessionBootstrapPersistence(tx, {
|
|
run: lockedRun,
|
|
selectedSessionId: nativeSessionId,
|
|
execution: nativeExecution!,
|
|
restoringCheckpoint: nativeResumeCheckpoint !== null,
|
|
}));
|
|
await measureRunPhase("runtime_selection.run_write", () => tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runtimeMode: "native",
|
|
controllerBootId: null,
|
|
controllerLeaseExpiresAt: null,
|
|
executionStage: null,
|
|
runtimeModeResolverVersion:
|
|
lockedRun.runtimeModeResolverVersion ??
|
|
nativeRuntimeResolution.resolverVersion,
|
|
runtimeModeReason:
|
|
lockedRun.runtimeModeReason ?? nativeRuntimeResolution.reason,
|
|
runtimeModeResolvedAt:
|
|
lockedRun.runtimeModeResolvedAt ?? new Date(),
|
|
runnerProfileJson: {
|
|
...nativeRuntimeResolution.profile,
|
|
...lockedProfile,
|
|
...(nativeExecution?.provider.kind === "cursor_cloud" ? { executionPlacement: resolveProviderApiPlacement({ agent, run }) } : {}),
|
|
nativePreparationFencingVersion: !lockedRun.runnerInstanceId && !lockedRun.nativeSessionId
|
|
? NATIVE_PREPARATION_FENCING_VERSION
|
|
: lockedProfile.nativePreparationFencingVersion,
|
|
...(lockedProfile.nativeExecutionInput
|
|
? {}
|
|
: { recoveryEventInventoryVersion: 1 }),
|
|
...(providerTraceRequested
|
|
? {
|
|
providerTrace: {
|
|
mode: "raw",
|
|
traceId: providerTraceCapture?.metadata.id ?? null,
|
|
maxBytes: PROVIDER_TRACE_MAX_BYTES,
|
|
},
|
|
}
|
|
: {}),
|
|
nativeExecutionInput:
|
|
lockedProfile.nativeExecutionInput ?? nativeExecution,
|
|
nativeResourceDisposition: providerResourceDispositionForRun,
|
|
nativeToolContractFingerprint:
|
|
nativeToolContractFingerprintForTarget(
|
|
executionTarget?.kind ?? "local",
|
|
nativeExecution!.provider.kind,
|
|
),
|
|
...(lockedProfile.sessionCheckpoint != null
|
|
? { sessionCheckpoint: lockedProfile.sessionCheckpoint }
|
|
: nativeResumeCheckpoint
|
|
? {
|
|
sessionCheckpoint:
|
|
nativeResumeCheckpoint as unknown as Record<
|
|
string,
|
|
unknown
|
|
>,
|
|
}
|
|
: {}),
|
|
},
|
|
runnerInstanceId:
|
|
previousNativeRun?.runnerInstanceId &&
|
|
persistedNativeSessionId === previousNativeRun.nativeSessionId
|
|
? previousNativeRun.runnerInstanceId
|
|
: lockedRun.nativeSessionId !== persistedNativeSessionId
|
|
? nativeRunnerInstanceId
|
|
: (lockedRun.runnerInstanceId ?? nativeRunnerInstanceId),
|
|
nativeSessionId: persistedNativeSessionId,
|
|
processPid:
|
|
lockedRun.processPid ??
|
|
(previousNativeRun?.nativeSessionId === nativeSessionId
|
|
? previousNativeRun.processPid
|
|
: null),
|
|
processGroupId:
|
|
lockedRun.processGroupId ??
|
|
(previousNativeRun?.nativeSessionId === nativeSessionId
|
|
? previousNativeRun.processGroupId
|
|
: null),
|
|
processStartedAt:
|
|
lockedRun.processStartedAt ??
|
|
(previousNativeRun?.nativeSessionId === nativeSessionId
|
|
? previousNativeRun.processStartedAt
|
|
: null),
|
|
nativeIssueId: lockedRun.nativeIssueId ?? issueRef.id,
|
|
driverKind:
|
|
lockedRun.driverKind ??
|
|
nativeExecution?.session.driverKind ??
|
|
"codex_app_server",
|
|
driverVersion: lockedRun.driverVersion ?? "phase6-v1",
|
|
completionContractId:
|
|
lockedRun.completionContractId ?? completionContract.row.id,
|
|
completionContractSha256:
|
|
lockedRun.completionContractSha256 ??
|
|
completionContract.row.canonicalSha256,
|
|
nativePhase: lockedRun.nativePhase ?? "observed",
|
|
nativePhaseUpdatedAt:
|
|
lockedRun.nativePhaseUpdatedAt ?? new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, run.id)));
|
|
await measureRunPhase("runtime_selection.finalization_write", () => tx
|
|
.insert(nativeRunFinalizations)
|
|
.values({
|
|
runId: run.id,
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
phase: "observed",
|
|
})
|
|
.onConflictDoNothing());
|
|
return true;
|
|
});
|
|
if (!nativeSelected) return { selected: false };
|
|
stopControllerLease();
|
|
if (nativeExecution?.provider.kind !== "cursor_cloud" && !activeEnvironmentLease) throw new Error("native_environment_lease_unavailable");
|
|
nativeWorkspaceSync = nativeExecution?.provider.kind === "cursor_cloud" || readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata) ? null
|
|
: await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({
|
|
db,
|
|
runId: run.id,
|
|
companyId: agent.companyId,
|
|
workspaceId: nativeExecutionWorkspaceId,
|
|
workspaceLocalDir: executionWorkspace.cwd,
|
|
target: executionTarget,
|
|
lease: activeEnvironmentLease!.lease,
|
|
restartRecovery: runOptions.nativeRestartRecovery,
|
|
sameRunRecovery: Boolean(runOptions.nativeLeaseOwner),
|
|
resourceDisposition: providerResourceDispositionForRun,
|
|
workspaceGitIndexRefresh: nativeExecution?.provider.kind === "opencode",
|
|
onCheckpoint: input.workspace.onCheckpoint,
|
|
onPhase: input.workspace.onPhase,
|
|
onEvent: onAdapterEvent,
|
|
}));
|
|
} else {
|
|
const legacyWarmLifecycle =
|
|
executionTarget?.kind === "remote" &&
|
|
executionTarget.transport === "sandbox" &&
|
|
executionTarget.runnerLifecyclePolicy?.mode === "warm"
|
|
? resolveReusableSandboxLifecycle({
|
|
lifecyclePolicy: executionTarget.runnerLifecyclePolicy,
|
|
target: executionTarget,
|
|
})
|
|
: null;
|
|
if (legacyWarmLifecycle?.sandboxResource === "keep_running") {
|
|
providerResourceDispositionForRun = "keep_running";
|
|
onProviderResourceDisposition(providerResourceDispositionForRun);
|
|
}
|
|
await measureRunPhase("runtime_selection.run_write", () => db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runtimeMode: "legacy",
|
|
runtimeModeResolverVersion:
|
|
nativeRuntimeResolution.resolverVersion,
|
|
runtimeModeReason: nativeRuntimeResolution.reason,
|
|
runtimeModeResolvedAt: run.runtimeModeResolvedAt ?? new Date(),
|
|
// Preserve server-owned admission and dispatch evidence on this
|
|
// row; never copy another run's execution profile.
|
|
runnerProfileJson: sql`(case when ${heartbeatRuns.runnerProfileJson} ? ${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}
|
|
then jsonb_build_object(${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}::text, ${heartbeatRuns.runnerProfileJson}->${CHAT_CONTROL_RECOVERY_ADMISSION_KEY})
|
|
else '{}'::jsonb end)
|
|
|| (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch'
|
|
then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch')
|
|
else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, run.id)));
|
|
}
|
|
return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync };
|
|
}
|