mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent workflow needs persistent facts and a safe way to transfer work. > - Creating a task for each phase splits the objective, discussion, workspace, and review policy. > - A case can retain the facts while one primary task moves between agents. > - Transfers must survive process failure and preserve private access, budgets, and completion gates. > - This pull request adds durable case handoffs, native case tools, and read-only case pages. > - Prompts describe the workflow. Existing task controls decide when work can run and finish. ## Linked Issues or Issue Description **Problem or motivation** Prompt-driven agent teams need a persistent record that can move through triage, implementation, verification, and correction. Ordinary reassignment does not provide a durable instruction, safe release barrier, or recoverable delivery receipt for an agent handing off its own running task. **Proposed solution** Extend standalone cases with one primary task, a descriptive phase, retained access sources, and a handoff ledger. Apply assignments after source execution and workspace settlement. Deliver each transfer through the existing durable wake queue. Provide `create_case`, `read_case`, `update_case`, and `handoff_case` in the native runner. Creation requires kickoff and explicit workflow text. An explicit handoff to `done` closes the case when normal task completion is accepted. **Alternatives considered** Do not add a workflow engine, filesystem queue, configured state graph, or one task per phase. The workflow document combines transition prose with an optional Mermaid `stateDiagram-v2` map. The server does not execute the diagram. The Pipelines experiment removed on master stays removed. **Roadmap alignment** This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run Recovery, and Activity Log attribution. Existing approval, review, budget, privacy, workspace, and responsible-user rules remain authoritative. Related work: [ordinary handoff wake recovery](https://github.com/paperclipai/paperclip/pull/15019). This feature adds a separate persisted case-handoff receipt and release barrier. ## What Changed - Add case routing fields, retained access sources, durable handoff history, delivery leases, and replay-safe migrations. - Keep the same primary task across agent phases. Preserve each returning agent's own session and the task's workspace. - Fence outgoing mutations, wait for provider and workspace release, and recover delivery without duplicate task comments or runs. Recheck current assignment permissions as the original caller before applying a delayed transfer. Scope a run Stop to that run’s own handoff intents, so repeated old Stop requests preserve newer work. Cancelling terminal intent without a source run also requires fenced current task ownership, including native retries and preparation with an old checkout. For Cloud callers, revalidate current stack membership with an uncached Cloud RPC and derive owner elevation from the current managed setting. - Keep case reads, documents, assets, Audit, and live events inside retained task/project permissions. - Add native case creation/read/update/handoff tools. Store required workflow text and initial documents on the first task. Page oversized case facts and referenced document bodies through the dedicated tool with bounded UTF-8 JSON, current routing versions, snapshot conflicts, and current source authorization on every page. - Add read-only case lists and detail pages, task-style list controls, compact dotted field tables, linked handoff tasks, and case filters in Audit. - Add native/legacy integration tests, migration/restart tests, browser coverage, and bounded live receipt fixtures. Extend the trusted sandbox condition to cover the new subscription fixture profile. - Hide obsolete paused-agent warnings on completed or cancelled task threads. - Synchronize the company-switch settings test with React effects after a CI failure. Preserve its security assertions and production remount behavior. ## Verification The branch directly includes master `9ef434fe79`. The required `ci / verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614) passes typecheck, build, tests, and all eight browser shards. Security checks pass. [Greptile gives this exact head 5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549), with no unresolved review threads. The optional Paperclip Review check remains queued because its assigned agent is paused. It is not a required branch check. No check result was overridden. Local workspace typecheck and build pass after the master merge, along with database migration checks, token gates, map checks, Storybook catalog checks, and the Copilot profile identity check. Standard builds refresh Runner vendor output. Focused checks pass 2,008 API/paging/context tests and 7 strict case protocol tests. The final commit changes one test fixture only; its ten tests, UI typecheck, and token gates pass. No production source changed in that commit. On current head `765fccd67c`, all **194 critical workflow tests** executed and passed with no skipped tests in those files: - 38 Stop/ownership tests, including board completion intent, native retries, stale Stop, and preparation with an old checkout. - 2 actual legacy process/HTTP handoff tests, including scratch failure, a closed-stdio child that continues writing, and recovery through a fresh worker after the process group stops. - 5 migration tests, including SQL column metadata, preserved null authority context, and all six prior-numbering receipts. - 3 process-settlement and 5 scratch tests. - 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool integration tests. - 7 bounded case/document-read projection tests and 7 strict protocol-contract tests. The current CI checkout was `fed65d49`, merging `765fccd67c` into master `9ef434fe79`. Its new document regressions reconstruct exact Unicode/escaped bodies and revisions through real native authority and authenticated HTTP with generic API tools disabled. They cover case-owned, primary-task, and linked-task documents; routing on every page; stale document snapshots; access loss and unlinking; unrelated/foreign/private sources; system documents; and quarantined content. Active-task document tools retain their existing behavior. These are current-head test results from the first CI attempt; the browser-only retry did not rerun these suites. The full `64fdd3403f` CI failed one separate settings test: after a company switch, it observed an unsettled API-key draft. The production component and test were unchanged from prior green source. Production switches remount both keyed settings layers; the test directly reused an unkeyed view and used a timer tick instead of waiting for React effects. The final commit replaces that fixture synchronization with React `act`. The exact secret-clearing assertions and all timeouts remain unchanged. The original assertion failure remains recorded; no unchanged retry was used. The narrow unchanged baseline and fixed fixture each pass 10 tests. The upstream result-paging merge preserves case capture namespaces and current asset privacy. Its API integration suite executed 39 of 41 tests successfully, with two skipped; the optional assigned-result PRP integration test was skipped. Neither skipped suite is counted in the 194 critical tests or as live provider proof. All 361 upstream migration entries and SQL bodies remain unchanged. Six case migrations follow master as 0363–0368 with identical SQL hashes. Official Drizzle generators produced the combined metadata and five retained snapshots. Generation reports no schema changes, and the snapshot drift test passes. The earlier `ce361ae5d0` CI passed on attempt 1, including its then-current 179 critical tests. Fresh review later identified unusable document references with generic API tools disabled; the bounded document reads and ten additional authenticated HTTP tests address that finding. The full local `pnpm test:run` attempt was cancelled after this Mac exhausted PostgreSQL semaphore capacity; it is **not** counted as a pass. Later database attempts also stopped before assertions or skipped, including one sandbox socket denial. Two original stale-Stop regressions did reproduce the bug against the old source. A REST serialization assertion passed after two preserved database-startup failures. No test timeout was increased. All 194 assertions listed above executed and passed in current-head CI. The complete 385-file, 3,963-test UI shard also passed after the fixture repair. A configured-provider receipt workflow passed 13 backend checks across triage → implementation → verification → correction → done, using one task and four work runs. The saved receipt is revision 2 with total 6 USD; case and task both closed. The original canonical attempt remains failed because its browser assertion confused the terminal destination with completed-task tooltips. The corrected full browser verifier passed on the same retained database and original runs, with no additional provider calls. The last clean-build browser readback is on `6a86c54a53`; later changes affect recovery, authorization, bounded reads, and settlement. Accounted work cost is a $1.010574 rate-card estimate; provider billing and setup-probe usage are unavailable. Copilot profile 49 passed the unchanged seven-fixture private protocol pack (46/46 checks, no retries) and the product file-edit canary after master's completion-replay change. All 60 bound implementation hashes remain unchanged on the current source. Historical profile-47/48 identities and results remain intact. See the [profile-49 compatibility record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json). This is bounded compatibility evidence, not a fresh full matrix, installed-package product test, or Daytona rollout. Preserved CI failures and retries: - On `20662930ac`, Slack managed setup failed a five-second success assertion after reload. Its test/application path was unchanged from a prior green head. One unchanged-source retry passed all 25 tests in that shard; no source or timeout changed. - On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19 UTC. Neither log had an assertion failure before interruption. One unchanged-source retry of each passed all Runner static checks and all 32 tests in browser shard 6; both aggregates passed. The next head, `7b4374edaf`, passed all CI without retries. - On `295f289b5a`, serialized shard 9 timed out during its first device-login test, then the next test found an active login lease. Direct login/test paths were unchanged from a prior green head. Cold import plus a timed-out async body using the next test's mutable actor/store is the supported fixture-race explanation; no request trace proves that sequence. The original failure remains recorded. One unchanged-source retry passed all 294 tests in 19 files, including all 41 device-login tests. No login source or timeout changed. The subsequent `ce361ae5d0` CI passed without retries. - On `64fdd3403f`, one company-switch settings assertion observed an unsettled draft. The final test-only commit uses React `act` without changing assertions, timeouts, or production code. On `765fccd67c`, all ten settings tests and the complete 385-file, 3,963-test UI shard passed. - On `765fccd67c`, browser shard 3 completed the split-handoff assertions, then cleanup received HTTP 500 from run cancellation after a one-second PostgreSQL row-lock timeout. The extra chat run was reporting completed work while teardown began. Cleanup stopped before restoring a global experiment flag, causing a later assertion to fail. Relevant production and test paths are unchanged from an earlier green commit. Stop's lock order is consistent, and no nested self-lock was found; the exact PostgreSQL blocker was not recorded. One unchanged-shard retry passed. The original failure, trace, and downstream flag failure remain recorded. No Stop implementation, cleanup assertion, or timeout changed. The retry supports intermittent cleanup contention; it does not identify the original blocking transaction. ## Risks - Local process transfers require host-owned PID/group absence. Missing or uncertain termination evidence keeps delivery pending, independently of scratch setup. - Handoff admission and task cancellation share execution ownership. New regression tests cover temporary holds, late source writes, and uncertain provider termination. - Retained private sources use an intersection of existing permissions. This intentionally fails closed when a source is missing or an existing task cannot preserve that boundary. - Migrations add tables and indexes. They preserve master history and can replay over prior feature installs. Migration 0368 adds nullable internal authorization provenance with no default or authority backfill. Historical transfers still awaiting release without this context fail closed; an operator must cancel them and request a new handoff. - Delayed Cloud handoffs depend on the current Cloud membership RPC. An unavailable, ambiguous, expired, or mismatched response holds the handoff for retry; it does not reuse cached authority or grant access. - Bounded reads reload and reauthorize the selected projection per page. Underlying work grows with the data size and number of pages. Changed data or routing requires restarting from offset zero without the old snapshot. - Phase names and workflow diagrams remain advisory. They do not satisfy review or approval gates. - This is a large, cross-cutting PR. The maintainer requested one PR for the whole feature; the commits separate data, runtime, UI, and qualification work. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated code review. The exact deployment model ID and context-window size were not exposed in the session. Live fixture agents used the explicitly configured provider model recorded in their test evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1091 lines
51 KiB
TypeScript
1091 lines
51 KiB
TypeScript
import { museRunnerBroker } from "../muse-runner-broker.js";
|
|
import { measureRunTransaction } from "../execution-transaction-timing.js";
|
|
import { measureRunPhase } from "../execution-preparation-timing.js";
|
|
import {
|
|
ConfigurationIncompleteFailure,
|
|
type buildPaperclipWakePayload,
|
|
type createHeartbeatRunPreparation,
|
|
} from "./run-preparation.js";
|
|
import { dotRunnerBroker } from "../dot-runner-broker.js";
|
|
import { isAgentDirectoryCopy } from "../agent-directory-working-copies.js";
|
|
import { isConversation } from "../agent-conversations.js";
|
|
import { renewLegacyControllerLease } from "../legacy-controller-lease.js";
|
|
import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js";
|
|
import { supportsManagedNativeWarmSession } from "../native-runtime/native-warm-capability.js";
|
|
import {
|
|
getNativeReviewAssignment,
|
|
readNativeReviewAssignmentContext,
|
|
} from "../native-runtime/native-review-participant.js";
|
|
import { buildNativeReviewRequest } from "../native-runtime/native-review-prompt.js";
|
|
import type { CaseExecutionContext } from "../case-context.js";
|
|
import type { buildExecutionContinuation } from "../execution-continuation.js";
|
|
import type { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { agentHarnessType, CHAT_PROVIDERS } from "@paperclipai/shared";
|
|
import {
|
|
type agents,
|
|
completionContracts,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
nativeRunFinalizations,
|
|
} from "@paperclipai/db";
|
|
import { getStartupTracer } from "../../instrumentation.js";
|
|
import { documentService } from "../documents.js";
|
|
import { managedAgentProfileService } from "../managed-agent-profiles.js";
|
|
import { remoteAgentProfileService } from "../remote-agent-profiles.js";
|
|
import {
|
|
buildNativeExecutionInput,
|
|
buildNativeExecutionWithCheckpoint,
|
|
buildNativeRuntimeContext,
|
|
ensureNativeCompletionContract,
|
|
findNativeSessionResumeRun,
|
|
isNativeSessionId,
|
|
isUnusedNativeSessionBootstrap,
|
|
isUnusedLegacyNativeRetryReplacement,
|
|
materializeNativeInteractionResponses,
|
|
nativeCompletionRequestsWithSources,
|
|
nativeCompletionSource,
|
|
nativeImmediateObjectiveSource,
|
|
nativeToolContractFingerprintForTarget,
|
|
prepareNativeSessionBootstrapPersistence,
|
|
prepareNativeWorkspaceSync,
|
|
type NativeRestartRecoveryClaim,
|
|
rebindNativeSessionCheckpoint,
|
|
type resolveHeartbeatNativeRuntimeMode,
|
|
} from "../native-runtime/index.js";
|
|
import {
|
|
assertAgentCoreProfileRecoveryBinding,
|
|
assertManagedProfileRecoveryBinding,
|
|
projectPaperclipRunnerTaskConfig,
|
|
resolvePaperclipRunnerNativeProviderInput,
|
|
} from "../native-runtime/provider-profile.js";
|
|
import { readRemoteCodexModelCliVersion } from "../native-runtime/codex-model-fallback.js";
|
|
import {
|
|
describeRunnerdNativeSessionBackend,
|
|
parseNativeExecutionInput,
|
|
type NativeExecutionInput,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import { PROVIDER_TRACE_MAX_BYTES } from "../provider-trace-store.js";
|
|
import type { AdapterRuntimeEvent } from "../../adapters/index.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "../native-runtime/external-chat-question-response.js";
|
|
import {
|
|
materializeExternalChatQuestionResponseInput,
|
|
} from "../native-runtime/external-chat-question-response-input.js";
|
|
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js";
|
|
import { NATIVE_PREPARATION_FENCING_VERSION } from "../native-runtime/native-active-runs.js";
|
|
import type { issueService } from "../issues.js";
|
|
import { buildPlanReviewContext } from "../plan-review-context.js";
|
|
import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils";
|
|
import { normalizePaperclipWakePayload, selectPaperclipTaskMarkdown } from "@paperclipai/adapter-utils/server-utils";
|
|
import type { ProviderResourceDisposition } from "../environment-runtime.js";
|
|
|
|
type Run = typeof heartbeatRuns.$inferSelect;
|
|
type Agent = typeof agents.$inferSelect;
|
|
type IssueContext = Awaited<ReturnType<ReturnType<typeof createHeartbeatRunPreparation>["getIssueExecutionContext"]>> | null;
|
|
type IssueRef = Parameters<typeof ensureNativeCompletionContract>[0]["issue"] & Parameters<typeof buildNativeExecutionInput>[0]["issue"];
|
|
|
|
export interface NativeLifecycleTelemetry {
|
|
provider: string;
|
|
harness: string;
|
|
lifecycleMode: "per_turn" | "warm";
|
|
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
|
}
|
|
|
|
export interface HeartbeatRuntimeSelectionInput {
|
|
run: Run;
|
|
agent: Agent;
|
|
nativeRuntimeResolution: ReturnType<typeof resolveHeartbeatNativeRuntimeMode>;
|
|
task: {
|
|
issueRef: IssueRef | null;
|
|
issueContext: IssueContext;
|
|
context: Record<string, unknown>;
|
|
executionContinuation: Awaited<ReturnType<typeof buildExecutionContinuation>> | null;
|
|
safeWakeComments: Parameters<typeof nativeCompletionRequestsWithSources>[0];
|
|
safeWakeCommentContext: { id: string; body: string } | null;
|
|
paperclipWakePayload: Awaited<ReturnType<typeof buildPaperclipWakePayload>>;
|
|
acceptedPlanContinuationWake: boolean;
|
|
};
|
|
session: {
|
|
persistedNativeExecutionInput: NativeExecutionInput | null;
|
|
persistedRunnerProfile: Record<string, unknown>;
|
|
taskSessionCredentialCompatible: boolean;
|
|
taskSessionDecodedParams: Record<string, unknown> | null;
|
|
taskSessionForRun: { lastRunId: string | null } | null;
|
|
isFailedChatRunRetry: boolean;
|
|
getFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
|
};
|
|
workspace: {
|
|
persistedExecutionWorkspace: { id: string } | null;
|
|
executionWorkspace: Omit<Parameters<typeof buildNativeExecutionInput>[0]["workspace"], "id">;
|
|
executionTarget: Parameters<typeof prepareNativeWorkspaceSync>[0]["target"];
|
|
activeEnvironmentLease: Pick<Parameters<typeof prepareNativeWorkspaceSync>[0], "lease">;
|
|
isDotRun: boolean;
|
|
projectRepositoryPaths: string[];
|
|
onCheckpoint?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onCheckpoint"];
|
|
onPhase?: Parameters<typeof prepareNativeWorkspaceSync>[0]["onPhase"];
|
|
};
|
|
config: {
|
|
runtimeConfig: Record<string, unknown>;
|
|
runtimeSkillEntries: Parameters<typeof buildNativeRuntimeContext>[0]["runtimeSkillEntries"];
|
|
runScopedMentionedSkillKeys: string[];
|
|
instructionCopy: Awaited<ReturnType<ReturnType<typeof agentInstructionWorkingCopyService>["prepare"]>>;
|
|
agentIdentity: { keyId: string } | undefined;
|
|
issueAssigneeOverrides: { adapterConfig: Record<string, unknown> | null } | null;
|
|
resolvedConfig: Record<string, unknown>;
|
|
managedAiRuntime: Awaited<ReturnType<typeof prepareManagedAiRuntime>> | undefined;
|
|
};
|
|
recovery: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim };
|
|
trace: { providerTraceRequested: boolean; providerTraceCapture: { metadata: { id: string } } | null };
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
|
|
runtimeEnv: Record<string, string | undefined>;
|
|
beforeNativeRuntimeSelection?: (runId: string) => Promise<void>;
|
|
onAdapterEvent: (event: AdapterRuntimeEvent) => Promise<void>;
|
|
// Teardown must observe these updates even when a later preparation step throws.
|
|
onNativeLifecycleSelected: (telemetry: NativeLifecycleTelemetry) => void;
|
|
onProviderResourceDisposition: (disposition: ProviderResourceDisposition | undefined) => void;
|
|
onNativeOwnershipHeld: () => void;
|
|
stopControllerLease: () => void;
|
|
}
|
|
|
|
export type HeartbeatRuntimeSelectionResult = { selected: false } | {
|
|
selected: true;
|
|
nativeExecution: NativeExecutionInput | null;
|
|
nativeRunnerInstanceId: string | null;
|
|
getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
|
nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>>;
|
|
};
|
|
|
|
export interface NativeSandboxLifecycle {
|
|
runnerProcess: "per_turn" | "warm";
|
|
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
|
failoverBackup: "verified";
|
|
}
|
|
|
|
export function resolveReusableSandboxLifecycle(input: {
|
|
lifecyclePolicy:
|
|
| { mode: "per_turn"; idleTimeoutMs: null }
|
|
| { mode: "warm"; idleTimeoutMs: number };
|
|
target: {
|
|
kind: "local" | "remote";
|
|
transport?: string;
|
|
reusableLeaseConfigured?: boolean;
|
|
effectiveCapabilities?: { reusableLeases: boolean } | null;
|
|
} | null;
|
|
}): NativeSandboxLifecycle | null {
|
|
if (input.target?.kind !== "remote" || input.target.transport !== "sandbox") {
|
|
return null;
|
|
}
|
|
const reusableLease =
|
|
input.target.reusableLeaseConfigured === true &&
|
|
input.target.effectiveCapabilities?.reusableLeases === true;
|
|
if (input.lifecyclePolicy.mode === "warm" && !reusableLease) {
|
|
throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease");
|
|
}
|
|
return {
|
|
runnerProcess: input.lifecyclePolicy.mode,
|
|
sandboxResource:
|
|
input.lifecyclePolicy.mode === "warm"
|
|
? "keep_running"
|
|
: reusableLease
|
|
? "stop_and_reuse"
|
|
: "destroy_after_turn",
|
|
failoverBackup: "verified",
|
|
};
|
|
}
|
|
|
|
export function resolveNativeSandboxLifecycle(input: {
|
|
adapterType: string;
|
|
lifecyclePolicy:
|
|
| { mode: "per_turn"; idleTimeoutMs: null }
|
|
| { mode: "warm"; idleTimeoutMs: number };
|
|
target: {
|
|
kind: "local" | "remote";
|
|
transport?: string;
|
|
reusableLeaseConfigured?: boolean;
|
|
effectiveCapabilities?: { reusableLeases: boolean } | null;
|
|
} | null;
|
|
}): NativeSandboxLifecycle | null {
|
|
if (
|
|
input.adapterType !== "paperclip_runner" ||
|
|
input.target?.kind !== "remote" ||
|
|
input.target.transport !== "sandbox"
|
|
)
|
|
return null;
|
|
return resolveReusableSandboxLifecycle(input);
|
|
}
|
|
|
|
export async function postNativeModelFallbackWarning(input: {
|
|
issuesSvc: Pick<ReturnType<typeof issueService>, "addComment">;
|
|
onEvent: (event: AdapterRuntimeEvent) => Promise<void>;
|
|
issueId: string;
|
|
runId: string;
|
|
requestedModel: string | null;
|
|
effectiveModel: string | null;
|
|
codexCliVersion: string;
|
|
}): Promise<void> {
|
|
const message = `Using ${input.effectiveModel} because the sandbox's Codex ${input.codexCliVersion} does not support ${input.requestedModel}. Work will continue with the compatible model. Update the sandbox's Codex CLI to use the requested model.`;
|
|
await input.onEvent({
|
|
eventType: "runner.model_fallback",
|
|
stream: "system",
|
|
level: "warn",
|
|
message,
|
|
payload: {
|
|
requestedModel: input.requestedModel,
|
|
effectiveModel: input.effectiveModel,
|
|
codexCliVersion: input.codexCliVersion,
|
|
},
|
|
});
|
|
await input.issuesSvc.addComment(input.issueId, message, { runId: input.runId }, {
|
|
authorType: "system",
|
|
presentation: {
|
|
kind: "system_notice",
|
|
tone: "warning",
|
|
title: `Using ${input.effectiveModel}`,
|
|
density: "compact",
|
|
detailsDefaultOpen: false,
|
|
},
|
|
});
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
|
}
|
|
|
|
/** Prepare and persist the selected runtime before the executor dispatches provider work. */
|
|
export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSelectionInput): Promise<HeartbeatRuntimeSelectionResult> {
|
|
const { run, agent, nativeRuntimeResolution, issuesSvc, runtimeEnv, beforeNativeRuntimeSelection, onAdapterEvent,
|
|
onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input;
|
|
const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext,
|
|
paperclipWakePayload, acceptedPlanContinuationWake } = input.task;
|
|
const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams,
|
|
taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session;
|
|
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace;
|
|
const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config;
|
|
const runOptions = input.recovery;
|
|
const { providerTraceRequested, providerTraceCapture } = input.trace;
|
|
let nativeExecution: NativeExecutionInput | null = null;
|
|
let getNativeFreshSessionHandoff: (() => Promise<string | null>) | undefined;
|
|
let nativeRunnerInstanceId: string | null = null;
|
|
let nativeWorkspaceSync: Awaited<ReturnType<typeof prepareNativeWorkspaceSync>> = null;
|
|
let providerResourceDispositionForRun: ProviderResourceDisposition | undefined;
|
|
if (nativeRuntimeResolution.kind === "native") {
|
|
if (!issueRef) {
|
|
throw new Error("native_runtime_ineligible: issue is required");
|
|
}
|
|
const nativeExecutionWorkspaceId =
|
|
persistedExecutionWorkspace?.id ?? run.id;
|
|
const nativeReviewContext = readNativeReviewAssignmentContext(context);
|
|
const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, {
|
|
companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id,
|
|
contextSnapshot: nativeReviewContext,
|
|
})) : null;
|
|
if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available");
|
|
const nativeReviewRequest = nativeReview
|
|
? buildNativeReviewRequest({
|
|
title: nativeReview.interaction.title,
|
|
summary: nativeReview.interaction.summary,
|
|
payload: nativeReview.interaction.payload,
|
|
})
|
|
: null;
|
|
const persistedContract = run.completionContractId
|
|
? await measureRunPhase("runtime_selection.contract_read", () => db
|
|
.select()
|
|
.from(completionContracts)
|
|
.where(
|
|
and(
|
|
eq(completionContracts.id, run.completionContractId!),
|
|
eq(completionContracts.companyId, agent.companyId),
|
|
eq(completionContracts.issueId, issueRef.id),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null))
|
|
: null;
|
|
// Only a server-verified human resolution may supply a current answer
|
|
// reference. Tool/agent results and generated summaries stay evidence.
|
|
const currentHumanResponseId = !nativeReviewRequest
|
|
? executionContinuation?.humanResponses?.find(
|
|
(response) => response.id === executionContinuation.trigger.interactionId,
|
|
)?.id
|
|
: undefined;
|
|
const immediateCompletion = (() => {
|
|
if (nativeReviewRequest) return { requests: [nativeReviewRequest], sources: [null] };
|
|
const { requests, sources } = nativeCompletionRequestsWithSources(
|
|
safeWakeComments.length > 0
|
|
? safeWakeComments
|
|
: safeWakeCommentContext?.body
|
|
? [safeWakeCommentContext]
|
|
: [],
|
|
{
|
|
requiredFullWakeCommentCount:
|
|
paperclipWakePayload?.fallbackFetchNeeded === true &&
|
|
CHAT_PROVIDERS.some(
|
|
(provider) =>
|
|
provider ===
|
|
paperclipWakePayload.externalChatProvider,
|
|
) &&
|
|
Array.isArray(paperclipWakePayload.commentIds)
|
|
? paperclipWakePayload.commentIds.length
|
|
: undefined,
|
|
},
|
|
);
|
|
// Preserve every admitted pending chat request while also
|
|
// retaining newer user direction materialized by recovery.
|
|
// A file-only wake must not inherit an old task objective.
|
|
const latestComment =
|
|
executionContinuation?.messages.findLast(
|
|
(message) =>
|
|
message.authorType === "user" &&
|
|
!message.createdByRunId &&
|
|
!message.deleted &&
|
|
message.body.trim().length > 0,
|
|
);
|
|
const latestRequest = latestComment?.body;
|
|
if (
|
|
latestRequest &&
|
|
!requests.some(
|
|
(request) => request === latestRequest.trim(),
|
|
)
|
|
) {
|
|
requests.push(latestRequest.trim());
|
|
sources.push(nativeCompletionSource("comment", latestComment!.id, latestRequest));
|
|
}
|
|
return { requests: requests.length > 0 ? requests : undefined, sources };
|
|
})();
|
|
// Rebuilding a default contract is not a change in user direction.
|
|
// In particular, an upgraded checkpoint may have an intentionally
|
|
// authored contract and no continuation envelope yet.
|
|
const completionContract =
|
|
persistedContract && persistedNativeExecutionInput
|
|
? {
|
|
row: persistedContract,
|
|
contract: persistedContract.contractJson as never,
|
|
}
|
|
: await measureRunPhase("runtime_selection.contract_ensure", () => ensureNativeCompletionContract({
|
|
db,
|
|
companyId: agent.companyId,
|
|
issue: issueRef,
|
|
actorId: agent.id,
|
|
immediateRequest:
|
|
nativeReviewRequest ?? (currentHumanResponseId
|
|
? null
|
|
: executionContinuation?.objective ?? safeWakeCommentContext?.body ?? null),
|
|
// The ordinary initial objective is selected by the server-owned
|
|
// continuation envelope. Carry its explicit description source
|
|
// through the singular-request compatibility path; do not infer
|
|
// provenance for review requests, answers, or wake fallbacks.
|
|
immediateRequestSource: nativeImmediateObjectiveSource({
|
|
issueId: issueRef.id,
|
|
objectiveSource: executionContinuation?.objectiveSource,
|
|
excluded: Boolean(nativeReviewRequest || currentHumanResponseId),
|
|
}),
|
|
humanResponseId: currentHumanResponseId,
|
|
immediateRequests: immediateCompletion.requests,
|
|
immediateRequestSources: immediateCompletion.sources,
|
|
}));
|
|
const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString(
|
|
taskSessionDecodedParams?.sessionId,
|
|
);
|
|
// Compatibility for native retry rows created before same-run restart
|
|
// recovery existed. Only an entirely unused replacement row may
|
|
// inherit its source checkpoint; any process/provider evidence on the
|
|
// replacement makes the ownership ambiguous and therefore ineligible.
|
|
const legacyRetrySource =
|
|
run.retryOfRunId && !isFailedChatRunRetry
|
|
? await measureRunPhase("runtime_selection.resume", () => db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId,
|
|
runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
status: heartbeatRuns.status,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.id, run.retryOfRunId!),
|
|
eq(heartbeatRuns.companyId, agent.companyId),
|
|
eq(heartbeatRuns.agentId, agent.id),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null))
|
|
: null;
|
|
const nativeBootstrapHasProviderEvidence =
|
|
legacyRetrySource ||
|
|
run.nativeSessionId ||
|
|
persistedNativeExecutionInput
|
|
? await measureRunPhase("runtime_selection.resume", () => db
|
|
.select({ id: heartbeatRunEvents.id })
|
|
.from(heartbeatRunEvents)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRunEvents.runId, run.id),
|
|
inArray(heartbeatRunEvents.eventType, [
|
|
"harness.ready",
|
|
"session.started",
|
|
"session.resumed",
|
|
"session.updated",
|
|
"turn.started",
|
|
"provider.event",
|
|
"provider.rpc_result",
|
|
]),
|
|
),
|
|
)
|
|
.limit(1)
|
|
.then((rows) => rows.length > 0))
|
|
: false;
|
|
const compatibleLegacyRetrySource =
|
|
!managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({
|
|
replacement: run,
|
|
source: legacyRetrySource,
|
|
hasProviderEvents: nativeBootstrapHasProviderEvidence,
|
|
})
|
|
? legacyRetrySource
|
|
: null;
|
|
const legacyRetrySessionId =
|
|
compatibleLegacyRetrySource?.nativeSessionId;
|
|
const taskResumeRunId =
|
|
taskSessionForRun?.lastRunId &&
|
|
taskSessionForRun.lastRunId !== run.id &&
|
|
isNativeSessionId(taskNativeSessionId)
|
|
? taskSessionForRun.lastRunId
|
|
: null;
|
|
const resumableTaskSessionId = isDotRun ? null : taskResumeRunId
|
|
? taskNativeSessionId
|
|
: (legacyRetrySessionId ?? null);
|
|
const requestedNativeSessionId =
|
|
run.nativeSessionId ?? resumableTaskSessionId;
|
|
// A task-session lastRunId can lag a failed turn or point at an older
|
|
// normalized session. Find the newest exact-session authority instead.
|
|
// Rows that already acquired provider authority are progress barriers,
|
|
// even when they do not contain a usable checkpoint.
|
|
const previousNativeRun =
|
|
requestedNativeSessionId &&
|
|
isUnusedNativeSessionBootstrap(
|
|
run,
|
|
nativeBootstrapHasProviderEvidence,
|
|
)
|
|
? await measureRunPhase("runtime_selection.resume", () => findNativeSessionResumeRun(db, {
|
|
companyId: agent.companyId,
|
|
agentId: agent.id,
|
|
issueId: issueRef.id,
|
|
normalizedSessionId: requestedNativeSessionId,
|
|
currentRunId: run.id,
|
|
beforeCreatedAt: run.createdAt,
|
|
}))
|
|
: null;
|
|
nativeRunnerInstanceId =
|
|
previousNativeRun?.runnerInstanceId &&
|
|
previousNativeRun.nativeSessionId ===
|
|
(run.nativeSessionId ?? resumableTaskSessionId)
|
|
? previousNativeRun.runnerInstanceId
|
|
: (run.runnerInstanceId ?? randomUUID());
|
|
let nativeSessionId =
|
|
run.nativeSessionId ?? resumableTaskSessionId ?? randomUUID();
|
|
let nativeResumeCheckpoint: ReturnType<
|
|
typeof rebindNativeSessionCheckpoint
|
|
> = null;
|
|
const agentLifecyclePolicy =
|
|
parseObject(agent.adapterConfig).lifecycleMode === "warm"
|
|
? {
|
|
mode: "warm" as const,
|
|
idleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs(
|
|
parseObject(agent.adapterConfig).idleTimeoutMs,
|
|
),
|
|
}
|
|
: { mode: "per_turn" as const, idleTimeoutMs: null };
|
|
const environmentLifecyclePolicy =
|
|
executionTarget?.kind === "remote" &&
|
|
executionTarget.transport === "sandbox"
|
|
? (executionTarget.runnerLifecyclePolicy ?? null)
|
|
: null;
|
|
// Codex refreshes its durable auth home; Copilot owns a fenced home
|
|
// and an explicitly bound token. Both retire on credential generation
|
|
// changes. Other managed harnesses require per-turn cleanup.
|
|
const supportsManagedWarmSession = agent.adapterType === "paperclip_runner" &&
|
|
supportsManagedNativeWarmSession({ ...nativeRuntimeResolution.profile, acpxAgent: parseObject(agent.adapterConfig).acpxAgent }, managedAiRuntime?.attribution);
|
|
const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ??
|
|
(["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession
|
|
? { mode: "per_turn" as const, idleTimeoutMs: null }
|
|
: environmentLifecyclePolicy ?? agentLifecyclePolicy);
|
|
if (
|
|
effectiveLifecyclePolicy.mode === "warm" &&
|
|
executionTarget?.kind === "remote" &&
|
|
executionTarget.transport === "sandbox" &&
|
|
(executionTarget.reusableLeaseConfigured !== true ||
|
|
executionTarget.effectiveCapabilities?.reusableLeases !== true)
|
|
) {
|
|
throw new Error("runner_warm_environment_requires_reusable_lease");
|
|
}
|
|
const persistedProfile = persistedRunnerProfile;
|
|
if (persistedNativeExecutionInput) {
|
|
nativeExecution = persistedNativeExecutionInput;
|
|
if (
|
|
nativeExecution.binding.companyId !== agent.companyId ||
|
|
nativeExecution.binding.runId !== run.id ||
|
|
nativeExecution.binding.issueId !== issueRef.id ||
|
|
nativeExecution.binding.agentId !== agent.id ||
|
|
nativeExecution.binding.executionWorkspaceId !==
|
|
nativeExecutionWorkspaceId ||
|
|
nativeExecution.completionContract.id !==
|
|
completionContract.row.id ||
|
|
nativeExecution.completionContract.sha256 !==
|
|
completionContract.row.canonicalSha256
|
|
)
|
|
throw new Error(
|
|
"native_execution_input_persisted_binding_mismatch",
|
|
);
|
|
if ("runtimeContext" in nativeExecution) {
|
|
const savedVersions = parseObject(run.contextSnapshot?.skillVersions);
|
|
const accessibleKeys = new Set(runtimeSkillEntries.filter(entry => entry.sourceStatus !== "missing").map(entry => entry.key));
|
|
if (nativeExecution.runtimeContext.skills.some(skill =>
|
|
Object.prototype.hasOwnProperty.call(savedVersions, skill.key) && !accessibleKeys.has(skill.key))) {
|
|
throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", {
|
|
configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" },
|
|
});
|
|
}
|
|
}
|
|
// Recover only the originally admitted request. A stored idle
|
|
// checkpoint can precede an already-started provider turn, so even
|
|
// apparent idleness is not authority to rewrite its contract.
|
|
// Newer user direction retains its separate durable wakeup cause.
|
|
// A failed pre-bootstrap attempt may have persisted its immutable
|
|
// input before discovering that lastRunId no longer names this
|
|
// session. Restore only an exactly compatible prior checkpoint;
|
|
// never rewrite the admitted input or skip current provider work.
|
|
if (
|
|
previousNativeRun &&
|
|
isUnusedNativeSessionBootstrap(
|
|
run,
|
|
nativeBootstrapHasProviderEvidence,
|
|
)
|
|
) {
|
|
nativeResumeCheckpoint = rebindNativeSessionCheckpoint({
|
|
previousRun: previousNativeRun,
|
|
currentExecution: nativeExecution,
|
|
executionTargetKind: executionTarget?.kind ?? "local",
|
|
});
|
|
}
|
|
if (nativeExecution.provider.kind === "claude_managed") {
|
|
const recoveryProfileId = nativeExecution.provider.managedProfile.profileId;
|
|
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(
|
|
db,
|
|
).requireQualified(
|
|
agent.companyId,
|
|
recoveryProfileId,
|
|
));
|
|
assertManagedProfileRecoveryBinding({
|
|
adapterConfig: agent.adapterConfig,
|
|
snapshot: nativeExecution.provider.managedProfile,
|
|
stored: recoveryProfile,
|
|
});
|
|
}
|
|
if (nativeExecution.provider.kind === "aws_agentcore") {
|
|
const recoveryProfileId = nativeExecution.provider.agentCoreProfile.profileId;
|
|
const recoveryProfile = await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(
|
|
db,
|
|
).requireQualified(
|
|
agent.companyId,
|
|
recoveryProfileId,
|
|
"aws_bedrock_agentcore_harness",
|
|
));
|
|
assertAgentCoreProfileRecoveryBinding({
|
|
snapshot: nativeExecution.provider.agentCoreProfile,
|
|
stored: recoveryProfile,
|
|
});
|
|
}
|
|
} else {
|
|
const interactionId = readNonEmptyString(context.interactionId);
|
|
const interactionResponses = context[
|
|
EXTERNAL_CHAT_QUESTION_RESPONSE_KEY
|
|
]
|
|
? await measureRunPhase("runtime_selection.interactions", () => materializeExternalChatQuestionResponseInput({
|
|
db,
|
|
binding: {
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
runId: run.id,
|
|
agentId: agent.id,
|
|
},
|
|
contextSnapshot: context,
|
|
}))
|
|
: await measureRunPhase("runtime_selection.interactions", () => materializeNativeInteractionResponses({
|
|
db,
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
runId: run.id,
|
|
agentId: agent.id,
|
|
interactionIds: Array.isArray(context.interactionIds)
|
|
? [
|
|
...new Set([
|
|
...(interactionId ? [interactionId] : []),
|
|
...context.interactionIds.filter(
|
|
(id): id is string => typeof id === "string",
|
|
),
|
|
]),
|
|
]
|
|
: interactionId
|
|
? [interactionId]
|
|
: [],
|
|
}));
|
|
const runnerAdapterConfig = parseObject(agent.adapterConfig);
|
|
const managedProfile =
|
|
nativeRuntimeResolution.profile.backend ===
|
|
"claude_managed_agents_api"
|
|
? await measureRunPhase("runtime_selection.profile", () => managedAgentProfileService(db).requireQualified(
|
|
agent.companyId,
|
|
readNonEmptyString(runnerAdapterConfig.managedProfileId) ??
|
|
"",
|
|
))
|
|
: null;
|
|
const agentCoreProfile =
|
|
nativeRuntimeResolution.profile.backend ===
|
|
"aws_agentcore_harness_api"
|
|
? await measureRunPhase("runtime_selection.profile", () => remoteAgentProfileService(db).requireQualified(
|
|
agent.companyId,
|
|
readNonEmptyString(
|
|
runnerAdapterConfig.agentCoreProfileId,
|
|
) ?? "",
|
|
"aws_bedrock_agentcore_harness",
|
|
))
|
|
: null;
|
|
if (managedProfile) {
|
|
const rawApiKeyBinding = parseObject(
|
|
runnerAdapterConfig.env,
|
|
).ANTHROPIC_API_KEY;
|
|
const boundSecretId =
|
|
typeof rawApiKeyBinding === "object" &&
|
|
rawApiKeyBinding !== null
|
|
? readNonEmptyString(
|
|
(rawApiKeyBinding as Record<string, unknown>).secretId,
|
|
)
|
|
: null;
|
|
if (boundSecretId !== managedProfile.apiKeySecretId) {
|
|
throw new ConfigurationIncompleteFailure(
|
|
"configuration incomplete: Claude Managed profile API key is not bound at env.ANTHROPIC_API_KEY",
|
|
{
|
|
configurationIncomplete: {
|
|
reason: "managed_agent_profile_secret_binding_mismatch",
|
|
companyId: agent.companyId,
|
|
agentId: agent.id,
|
|
profileId: managedProfile.id,
|
|
requiredEnvKeys: ["ANTHROPIC_API_KEY"],
|
|
},
|
|
},
|
|
);
|
|
}
|
|
}
|
|
const executionMode =
|
|
issueRef.workMode === "planning" && !isConversation(issueContext) && !acceptedPlanContinuationWake
|
|
? ("plan" as const)
|
|
: ("default" as const);
|
|
const pinnedPlan =
|
|
executionMode === "plan"
|
|
? await documentService(db).getIssueDocumentByKey(
|
|
issueRef.id,
|
|
"plan",
|
|
)
|
|
: null;
|
|
const pinnedReviewContext =
|
|
executionMode === "plan"
|
|
? await buildPlanReviewContext({
|
|
db,
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
issueWorkMode: issueRef.workMode,
|
|
interactionId: readNonEmptyString(context.interactionId),
|
|
})
|
|
: null;
|
|
const pinnedPlanMarkdown = pinnedPlan?.body ?? "";
|
|
const museBinding = nativeRuntimeResolution.profile.backend === "muse_external"
|
|
? await museRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig).museBindingId ?? "")) : undefined;
|
|
const dotBinding = ["openai_dot_mcp", "grok_bot_mcp"].includes(nativeRuntimeResolution.profile.backend)
|
|
? await dotRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig)[nativeRuntimeResolution.profile.backend === "grok_bot_mcp" ? "externalBindingId" : "dotBindingId"] ?? "")) : undefined;
|
|
const nativeRuntimeContext = await buildNativeRuntimeContext({
|
|
db,
|
|
agent,
|
|
runId: run.id,
|
|
runtimeConfig,
|
|
runtimeSkillEntries,
|
|
instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined,
|
|
});
|
|
getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff;
|
|
const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "codex_app_server"
|
|
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime" && agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig)) === "claude_local")
|
|
|| nativeRuntimeResolution.profile.backend === "opencode_server"
|
|
|| (nativeRuntimeResolution.profile.backend === "acpx_runtime"
|
|
&& parseObject(agent.adapterConfig).acpxAgent === "copilot")
|
|
? projectPaperclipRunnerTaskConfig(
|
|
nativeRuntimeResolution.profile.backend,
|
|
agent.adapterConfig,
|
|
issueAssigneeOverrides?.adapterConfig,
|
|
managedAiRuntime ? readNonEmptyString(resolvedConfig.model) ?? undefined : undefined,
|
|
)
|
|
: agent.adapterConfig;
|
|
const requestedNativeProvider = resolvePaperclipRunnerNativeProviderInput({
|
|
backend: nativeRuntimeResolution.profile.backend,
|
|
adapterConfig: nativeProviderConfig, managedProfile, agentCoreProfile, dotBinding, museBinding,
|
|
});
|
|
const codexCliVersion = agent.adapterType === "paperclip_runner" && requestedNativeProvider.provider === "codex"
|
|
? await readRemoteCodexModelCliVersion({
|
|
model: requestedNativeProvider.model,
|
|
target: executionTarget,
|
|
remoteCodexPath: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH,
|
|
remoteCodexNpmSpec: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC,
|
|
}) : null;
|
|
const nativeProvider = codexCliVersion
|
|
? resolvePaperclipRunnerNativeProviderInput({
|
|
backend: nativeRuntimeResolution.profile.backend,
|
|
adapterConfig: nativeProviderConfig, codexCliVersion, managedProfile, agentCoreProfile, dotBinding, museBinding,
|
|
}) : requestedNativeProvider;
|
|
if (nativeProvider.model !== requestedNativeProvider.model) {
|
|
await postNativeModelFallbackWarning({
|
|
issuesSvc, onEvent: onAdapterEvent, issueId: issueRef.id, runId: run.id,
|
|
requestedModel: requestedNativeProvider.model, effectiveModel: nativeProvider.model,
|
|
codexCliVersion: codexCliVersion!,
|
|
});
|
|
}
|
|
const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) =>
|
|
buildNativeExecutionInput({
|
|
agentKeyId: agentIdentity?.keyId,
|
|
companyId: agent.companyId,
|
|
runId: run.id,
|
|
issue: nativeReviewRequest ? { ...issueRef, title: `Review: ${issueRef.title}`, description: nativeReviewRequest } : issueRef,
|
|
taskPrompt: [
|
|
nativeReviewRequest ?? readNonEmptyString(
|
|
selectPaperclipTaskMarkdown(context, {
|
|
resumedSession: false,
|
|
includeCommunicationGuidance: false,
|
|
}),
|
|
) ??
|
|
`# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`,
|
|
!["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend) && projectRepositoryPaths.length > 0
|
|
? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}`
|
|
: null,
|
|
].filter(Boolean).join("\n\n"),
|
|
initialCommunicationGuidance: nativeReviewRequest ? null : readNonEmptyString(context.paperclipTaskCommunicationGuidance),
|
|
wakePayload: context.paperclipWake,
|
|
caseContext: nativeReviewRequest ? null : context.paperclipCase as CaseExecutionContext | undefined,
|
|
turnContext: context.paperclipTurnContext,
|
|
githubInstructionSkillKeys: runScopedMentionedSkillKeys,
|
|
resumedSession,
|
|
previousTurn: (() => {
|
|
if (!previousNativeRun || nativeReviewRequest) return null;
|
|
try {
|
|
const previousInput = parseNativeExecutionInput(parseObject(previousNativeRun.runnerProfileJson).nativeExecutionInput);
|
|
const previousTask = previousInput.task;
|
|
const previousWake = normalizePaperclipWakePayload(parseObject(previousNativeRun.contextSnapshot).paperclipWake);
|
|
const githubReviewConfigurationKeys = previousWake?.githubReviewConfigurations.map((config) => config.key);
|
|
if (paperclipWakePayload?.externalChatProvider || previousWake?.externalChatProvider) {
|
|
// External native inputs use a neutral task title. Compare
|
|
// the saved canonical brief so old provider text is not
|
|
// repeated as a change, while genuine edits still arrive.
|
|
const savedIssue = parseObject(parseObject(previousNativeRun.contextSnapshot).paperclipIssue);
|
|
if (savedIssue.id !== issueRef.id || typeof savedIssue.title !== "string" ||
|
|
(savedIssue.description !== null && typeof savedIssue.description !== "string")) return null;
|
|
return { runId: previousNativeRun.id, task: { title: savedIssue.title, description: savedIssue.description },
|
|
githubReviewConfigurationKeys, nativeExecutionInput: previousInput,
|
|
};
|
|
}
|
|
return { runId: previousNativeRun.id, task: previousTask, githubReviewConfigurationKeys, nativeExecutionInput: previousInput };
|
|
} catch {
|
|
// An invalid prior snapshot must use the fresh bootstrap.
|
|
return null;
|
|
}
|
|
})(),
|
|
conversationMode: context.conversationMode === true,
|
|
agentId: agent.id,
|
|
workspace: {
|
|
// Projectless paperclip_runner tasks still have a resolved local cwd. Bind that
|
|
// transient workspace to the run id so the native input remains durable and replayable
|
|
// without fabricating a project-scoped execution_workspaces row.
|
|
id: nativeExecutionWorkspaceId,
|
|
cwd: executionWorkspace.cwd,
|
|
repoUrl: executionWorkspace.repoUrl,
|
|
repoRef: executionWorkspace.repoRef,
|
|
branchName: executionWorkspace.branchName,
|
|
},
|
|
normalizedSessionId,
|
|
executionMode,
|
|
planningContext:
|
|
executionMode === "plan"
|
|
? {
|
|
documentId: pinnedPlan?.id ?? null,
|
|
baseRevisionId:
|
|
pinnedPlan?.latestRevisionId ?? null,
|
|
baseRevisionNumber:
|
|
pinnedPlan?.latestRevisionNumber ?? 0,
|
|
markdown: pinnedPlanMarkdown,
|
|
sha256: createHash("sha256")
|
|
.update(pinnedPlanMarkdown)
|
|
.digest("hex"),
|
|
reviewContext: pinnedReviewContext
|
|
? (structuredClone(
|
|
pinnedReviewContext,
|
|
) as unknown as Record<string, unknown>)
|
|
: {},
|
|
}
|
|
: null,
|
|
...nativeProvider,
|
|
lifecyclePolicy: effectiveLifecyclePolicy,
|
|
interactionResponses,
|
|
completionContract: {
|
|
id: completionContract.row.id,
|
|
sha256: completionContract.row.canonicalSha256,
|
|
schemaVersion: completionContract.row.schemaVersion,
|
|
contract: completionContract.contract,
|
|
sources: "sources" in completionContract ? completionContract.sources : undefined,
|
|
},
|
|
runtimeContext: nativeRuntimeContext,
|
|
});
|
|
const backendDescriptor = await measureRunPhase("runtime_selection.provider_discovery", () => describeRunnerdNativeSessionBackend(buildExecution({
|
|
normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null,
|
|
})));
|
|
const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({
|
|
previousRun: previousNativeRun,
|
|
normalizedSessionId: nativeSessionId,
|
|
executionTargetKind: executionTarget?.kind ?? "local",
|
|
toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true,
|
|
refreshTools: context.refreshTools === true,
|
|
buildExecution,
|
|
});
|
|
nativeExecution = nativeExecutionWithCheckpoint.execution;
|
|
nativeResumeCheckpoint = nativeExecutionWithCheckpoint.checkpoint;
|
|
if (
|
|
nativeSessionId !==
|
|
nativeExecutionWithCheckpoint.normalizedSessionId
|
|
) {
|
|
nativeRunnerInstanceId = randomUUID();
|
|
}
|
|
nativeSessionId = nativeExecutionWithCheckpoint.normalizedSessionId;
|
|
}
|
|
const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({
|
|
adapterType: agent.adapterType,
|
|
lifecyclePolicy: nativeExecution.session.lifecyclePolicy,
|
|
target: executionTarget,
|
|
});
|
|
if (nativeSandboxLifecycle) {
|
|
onNativeLifecycleSelected({
|
|
provider: nativeExecution.provider.kind,
|
|
harness: nativeExecution.session.driverKind,
|
|
lifecycleMode: nativeExecution.session.lifecyclePolicy.mode,
|
|
sandboxResource: nativeSandboxLifecycle.sandboxResource,
|
|
});
|
|
const selectedLifecycleSpan = getStartupTracer(
|
|
"paperclip.environment-lifecycle",
|
|
).startSpan("sandbox.lifecycle.selected", {
|
|
attributes: {
|
|
"paperclip.native.span.provider": nativeExecution.provider.kind,
|
|
"paperclip.native.span.harness":
|
|
nativeExecution.session.driverKind,
|
|
"paperclip.native.span.lifecycle_mode":
|
|
nativeExecution.session.lifecyclePolicy.mode,
|
|
"paperclip.native.span.sandbox_resource":
|
|
nativeSandboxLifecycle.sandboxResource,
|
|
"paperclip.native.span.outcome": "selected",
|
|
"paperclip.native.span.bytes_transferred": 0,
|
|
},
|
|
});
|
|
selectedLifecycleSpan.end();
|
|
}
|
|
providerResourceDispositionForRun =
|
|
nativeSandboxLifecycle?.sandboxResource === "keep_running"
|
|
? "keep_running"
|
|
: nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse"
|
|
? "stop_and_retain"
|
|
: nativeSandboxLifecycle?.sandboxResource ===
|
|
"destroy_after_turn"
|
|
? "destroy"
|
|
: undefined;
|
|
onProviderResourceDisposition(providerResourceDispositionForRun);
|
|
await beforeNativeRuntimeSelection?.(run.id);
|
|
const nativeSelected = await measureRunTransaction(db, "runtime_selection", async (tx) => {
|
|
const lockedRun = await measureRunPhase("runtime_selection.row_lock", () => tx
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, run.id))
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null));
|
|
if (!lockedRun) throw new Error("native_runtime_run_missing");
|
|
// Cancellation and runtime selection serialize on this row. A
|
|
// stopped preparation must never create a new native coordinator.
|
|
if (lockedRun.status !== "running" || lockedRun.resultJson?.startupCancellation) return false;
|
|
if (lockedRun.runtimeMode === "legacy" && lockedRun.controllerBootId &&
|
|
!(await renewLegacyControllerLease(tx as unknown as Db, lockedRun))) {
|
|
onNativeOwnershipHeld();
|
|
return false;
|
|
}
|
|
if (
|
|
lockedRun.runtimeModeResolvedAt &&
|
|
lockedRun.runtimeMode !== "native"
|
|
) {
|
|
throw new Error("native_runtime_mode_conflict");
|
|
}
|
|
const lockedProfile = parseObject(lockedRun.runnerProfileJson);
|
|
const persistedNativeSessionId =
|
|
await measureRunPhase("runtime_selection.bootstrap", () => prepareNativeSessionBootstrapPersistence(tx, {
|
|
run: lockedRun,
|
|
selectedSessionId: nativeSessionId,
|
|
execution: nativeExecution!,
|
|
restoringCheckpoint: nativeResumeCheckpoint !== null,
|
|
}));
|
|
await measureRunPhase("runtime_selection.run_write", () => tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runtimeMode: "native",
|
|
runtimeModeResolverVersion:
|
|
lockedRun.runtimeModeResolverVersion ??
|
|
nativeRuntimeResolution.resolverVersion,
|
|
runtimeModeReason:
|
|
lockedRun.runtimeModeReason ?? nativeRuntimeResolution.reason,
|
|
runtimeModeResolvedAt:
|
|
lockedRun.runtimeModeResolvedAt ?? new Date(),
|
|
runnerProfileJson: {
|
|
...nativeRuntimeResolution.profile,
|
|
...lockedProfile,
|
|
nativePreparationFencingVersion: !lockedRun.runnerInstanceId && !lockedRun.nativeSessionId
|
|
? NATIVE_PREPARATION_FENCING_VERSION
|
|
: lockedProfile.nativePreparationFencingVersion,
|
|
...(lockedProfile.nativeExecutionInput
|
|
? {}
|
|
: { recoveryEventInventoryVersion: 1 }),
|
|
...(providerTraceRequested
|
|
? {
|
|
providerTrace: {
|
|
mode: "raw",
|
|
traceId: providerTraceCapture?.metadata.id ?? null,
|
|
maxBytes: PROVIDER_TRACE_MAX_BYTES,
|
|
},
|
|
}
|
|
: {}),
|
|
nativeExecutionInput:
|
|
lockedProfile.nativeExecutionInput ?? nativeExecution,
|
|
nativeToolContractFingerprint:
|
|
nativeToolContractFingerprintForTarget(
|
|
executionTarget?.kind ?? "local",
|
|
),
|
|
...(lockedProfile.sessionCheckpoint != null
|
|
? { sessionCheckpoint: lockedProfile.sessionCheckpoint }
|
|
: nativeResumeCheckpoint
|
|
? {
|
|
sessionCheckpoint:
|
|
nativeResumeCheckpoint as unknown as Record<
|
|
string,
|
|
unknown
|
|
>,
|
|
}
|
|
: {}),
|
|
},
|
|
runnerInstanceId:
|
|
previousNativeRun?.runnerInstanceId &&
|
|
persistedNativeSessionId === previousNativeRun.nativeSessionId
|
|
? previousNativeRun.runnerInstanceId
|
|
: lockedRun.nativeSessionId !== persistedNativeSessionId
|
|
? nativeRunnerInstanceId
|
|
: (lockedRun.runnerInstanceId ?? nativeRunnerInstanceId),
|
|
nativeSessionId: persistedNativeSessionId,
|
|
processPid:
|
|
lockedRun.processPid ??
|
|
(previousNativeRun?.nativeSessionId === nativeSessionId
|
|
? previousNativeRun.processPid
|
|
: null),
|
|
processGroupId:
|
|
lockedRun.processGroupId ??
|
|
(previousNativeRun?.nativeSessionId === nativeSessionId
|
|
? previousNativeRun.processGroupId
|
|
: null),
|
|
processStartedAt:
|
|
lockedRun.processStartedAt ??
|
|
(previousNativeRun?.nativeSessionId === nativeSessionId
|
|
? previousNativeRun.processStartedAt
|
|
: null),
|
|
nativeIssueId: lockedRun.nativeIssueId ?? issueRef.id,
|
|
driverKind:
|
|
lockedRun.driverKind ??
|
|
nativeExecution?.session.driverKind ??
|
|
"codex_app_server",
|
|
driverVersion: lockedRun.driverVersion ?? "phase6-v1",
|
|
completionContractId:
|
|
lockedRun.completionContractId ?? completionContract.row.id,
|
|
completionContractSha256:
|
|
lockedRun.completionContractSha256 ??
|
|
completionContract.row.canonicalSha256,
|
|
nativePhase: lockedRun.nativePhase ?? "observed",
|
|
nativePhaseUpdatedAt:
|
|
lockedRun.nativePhaseUpdatedAt ?? new Date(),
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, run.id)));
|
|
await measureRunPhase("runtime_selection.finalization_write", () => tx
|
|
.insert(nativeRunFinalizations)
|
|
.values({
|
|
runId: run.id,
|
|
companyId: agent.companyId,
|
|
issueId: issueRef.id,
|
|
phase: "observed",
|
|
})
|
|
.onConflictDoNothing());
|
|
return true;
|
|
});
|
|
if (!nativeSelected) return { selected: false };
|
|
stopControllerLease();
|
|
nativeWorkspaceSync = await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({
|
|
db,
|
|
runId: run.id,
|
|
companyId: agent.companyId,
|
|
workspaceId: nativeExecutionWorkspaceId,
|
|
workspaceLocalDir: executionWorkspace.cwd,
|
|
target: executionTarget,
|
|
lease: activeEnvironmentLease.lease,
|
|
restartRecovery: runOptions.nativeRestartRecovery,
|
|
sameRunRecovery: Boolean(runOptions.nativeLeaseOwner),
|
|
resourceDisposition: providerResourceDispositionForRun,
|
|
onCheckpoint: input.workspace.onCheckpoint,
|
|
onPhase: input.workspace.onPhase,
|
|
onEvent: onAdapterEvent,
|
|
}));
|
|
} else {
|
|
const legacyWarmLifecycle =
|
|
executionTarget?.kind === "remote" &&
|
|
executionTarget.transport === "sandbox" &&
|
|
executionTarget.runnerLifecyclePolicy?.mode === "warm"
|
|
? resolveReusableSandboxLifecycle({
|
|
lifecyclePolicy: executionTarget.runnerLifecyclePolicy,
|
|
target: executionTarget,
|
|
})
|
|
: null;
|
|
if (legacyWarmLifecycle?.sandboxResource === "keep_running") {
|
|
providerResourceDispositionForRun = "keep_running";
|
|
onProviderResourceDisposition(providerResourceDispositionForRun);
|
|
}
|
|
await measureRunPhase("runtime_selection.run_write", () => db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runtimeMode: "legacy",
|
|
runtimeModeResolverVersion:
|
|
nativeRuntimeResolution.resolverVersion,
|
|
runtimeModeReason: nativeRuntimeResolution.reason,
|
|
runtimeModeResolvedAt: run.runtimeModeResolvedAt ?? new Date(),
|
|
// Preserve server-owned admission and dispatch evidence on this
|
|
// row; never copy another run's execution profile.
|
|
runnerProfileJson: sql`(case when ${heartbeatRuns.runnerProfileJson} ? ${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}
|
|
then jsonb_build_object(${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}::text, ${heartbeatRuns.runnerProfileJson}->${CHAT_CONTROL_RECOVERY_ADMISSION_KEY})
|
|
else '{}'::jsonb end)
|
|
|| (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch'
|
|
then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch')
|
|
else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, run.id)));
|
|
}
|
|
return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync };
|
|
}
|