Files
PaperClipAI/server/src/services/native-runtime/runtime-mode.ts
T
Dotta fdf8c8464d feat(runner): add managed provider backends (#12699)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner provides durable, provider-neutral agent
execution.
> - The current stack supports qualified local providers but omits the
managed provider paths from the integration branch.
> - Claude Managed Agents and AWS AgentCore need explicit profile
qualification, durable recovery, usage accounting, and cleanup controls.
> - This pull request adds those managed backends as the third part of
the Runner parity stack.
> - The benefit is managed execution without weakening the default-off
Runner rollout gate.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting: Runner, server orchestration, database profiles, CLI, and
adapter configuration UI.

**Problem or motivation**

The current Runner stack cannot select or execute the managed Claude
Agents API or AWS Bedrock AgentCore Harness backends. It also lacks
qualified profile storage and recovery checks for those remote
resources.

**Proposed solution**

Add qualified managed and remote profiles, API and CLI management, exact
provider selection, durable lifecycle handling, cumulative usage
accounting, bounded cleanup, and retention acknowledgement. Keep
`enableNativeRunner` default-off.

**Alternatives considered**

A direct copy of the old integration branch was rejected because its
provider contracts, model values, credential flow, and migration history
no longer match the current base. A single large parity pull request was
also rejected because stacked review keeps each subsystem bounded.

**Roadmap alignment**

This continues the existing Runner architecture and rollout work. It
does not introduce a separate execution system.

**Additional context**

This pull request is based on the merged #12691 and #12685 stack. It
also closes the delayed security-review findings reported on #12691 by
binding qualified ACPX and OpenCode launch artifacts to the bytes
actually executed. A GitHub search for managed agent, AgentCore, and
Claude managed work found no duplicate public issue or pull request.

## What Changed

- Add Claude Managed Agents and AWS AgentCore provider executors to
runnerd.
- Add qualified managed and remote profile storage, routes, OpenAPI
contracts, CLI commands, and migration 0237.
- Validate profile ownership, enabled state, exact qualified revision,
model, agent version, and secret binding before persistence and
recovery.
- Persist durable provider session and owned skill state for
restart-safe cleanup.
- Reconcile uncertain create responses and delete remote sessions before
owned skills.
- Track cumulative provider usage and enforce positive session spend
caps.
- Recover interrupted AgentCore usage at the next turn boundary by
charging the prior invocation ceiling exactly once; keep the session
gated until an explicit monotonic budget raise.
- Isolate AgentCore AWS configuration from host profiles and
credential-process/SSO configuration while preserving workload identity.
- Require OpenCode 1.18.17 and fixed build-owned provider-pack artifact
paths; remove the ambient executable override.
- Snapshot and content-verify ACPX and OpenCode commands, scripts, and
provider executables before launch. Linux executes sealed inherited
descriptors; macOS uses authenticated private snapshots with retry-safe
rematerialization at the spawn boundary.
- Persist canonical ACPX and OpenCode launch-profile digests, reject
drift across fresh recovery, and make recovery failures sticky.
- Close and journal unsafe ACPX active-turn recovery before any provider
bootstrap or reconnect.
- Add managed provider fields to the Runner configuration UI and
permission projection.
- Preserve the default-off `enableNativeRunner` experimental flag.

## Verification

- `pnpm -r typecheck`
- `pnpm build`
- Focused managed server, database, CLI, Runner TypeScript, Rust,
Claude, AgentCore, ACPX, OpenCode, process-supervisor, and
durable-recovery tests passed.
- `cargo test -p paperclip-runner-core --lib --locked` (160 tests)
- `cargo check --workspace --all-targets --locked`
- Native Codex integration tests passed (60 tests); native provider
tests passed (7 tests); server native-runtime tests passed (87 tests).
- Verified-launch replacement, nested-spawn retry, exact-version,
profile-drift, sticky-failure, and no-bootstrap active-recovery tests
passed.
- `git diff --check`
- The PR changes 91 files. `pnpm-lock.yaml` is unchanged. The Rust
workspace lockfile adds the approved `rustix` dependency used for safe
descriptor handling while `#![forbid(unsafe_code)]` remains enabled.

## Risks

- The provider APIs can change while they are in beta. Exact
qualification and fail-closed recovery checks limit drift.
- Remote cleanup can fail after a partial create. Durable ownership
inventories and retry-safe deletion preserve recovery state.
- Migration 0237 adds profile tables. The generated migration and
snapshot pass the repository migration checks.
- Managed execution can incur provider cost. Positive default spend caps
and explicit retention acknowledgement limit accidental use.
- An interrupted AgentCore invocation without final metadata is
conservatively charged to its active session ceiling. This can overstate
cost, but cannot undercount it; later work requires an explicit budget
increase.
- Linux qualified launches use sealed memory descriptors. macOS lacks
executable-descriptor APIs, so the runner uses owner-only private
snapshots and minimizes linked-path lifetime; hostile same-UID processes
remain outside the documented local-host trust boundary.
- The global Runner feature remains default-off.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex, GPT-5, with tool use, code execution, and subagent review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-02 00:48:30 -05:00

536 lines
17 KiB
TypeScript

import {
NATIVE_STATUS_ARBITER_POLICY_VERSION,
type NativeAuthoritativeIssueStatus,
type NativeStatusDecision,
} from "./status-arbiter.js";
import {
PaperclipRunnerProviderProfileError,
resolvePaperclipRunnerProviderProfile,
type PaperclipRunnerProviderProfile,
} from "./provider-profile.js";
/**
* Public compatibility resolver version. This value is persisted by the
* original heartbeat selection seam and must remain stable for existing runs
* and downstream importers.
*/
export const NATIVE_RUNTIME_RESOLVER_VERSION = "paperclip-runner-v1" as const;
/** Resolver version for the richer native runtime profile used by runnerd. */
export const NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION = "phase6-v1" as const;
export type HeartbeatRuntimeResolution =
| {
kind: "legacy";
resolverVersion: typeof NATIVE_RUNTIME_RESOLVER_VERSION;
reason: "direct_adapter" | "persisted_legacy_selection";
}
| {
kind: "native";
resolverVersion: typeof NATIVE_RUNTIME_RESOLVER_VERSION;
reason: "explicit_paperclip_runner" | "persisted_native_selection";
provider: PaperclipRunnerProviderProfile["provider"];
};
export type NativeRuntimeResolution =
| {
kind: "legacy";
resolverVersion: typeof NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION;
reason: string;
authorityDecision?: NativeStatusDecision;
}
| {
kind: "native";
resolverVersion: typeof NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION;
reason: "eligible_opt_in";
profile: {
mode: "native";
backend:
| "codex_app_server"
| "opencode_server"
| "claude_managed_agents_api"
| "aws_agentcore_harness_api"
| "acpx_runtime";
protocolVersion: 1;
};
authorityDecision: NativeStatusDecision;
};
/**
* Runner ingress follows the resolved runtime decision, not a second instance
* flag. Persisted native runs therefore keep their transport during recovery
* after the rollout flag is disabled, while legacy runs never gain ingress.
*/
export function isRunnerIngressAuthorized(
resolution: NativeRuntimeResolution,
): boolean {
return resolution.kind === "native";
}
export class NativeRunnerSelectionError extends Error {
constructor(readonly code: string, message: string) {
super(message);
this.name = "NativeRunnerSelectionError";
}
}
export class NativeRuntimeEligibilityError extends NativeRunnerSelectionError {
constructor(
code: string,
reason?: string,
) {
super(code, reason ?? `Native runner profile is ineligible: ${code}`);
this.name = "NativeRuntimeEligibilityError";
}
}
function ineligible(
code: string,
reason: string,
): NativeRuntimeEligibilityError {
return new NativeRuntimeEligibilityError(
code,
reason,
);
}
export function resolveNativeRuntimeMode(input: {
enabled: boolean;
runtimeConfig: unknown;
adapterConfig?: unknown;
agent: { id?: string; status: string; adapterType: string | null };
issue: { id: string; workMode: string; executionWorkspaceId?: string | null } | null;
target: { kind?: string } | null | undefined;
workspaceId: string | null;
}): NativeRuntimeResolution {
const runnerAdapterSelected = input.agent.adapterType === "paperclip_runner";
// Fresh direct-adapter runs never enter the native control plane, even if an
// obsolete runtimeConfig.nativeRunner value is still present. Persisted
// native runs are handled by resolveHeartbeatNativeRuntimeMode above this
// fresh-selection seam so they remain recoverable after rollout changes.
if (!runnerAdapterSelected) {
return {
kind: "legacy",
resolverVersion: NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION,
reason: "direct_adapter",
};
}
if (!input.enabled) {
throw ineligible(
"paperclip_runner_rollout_disabled",
"Paperclip Runner is experimental and disabled on this instance.",
);
}
let runnerProfile: PaperclipRunnerProviderProfile;
try {
runnerProfile = resolvePaperclipRunnerProviderProfile(input.adapterConfig);
} catch (error) {
if (error instanceof PaperclipRunnerProviderProfileError) {
throw ineligible(error.code, error.message);
}
throw error;
}
if (
input.agent.adapterType !== "paperclip_runner"
|| input.agent.status !== "active" && input.agent.status !== "running"
) {
throw ineligible(
"paperclip_runner_agent_ineligible",
"Paperclip Runner requires an active agent.",
);
}
const allowedWorkModes = ["standard", "planning", "ask"];
if (!input.issue || !allowedWorkModes.includes(input.issue.workMode)) {
throw ineligible(
"paperclip_runner_issue_ineligible",
"Paperclip Runner requires a standard, planning, or ask task.",
);
}
const rollout = resolveNativeMigrationStatus({
facts: { applicationEnabled: true },
priorIssueStatus: "in_progress",
agentId: input.agent.id ?? "00000000-0000-4000-8000-000000000000",
});
if (!rollout.effects.some((effect) => effect.kind === "record_mode_native")) {
throw ineligible(
"paperclip_runner_rollout_policy_rejected",
"Native rollout policy did not select native mode.",
);
}
return {
kind: "native",
resolverVersion: NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION,
reason: "eligible_opt_in",
profile: {
mode: "native",
backend: runnerProfile.backend,
protocolVersion: 1,
},
authorityDecision: rollout,
};
}
/**
* Backward-compatible heartbeat selection API. New runnerd code consumes the
* richer profile from resolveHeartbeatNativeRuntimeMode; this public seam
* keeps the original result shape, reason codes, and resolver version.
*/
export function resolveHeartbeatRuntimeMode(input: {
persisted: {
runtimeMode: string | null;
runtimeModeResolvedAt: Date | null;
};
enabled: boolean;
adapterType: string | null;
adapterConfig: unknown;
agentStatus: string;
issue: { workMode: string } | null;
executionTarget: { kind?: string } | null | undefined;
}): HeartbeatRuntimeResolution {
if (input.persisted.runtimeModeResolvedAt) {
if (input.persisted.runtimeMode === "native") {
return {
kind: "native",
resolverVersion: NATIVE_RUNTIME_RESOLVER_VERSION,
reason: "persisted_native_selection",
provider: "codex",
};
}
return {
kind: "legacy",
resolverVersion: NATIVE_RUNTIME_RESOLVER_VERSION,
reason: "persisted_legacy_selection",
};
}
let resolution: NativeRuntimeResolution;
try {
resolution = resolveNativeRuntimeMode({
enabled: input.enabled,
runtimeConfig: {},
adapterConfig: input.adapterConfig,
agent: {
status: input.agentStatus,
adapterType: input.adapterType,
},
issue: input.issue
? { id: "heartbeat-runtime-selection", workMode: input.issue.workMode }
: null,
target: input.executionTarget,
workspaceId: null,
});
} catch (error) {
if (error instanceof NativeRuntimeEligibilityError) {
throw new NativeRunnerSelectionError(error.code, error.message);
}
throw error;
}
if (resolution.kind === "legacy") {
return {
kind: "legacy",
resolverVersion: NATIVE_RUNTIME_RESOLVER_VERSION,
reason: "direct_adapter",
};
}
return {
kind: "native",
resolverVersion: NATIVE_RUNTIME_RESOLVER_VERSION,
reason: "explicit_paperclip_runner",
provider: resolution.profile.backend === "opencode_server"
? "opencode"
: resolution.profile.backend === "claude_managed_agents_api"
? "claude_managed"
: resolution.profile.backend === "aws_agentcore_harness_api"
? "aws_agentcore"
: resolution.profile.backend === "acpx_runtime"
? "acpx"
: "codex",
};
}
/**
* Production heartbeat selection seam. A resolved run keeps its persisted
* mode across configuration changes; only a fresh unresolved run consults the
* current global flag and agent profile.
*/
export function resolveHeartbeatNativeRuntimeMode(input: {
persisted: {
runtimeMode: string | null;
runtimeModeReason: string | null;
runtimeModeResolvedAt: Date | null;
driverKind?: string | null;
};
enabled: boolean;
runtimeConfig: unknown;
adapterConfig?: unknown;
agent: { id?: string; status: string; adapterType: string | null };
issue: { id: string; workMode: string; executionWorkspaceId?: string | null } | null;
target: { kind?: string } | null | undefined;
workspaceId: string | null;
}): NativeRuntimeResolution {
if (input.persisted.runtimeModeResolvedAt) {
if (input.persisted.runtimeMode === "native") {
if (input.agent.adapterType !== "paperclip_runner") {
throw ineligible(
"paperclip_runner_adapter_binding_mismatch",
"A persisted native run must remain bound to the Paperclip Runner adapter.",
);
}
if (
input.agent.status !== "active" &&
input.agent.status !== "running"
) {
throw ineligible(
"paperclip_runner_agent_ineligible",
"A persisted Paperclip Runner run cannot recover through a non-invokable agent.",
);
}
const driverKind = input.persisted.driverKind;
const backend = driverKind === "opencode_server"
? "opencode_server"
: driverKind === "claude_managed_agents_api"
? "claude_managed_agents_api"
: driverKind === "aws_agentcore_harness_api"
? "aws_agentcore_harness_api"
: driverKind === "acpx_runtime"
? "acpx_runtime"
: driverKind === null
|| driverKind === undefined
|| driverKind === "codex"
|| driverKind === "codex_app_server"
? "codex_app_server"
: null;
if (backend === null) {
throw ineligible(
"paperclip_runner_driver_unsupported",
`Persisted Paperclip Runner driver is unsupported: ${driverKind}`,
);
}
return {
kind: "native",
resolverVersion: NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION,
reason: "eligible_opt_in",
profile: {
mode: "native",
backend,
protocolVersion: 1,
},
authorityDecision: resolveNativeMigrationStatus({
facts: input.enabled
? { applicationEnabled: true }
: { killSwitchActiveForNewRuns: true },
priorIssueStatus: "in_progress",
agentId: input.agent.id ?? "00000000-0000-4000-8000-000000000000",
}),
};
}
return {
kind: "legacy",
resolverVersion: NATIVE_RUNTIME_PROFILE_RESOLVER_VERSION,
reason: input.persisted.runtimeModeReason ?? "persisted_legacy_selection",
};
}
return resolveNativeRuntimeMode(input);
}
/** Production read-model facts used by compatibility and mixed-ledger views. */
export function inspectNativeCompatibilityState(input: {
resolution: NativeRuntimeResolution;
nativeRecordCount: number;
decisionCount: number;
issueStatus: string;
statusVersion: number;
persistedEffectKinds: string[];
}) {
const effects = input.persistedEffectKinds.length > 0
? [...input.persistedEffectKinds]
: input.resolution.kind === "legacy"
? ["legacy_existing_behavior"]
: input.nativeRecordCount === 0 && input.statusVersion === 0
? ["initialize_status_version_zero"]
: [];
return {
mode: input.resolution.kind,
native: input.nativeRecordCount > 0,
hasNativeDecisionLineage: input.decisionCount > 0,
issueStatus: input.issueStatus,
statusVersion: input.statusVersion,
statusAction: input.resolution.kind === "legacy" ? "legacy_finalizer" : "preserve",
reasonCode: null,
effects,
} as const;
}
/** Expand-only migration evidence; it never mutates or synthesizes history. */
export function inspectNativeMigrationState(input: {
resolution: NativeRuntimeResolution;
nativeRecordCount: number;
decisionCount: number;
issueStatusBefore: string;
issueStatusAfter: string;
statusVersion: number;
hasPendingReview: boolean;
}) {
const effects = input.resolution.kind === "legacy"
? input.issueStatusBefore === "done"
? ["retain_legacy_mode", "retain_audit_lineage"]
: ["return_native_false"]
: input.nativeRecordCount === 0 && input.hasPendingReview && input.statusVersion > 0
? ["increment_status_version_once", "bind_reviewer"]
: input.nativeRecordCount === 0
? ["expand_schema", "status_version_default_zero"]
: [];
return {
mode: input.resolution.kind,
native: input.nativeRecordCount > 0,
hasSyntheticHistory: input.nativeRecordCount === 0 && input.decisionCount > 0,
statusPreserved: input.issueStatusBefore === input.issueStatusAfter,
statusVersion: input.statusVersion,
statusAction: input.resolution.kind === "legacy" ? "legacy_finalizer"
: input.hasPendingReview ? input.issueStatusAfter : "preserve",
reasonCode: null,
effects,
} as const;
}
export type NativeCompatibilityFacts = {
invalidNativeFinalization?: boolean;
terminalResumeAuthorized?: boolean;
shadowApplicationDisabled?: boolean;
mixedLedger?: boolean;
statusWriterAdvancedVersion?: boolean;
};
export function resolveNativeCompatibilityStatus(input: {
facts: NativeCompatibilityFacts;
priorIssueStatus: NativeAuthoritativeIssueStatus;
agentId: string;
}): NativeStatusDecision {
const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "preserve",
toStatus: input.priorIssueStatus,
reasonCode,
unblockDescriptor: null,
effects,
});
if (input.facts.invalidNativeFinalization) {
return preserve("native_finalization_invalid", [{
kind: "record_finalization_error",
cause: "native_finalization_invalid",
nextAction: "Repair the persisted native result.",
agentId: input.agentId,
}]);
}
if (input.facts.terminalResumeAuthorized) {
return {
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "in_progress",
toStatus: "in_progress",
reasonCode: "authorized_resume",
unblockDescriptor: null,
effects: [{
kind: "enqueue_continuation",
continuationKind: "same_agent",
summary: "Resume the terminal issue through the authorized compatibility path.",
idempotencyKey: "native-compatibility:authorized-resume",
agentId: input.agentId,
}],
};
}
if (input.facts.shadowApplicationDisabled) {
return preserve("completion_contract_satisfied", [{ kind: "record_shadow_decision" }]);
}
if (input.facts.mixedLedger) {
return preserve("completion_contract_satisfied", [{ kind: "render_four_layers" }]);
}
if (input.facts.statusWriterAdvancedVersion) {
return preserve("arbitration_conflict_reloaded", [
{ kind: "increment_status_version" },
{ kind: "schedule_reconciliation" },
]);
}
throw new Error("native_compatibility_facts_invalid");
}
export type NativeMigrationFacts = {
shadowMaterialization?: boolean;
classifiedDivergence?: boolean;
applicationEnabled?: boolean;
policyPinned?: boolean;
killSwitchActiveForNewRuns?: boolean;
};
export function resolveNativeMigrationStatus(input: {
facts: NativeMigrationFacts;
priorIssueStatus: NativeAuthoritativeIssueStatus;
agentId: string;
}): NativeStatusDecision {
const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "preserve",
toStatus: input.priorIssueStatus,
reasonCode,
unblockDescriptor: null,
effects,
});
if (input.facts.shadowMaterialization) {
return preserve("completion_contract_satisfied", [
{ kind: "materialize_contract" },
{ kind: "record_shadow_decision" },
]);
}
if (input.facts.classifiedDivergence) {
return preserve("completion_evidence_incomplete", [{ kind: "record_mode_labeled_divergence" }]);
}
if (input.facts.killSwitchActiveForNewRuns) {
return {
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "in_progress",
toStatus: "in_progress",
reasonCode: "live_continuation_registered",
unblockDescriptor: null,
effects: [
{
kind: "enqueue_continuation",
continuationKind: "same_agent",
summary: "Finish the already-active run in native mode.",
idempotencyKey: "native-migration:kill-switch-active-run",
agentId: input.agentId,
},
{ kind: "finish_as_native" },
],
};
}
if (input.facts.policyPinned) {
return {
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "done",
toStatus: "done",
reasonCode: "completion_contract_satisfied",
unblockDescriptor: null,
effects: [{ kind: "record_mode_native" }, { kind: "record_policy_version" }],
};
}
if (input.facts.applicationEnabled) {
return {
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
statusAction: "in_progress",
toStatus: "in_progress",
reasonCode: "live_continuation_registered",
unblockDescriptor: null,
effects: [
{
kind: "enqueue_continuation",
continuationKind: "same_agent",
summary: "Continue the allowlisted native run.",
idempotencyKey: "native-migration:application-enabled",
agentId: input.agentId,
},
{ kind: "record_mode_native" },
],
};
}
throw new Error("native_migration_facts_invalid");
}