Files
PaperClipAI/server/src/services/heartbeat/execution-configuration.ts
T
Nicky LeachandClaude Opus 5.5 6d8ebc4a85 fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can bind a company secret to an environment variable for an
agent, an environment, a project, or a routine
> - Core resolves these values for each run and gives them to the agent
process or to the Runner
> - Core stores the run output: the run log, the run excerpts, adapter
events, and native (Runner protocol) transcript events
> - The run-output redactor did not know the resolved secret values.
Only failure diagnostics and the `meta.env` keys of `adapter.invoke`
were masked
> - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the
secret in plaintext, and every person who can view the run could read it
> - This pull request gives the resolved values of secret-bound keys to
the run-output redactor and to the native control-plane port
> - The benefit is that an accidental print of a secret does not put the
secret into stored run output, for local and native runs

## Linked Issues or Issue Description

**What happened?**
An agent printed a secret-bound environment variable during a run (for
example with `env` or `echo $STRIPE_KEY`). Core stored the value in
plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in
the transcript events. Anyone who can view the run can read the value.

**Expected behavior**
Core masks the value of each secret-bound environment variable before it
stores run output. Core does not mask plain (non-secret) configured
values.

**Steps to reproduce**
1. Create a company secret and bind it to an agent environment variable
with `secret_ref`.
2. Configure the agent so that its command prints the variable.
3. Run the agent and open the run log. The value shows in plaintext.

**Related**
Refs #21330. That PR keeps configured secrets out of OpenCode's own
storage and model requests inside the Runner. This PR is the Core side:
it masks secrets in the run output that Core stores, for every run type.
The two changes are independent.

## What Changed

- `secretEnvironmentValues(env, secretKeys)` in
`agent-identity-redaction.ts` selects the raw values of keys in
`secretKeys` (the keys that came from `secret_ref` or `user_secret_ref`
bindings). `secretOutputForms` derives the forms to mask: the full
value, the value without outer whitespace, the JSON-escaped form, and
each line of a multi-line value (PEM armor lines are not added). It
skips a form that has fewer than 8 characters after trimming. This is
the same 8-character limit that diagnostic redaction uses for unknown
values.
- The output redactor keeps these secrets in a separate set, beside the
existing agent private-key values. The private-key values are masked
everywhere, as before. Environment secrets are masked only in output
text. They are not masked in protocol fields: `schema`, `type`, `kind`,
`status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids,
refs, codes, revisions, versions, digests, timestamps, and names that
end in these words (for example `eventType`, `turnTerminalState`,
`reportedWorkDisposition`). So a secret that equals a protocol word (for
example `completed`) cannot change an event type or a terminal state.
- The streaming match now scans left to right. At each position it tries
the longest value first. If a longer value could still start at that
position, it holds the text until the next chunk, also when a shorter
value already matches there. So the tail of a longer secret cannot leak
when a shorter secret is its prefix.
- `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to
the run's output redactor immediately after it resolves them. This one
redactor already masks the local adapter log stream, the excerpts, live
log events, adapter events, `adapter.invoke` metadata, the adapter
result, and the canonical native log export.
- For a native run, `executeHeartbeatRuntime` stores the delivered
secrets (only values that have a form to mask) encrypted under a new
context key, `paperclipOutputSecretRedactions`, in one transaction. It
then adds all stored values of the run to the redactor. A run that
reconnects to a live Runner after a `"latest"` secret rotation then
masks the value that the Runner holds and the new value. This store
reuses the encryption and fingerprint logic of the run redaction
registry, but it is separate: API read redaction does not apply its
values, so a secret cannot change ids, event names or terminal states in
API responses. API reads hide the new key like the existing registry
key.
- `executeHeartbeatRuntime` gives the native executor the run's secrets
and the observability credentials (`observabilitySecretValues`).
- The native executor gives the secrets to its failure-diagnostic
redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks
them in each PRP event before it persists the event, with the existing
delta-aware handling for values that are split across `item.delta`
events. This covers native runs on the local Runner and on plugin-hosted
agent computers.
- After a controller restart, a new control-plane port cannot know the
text that the old port held back, because that text is never stored. On
the first delta of each stream it has not seen, the port masks leading
text that matches the tail of a secret (4 or more characters). It skips
text that can start a secret, and it keeps checking the stream while a
delta falls entirely inside such a tail.
- The native executor classifies failures from the original error. It
masks only the agent private key in place, as before. It masks
environment secrets in a separate redactor that it uses only for text
that it writes or logs (`failureDetail`, failure messages, stderr
diagnostics). Core masks the rethrown error before it stores it.
- `coalesceIntoActiveExecutionRun` (wake queue) writes the merged
context with `withRunRedactionHistory`, which takes
`paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from
the locked current row. Before, a coalesced wake built from an older
context could delete either key. This also protects the existing
registry. This is a change outside the secrets module, in the one place
that replaces a running run's context concurrently.
- When an adapter or native run fails, the heartbeat keeps the original
error for retry decisions and rethrows it. The copies that it saves in
`workspace_finalize` metadata (`errorMessage`) and in a failed session
goal action go through the run's output redactor first.
- `docs/deploy/secrets.md` describes the mask and its limits.

## Verification

- `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new,
embedded Postgres): a real `process` adapter run with a `secret_ref`
bound to `"latest"`. The agent prints the secret split across two
writes, in JSON, and on stderr. The stored log, run row, and run events
do not contain the value. The plain value and a 5-character secret are
not masked. The test then rotates the secret and runs again. The second
run masks the rotated value, and a SHA-256 digest in the output proves
the run received the rotated value. Red-green: it fails without the new
line in `execution-configuration.ts`.
-
`server/src/services/native-runtime/paperclip-control-plane-port.test.ts`:
the port masks a secret in a persisted tool event and in a value split
across two `item.delta` events; plain and short values stay (red-green:
fails without the port change). A new test binds secrets equal to
`interrupted`, `completed`, `succeeded` and `satisfied`, then appends a
`turn.interrupted` event, checkpoints, and completes the run. The event
type, payload status, terminal states and criterion status stay
unchanged and the result is saved (red-green: fails when protocol fields
are masked).
- `server/src/services/heartbeat/runtime-execution.test.ts`: a value
stored by an earlier attempt, the current value, and an observability
header value all reach native dispatch and the heartbeat redactor. Short
values are not stored. The shared run redaction registry stays empty,
and an API read hides the new key (red-green: fails without the store
read).
- `server/src/services/native-runtime/native-session-executor.test.ts`:
a native failure that contains the secret is masked in failure writes,
logs and the rethrown error, and the executor gives the secrets to the
control-plane port.
- `server/src/services/agent-identity-redaction.test.ts` (new): value
selection, the minimum length, multi-line and JSON-escaped forms, nested
values, every split point of a streamed value, overlapping secrets
(`abcdefgh` and `abcdefgh1234567890`) split at the shorter value,
protocol fields in events, terminal states and deltas, and secrets added
by a later attempt.
- Restart, retry and coalescing (each fails without its fix): a port
test restarts between two deltas of a secret and stores `echo ` and
`***REDACTED*** done`; an executor test binds `timed out` and keeps
`provider_session_open_timeout`; a wake-queue adapter test keeps both
redaction keys when a coalesced wake replaces the context.
- `server/src/services/heartbeat/run-output.test.ts`: the local output
path masks the log, excerpt, live event, adapter event, and adapter
metadata.
- `pnpm --filter @paperclipai/server typecheck` passes. The changed test
files and the other users of the redactor (`agent-identity`, `run-log`,
`heartbeat-case-handoff`, `native-run-log-export.integration`) pass in
full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test
files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify`
exits 2 (broader verification required) because `docs/deploy/secrets.md`
is a global CI input, so required CI supplies broad coverage.

## Risks

- This is a guard against accidental leaks. It is not a security
boundary. An agent can change a value before it prints it (for example,
encode it in base64 or print one character at a time), and Core cannot
detect that.
- Core does not mask a secret value or line that has fewer than 8
characters. This prevents a short value from erasing ordinary output. A
secret that short is not masked in run output.
- Core does not mask environment secrets in protocol fields (see What
Changed). If a secret appears in such a field, it stays. These fields
hold protocol words and generated identifiers, not agent output.
- If a value is common text of 8 or more characters, Core masks that
text wherever it occurs in that run's output text. This only affects
runs that bind that secret.
- If a key is secret-bound in one source and a later source sets the
same key to a plain value, `secretKeys` still includes the key, so Core
masks the plain value. This is the existing meaning of `secretKeys`.
- For native runs, Core now stores the delivered secrets encrypted in
`contextSnapshot` under `paperclipOutputSecretRedactions`, with the same
local encryption as the existing run redaction registry. It adds one
transaction per native run that has secrets. API reads do not apply
these values and hide the key.
- After a controller restart in the middle of a secret, a tail shorter
than 4 characters can still be stored. A new stream that starts with the
tail of a secret (4 or more characters) is masked at its start even
without a restart. This is a small amount of extra masking.
- Core masks values at write time. Run output that Core stored before
this change is not changed.
- No schema change and no migration. The new fields are optional and
in-process only.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:41 -08:00

486 lines
33 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { supportsPaperclipRunnerPreconfiguredHome } from "../native-runtime/provider-profile.js";
import { agentHarnessType } from "@paperclipai/shared";
import { observabilitySecretValues } from "@paperclipai/adapter-utils/agent-observability";
import { agentObservabilityService, unavailableAgentObservability, type ResolvedAgentObservability } from "../agent-observability.js";
import { agentObservabilityLaunch } from "../agent-observability-launch.js";
import { withSkillActor } from "../skill-access.js";
import type { AuthorizationActor } from "../authorization.js";
import { eq, sql } from "drizzle-orm";
import { heartbeatRuns, type Db, type agents, type projects } from "@paperclipai/db";
import type { AiConnectionBinding } from "@paperclipai/shared";
import { readPaperclipSkillSyncPreference, resolvePaperclipDesiredSkillNames, resolveLegacyPaperclipDesiredSkillNames,
writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils";
import { parseObject } from "../../adapters/utils.js";
import { logger } from "../../middleware/logger.js";
import { AI_CONNECTION_BUSY_RETRY_REASON, extractWakeCommentIds, isNonAssigneeWorkspaceBusyRetry } from "../../modules/run-dispatch/index.js";
import { createAgentIdentityRedactor, secretEnvironmentValues } from "../agent-identity-redaction.js";
import { agentIdentityService, supportsManagedAgentIdentity } from "../agent-identity.js";
import { readAiConnectionConfigurationFailure, readAiCredentialAccessFailure } from "../ai-connection-configuration-failure.js";
import { AI_AUTH_ENV_KEYS, isAiConnectionBusy, prepareManagedAiRuntime, stripAiAuthBindings } from "../ai-connection-runtime.js";
import { managedAiSessionIdentityCompatible } from "../ai-connection-session.js";
import type { companySkillService } from "../company-skills.js";
import { connectionIntentService } from "../connection-intents.js";
import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED, pluginEnvironmentRunnerTaskDriver, pluginRunnerAiConnectionRequiredMessage } from "../plugin-runner-ai-connection.js";
import { prepareConnectionInstructionDelivery } from "../connection-instructions.js";
import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "../connector-runtime.js";
import { resolveManagedGitHubIdentitySelection } from "../git-credentials.js";
import { resolveAssignedConnectionInstructionsForRun, resolveMcpConnectionSkillsForRun } from "../native-runtime/assigned-mcp-tools.js";
import { applyMcpConnectionSkills } from "../mcp-skill-runtime.js";
import { recordFailedSkillPreparation, type NativeRunHistoricalSpan } from "../native-runtime/native-run-trace.js";
import { collectRunFailureSecretValues } from "../run-failure-diagnostics.js";
import { skillVersionSelectionMap } from "../runtime-skill-selections.js";
import { environmentSkillConfig } from "../environment-skills.js";
import type { selectHeartbeatEnvironment } from "./environment-selection.js";
import type { createHeartbeatRetries } from "./retries.js";
import type { createHeartbeatLifecycle } from "./run-lifecycle.js";
import { createHeartbeatPreparationTiming } from "./preparation-timing.js";
import { mcpAppViewsService, renderMcpAppRunContext } from "../mcp-app-views.js";
import {
ConfigurationIncompleteFailure, applyRunScopedMentionedSkillKeys, resolveExecutionRunAdapterConfig,
resolveRunScopedMentionedSkillKeys, type createHeartbeatRunPreparation, type resolveAcceptedPlanWakeRoutingDecision,
} from "./run-preparation.js";
import { buildExecutionWorkspaceConfigSnapshot, stripWorkspaceRuntimeFromExecutionRunConfig } from "./workspaces.js";
import type { HeartbeatWorkspacePreparationInput } from "./workspace-preparation.js";
const ACCEPTED_PLAN_CONVERSION_SKILL_KEY =
"paperclipai/paperclip/paperclip-converting-plans-to-tasks";
type ManagedAiRuntime = Awaited<ReturnType<typeof prepareManagedAiRuntime>>;
type EnvironmentSelection = Awaited<ReturnType<typeof selectHeartbeatEnvironment>>;
/** Preparation and routing must use the same immutable mode as native recovery. */
export function bindHeartbeatRunnerHomeMode(config: Record<string, unknown>, persistedExecution: unknown): Record<string, unknown> {
const saved = parseObject(persistedExecution);
if (Object.keys(saved).length === 0) return config;
// Execution schemas before v7 always used isolated homes.
const homeMode = saved.homeMode ?? "isolated";
// Omission already means isolated. Recovery must not add a fingerprinted
// configuration property merely to restate that unchanged default.
if (homeMode === "isolated" && config.homeMode === undefined) return config;
return { ...config, homeMode };
}
export interface HeartbeatExecutionConfigurationInput {
run: typeof heartbeatRuns.$inferSelect;
agent: typeof agents.$inferSelect;
task: {
issueId: string | null;
context: Record<string, unknown>;
responsibleUserId: string | null;
acceptedPlanContinuationWake: boolean;
acceptedPlanWakeRoutingDecision: Awaited<ReturnType<typeof resolveAcceptedPlanWakeRoutingDecision>> | null;
taskProjectContext: Pick<typeof projects.$inferSelect, "id" | "env"> | null;
routineEnvContext: Awaited<ReturnType<ReturnType<typeof createHeartbeatRunPreparation>["getRoutineEnvForExecutionIssue"]>>;
/** Resolved from the durable Studio test row by task-context, never wake input. */
pinnedSkillTest?: { key: string; versionId: string } | null;
/** Restart recovery is reconnecting to this run's live remote Runner. */
adoptingLiveRunner?: boolean;
};
config: {
sourceWorkspaceConfig: Record<string, unknown>;
aiBinding: AiConnectionBinding | undefined;
};
policy: Pick<HeartbeatWorkspacePreparationInput["policy"],
"trustPreset" | "useIsolatedTaskDirectory" | "requestedExecutionWorkspaceMode"> & { explicitTaskDirectory: boolean };
environment: Pick<EnvironmentSelection,
"selectedEnvironmentId" | "selectedEnvironmentForConfig" | "resolvedInstanceSettings" | "persistedNativeExecutionInput">;
services: {
secretsSvc: Parameters<typeof resolveExecutionRunAdapterConfig>[0]["secretsSvc"];
companySkills: Pick<ReturnType<typeof companySkillService>, "withRuntimeSkillEntries">;
appendRunEvent: ReturnType<typeof createHeartbeatLifecycle>["appendRunEvent"];
finalizeAiConnectionBusyDeferral: ReturnType<typeof createHeartbeatRetries>["finalizeAiConnectionBusyDeferral"];
};
// The executor must observe these before a later preparation step can throw.
onFailureSecretsResolved: (readSecrets: () => string[]) => void;
onManagedAiRuntimeAcquired: (runtime: ManagedAiRuntime) => void;
onIdentityRedactorPrepared: (redactor: ReturnType<typeof createAgentIdentityRedactor>) => void;
}
export function buildHeartbeatWorkspaceConfiguration(
sourceWorkspaceConfig: Record<string, unknown>,
{ useIsolatedTaskDirectory, explicitTaskDirectory, requestedExecutionWorkspaceMode }: HeartbeatExecutionConfigurationInput["policy"],
selectedEnvironmentId: string | null,
) {
const forceDirectoryStrategy = useIsolatedTaskDirectory || (explicitTaskDirectory && (
requestedExecutionWorkspaceMode === "agent_default" || Object.keys(parseObject(sourceWorkspaceConfig.workspaceStrategy)).length > 0
));
const mergedConfig: Record<string, unknown> = {
...sourceWorkspaceConfig,
// The base below is already task-owned. Keep directory transport while
// preserving isolated mode and the mandatory sandbox preflight.
...(forceDirectoryStrategy ? { workspaceStrategy: { type: "project_primary" } } : {}),
};
return { mergedConfig, configSnapshot: buildExecutionWorkspaceConfigSnapshot(mergedConfig, selectedEnvironmentId) };
}
/** Assembles effective run configuration; the executor retains cleanup ownership. */
export async function prepareHeartbeatExecutionConfiguration(db: Db, input: HeartbeatExecutionConfigurationInput) {
const { run, agent, onFailureSecretsResolved, onManagedAiRuntimeAcquired, onIdentityRedactorPrepared } = input;
const time = createHeartbeatPreparationTiming(run.id);
const { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext, pinnedSkillTest } = input.task;
const { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput } = input.environment;
const useSshHostAuth = selectedEnvironmentForConfig?.driver === "ssh" && selectedEnvironmentForConfig.config.useHostAuth !== false && !persistedNativeExecutionInput;
let sshRunnerHomeMode: "isolated" | "preconfigured" | undefined;
if (agent.adapterType === "paperclip_runner" && selectedEnvironmentForConfig?.driver === "ssh" && !persistedNativeExecutionInput) {
sshRunnerHomeMode = useSshHostAuth && supportsPaperclipRunnerPreconfiguredHome(agent.adapterConfig, { companyId: agent.companyId }) ? "preconfigured" : "isolated";
}
const sourceWorkspaceConfig = agent.adapterType === "paperclip_runner"
? { ...input.config.sourceWorkspaceConfig,
homeMode: bindHeartbeatRunnerHomeMode({ ...parseObject(agent.adapterConfig), ...(sshRunnerHomeMode ? { homeMode: sshRunnerHomeMode } : {}) }, persistedNativeExecutionInput).homeMode }
: input.config.sourceWorkspaceConfig;
const preconfiguredHome = agent.adapterType === "paperclip_runner" && sourceWorkspaceConfig.homeMode === "preconfigured";
if (preconfiguredHome && pinnedSkillTest) {
throw new ConfigurationIncompleteFailure("Skill tests require an isolated Runner home to deliver the saved version.", {
configurationIncomplete: { reason: "skill_unavailable", actionUrl: `/agents/${agent.id}/runtime` },
});
}
// A plugin Runner-task environment runs only managed AI connections. Refuse a new
// admission before secrets, leases, or provider calls. Resumed native input keeps
// the contract it was admitted under.
if (agent.adapterType === "paperclip_runner" && (preconfiguredHome || !input.config.aiBinding) && !persistedNativeExecutionInput
&& selectedEnvironmentForConfig && await pluginEnvironmentRunnerTaskDriver(db, selectedEnvironmentForConfig)) {
throw new ConfigurationIncompleteFailure(pluginRunnerAiConnectionRequiredMessage(preconfiguredHome), {
configurationIncomplete: { reason: PLUGIN_RUNNER_AI_CONNECTION_REQUIRED, actionUrl: `/agents/${agent.id}/runtime`,
...(preconfiguredHome ? { homeMode: "preconfigured" } : {}) },
});
}
const initialHarness = agentHarnessType(agent.adapterType, sourceWorkspaceConfig);
const observabilityKind = (harness: string) => harness === "codex_local" ? "codex" as const
: harness === "claude_local" ? "claude" as const
: harness === "opencode_local" && agent.adapterType === "paperclip_runner" ? "opencode" as const : undefined;
const unsupportedObservability = !observabilityKind(initialHarness)
? "unsupported_harness" as const : preconfiguredHome && initialHarness === "codex_local" ? "unsupported_home_mode" as const : undefined;
const observabilitySnapshot: ResolvedAgentObservability = await agentObservabilityService(db).resolve(agent.companyId, { runId: run.id, agentId: agent.id, responsibleUserId, issueId, unavailableReason: unsupportedObservability })
.catch(() => unavailableAgentObservability(agent.companyId, "configuration_invalid"));
const identityRedactor = createAgentIdentityRedactor();
const earlyObservability = unsupportedObservability === undefined && observabilityKind(initialHarness)
? agentObservabilityLaunch(observabilitySnapshot, { kind: observabilityKind(initialHarness)!, companyId: agent.companyId, agentId: agent.id }) : undefined;
identityRedactor.values.push(...observabilitySecretValues(earlyObservability));
onIdentityRedactorPrepared(identityRedactor);
// The executor keeps these values in its failure redactor before any later
// preparation step can fail. Subsequent provider/identity values extend it.
onFailureSecretsResolved(() => []);
const aiBinding = preconfiguredHome || useSshHostAuth ? undefined : input.config.aiBinding;
if (selectedEnvironmentForConfig?.driver === "ssh" && selectedEnvironmentForConfig.config.useHostAuth === false && !aiBinding && !persistedNativeExecutionInput) {
throw new ConfigurationIncompleteFailure("Select an AI account for this agent or enable authentication configured on the remote machine in the SSH environment's advanced settings.", {
configurationIncomplete: { reason: "ssh_ai_connection_required", actionUrl: `/agents/${agent.id}/runtime` },
});
}
const { trustPreset } = input.policy;
const { secretsSvc, companySkills, appendRunEvent, finalizeAiConnectionBusyDeferral } = input.services;
let managedAiRuntime: ManagedAiRuntime | undefined;
const { mergedConfig, configSnapshot } = buildHeartbeatWorkspaceConfiguration(sourceWorkspaceConfig, input.policy, selectedEnvironmentId);
const executionRunConfig =
stripWorkspaceRuntimeFromExecutionRunConfig(mergedConfig);
const skillActor: AuthorizationActor = { type: "agent", agentId: agent.id, companyId: agent.companyId,
runId: run.id, source: "agent_jwt", onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId };
const [mentionedSkills, githubIdentity] = await Promise.allSettled([
preconfiguredHome || pinnedSkillTest ? Promise.resolve([]) : time("configuration_mentions", () => resolveRunScopedMentionedSkillKeys({
db,
companyId: agent.companyId,
issueId,
commentIds: extractWakeCommentIds(context),
actor: skillActor,
})),
time("configuration_github_identity", () => resolveManagedGitHubIdentitySelection(db, agent.companyId, {
agentId: agent.id, responsibleUserId, allowStandingDelegation: false,
})),
]);
if (mentionedSkills.status === "rejected") throw mentionedSkills.reason;
if (githubIdentity.status === "rejected") throw githubIdentity.reason;
const runScopedMentionedSkillKeys = mentionedSkills.value;
const runScopedSkillKeys =
!pinnedSkillTest && acceptedPlanContinuationWake &&
!acceptedPlanWakeRoutingDecision?.suppressAcceptedContinuation
? [...runScopedMentionedSkillKeys, ACCEPTED_PLAN_CONVERSION_SKILL_KEY]
: runScopedMentionedSkillKeys;
const githubSelection = githubIdentity.value;
const useHostGitHub =
!githubSelection.configured &&
trustPreset.kind === "standard" &&
["local", "ssh"].includes(
selectedEnvironmentForConfig?.driver ?? "local",
);
const { resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest } =
await time("configuration_secrets", () => resolveExecutionRunAdapterConfig({
managedAiCredentials: Boolean(aiBinding),
managedGitHubCredentials: !useHostGitHub,
companyId: agent.companyId,
agentId: agent.id,
adapterType: agent.adapterType,
issueId,
heartbeatRunId: run.id,
environmentId: selectedEnvironmentForConfig?.id ?? null,
environmentEnv: aiBinding ? stripAiAuthBindings(selectedEnvironmentForConfig?.envVars) : selectedEnvironmentForConfig?.envVars ?? null,
environmentDriver: selectedEnvironmentForConfig?.driver ?? null,
projectId: taskProjectContext?.id ?? null,
routineId: routineEnvContext.routineId,
responsibleUserId,
executionRunConfig: aiBinding ? { ...executionRunConfig, env: stripAiAuthBindings(executionRunConfig.env) } : executionRunConfig,
projectEnv: aiBinding ? stripAiAuthBindings(taskProjectContext?.env) : taskProjectContext?.env ?? null,
routineEnv: aiBinding ? stripAiAuthBindings(routineEnvContext.env) : routineEnvContext.env,
secretsSvc,
trustPreset,
}));
onFailureSecretsResolved(() => collectRunFailureSecretValues(resolvedConfig.env, secretKeys));
identityRedactor.addSecretValues(secretEnvironmentValues(resolvedConfig.env, secretKeys));
if (aiBinding) {
try {
managedAiRuntime = await time("configuration_ai_runtime", () => prepareManagedAiRuntime(db, { companyId: agent.companyId, agentId: agent.id, responsibleUserId, adapterType: agent.adapterType, binding: aiBinding, config: resolvedConfig }));
onManagedAiRuntimeAcquired(managedAiRuntime);
} catch (error) {
// Only fresh executions can receive a pre-provider wait receipt. A
// persisted native input may already have provider effects to recover.
if (isAiConnectionBusy(error) && !persistedNativeExecutionInput) {
// Use the authority recorded by the locked admission gate, never
// the issue's mutable assignee observed during runtime preparation.
const authorizedNonAssigneeWake =
parseObject(run.runnerProfileJson).aiConnectionNonAssigneeCommentWake === true ||
(run.scheduledRetryReason === AI_CONNECTION_BUSY_RETRY_REASON &&
isNonAssigneeWorkspaceBusyRetry(run.scheduledRetryReason, parseObject(run.contextSnapshot)));
await finalizeAiConnectionBusyDeferral(run, error, !authorizedNonAssigneeWake);
return { prepared: false as const };
}
if (responsibleUserId && issueId) {
await connectionIntentService(db).request({ sub: agent.id, company_id: agent.companyId, run_id: run.id, responsible_user_id: responsibleUserId }, aiBinding.provider, { purpose: "ai" }).catch(() => {
logger.warn({ runId: run.id, agentId: agent.id }, "Could not attach AI connection request; runtime configuration action remains available");
});
}
throw new ConfigurationIncompleteFailure(error instanceof Error ? error.message : "Configure this agent’s AI connection", {
configurationIncomplete: { reason: "ai_connection_unavailable", companyId: agent.companyId, agentId: agent.id, responsibleUserId,
...(!persistedNativeExecutionInput && readAiConnectionConfigurationFailure(error)
? { selectionFailure: readAiConnectionConfigurationFailure(error) } : {}),
...(readAiCredentialAccessFailure(error)
? { credentialAccess: readAiCredentialAccessFailure(error) } : {}),
provider: aiBinding.provider, method: aiBinding.method, actionUrl: `/agents/${agent.id}/runtime`,
fingerprint: `ai:${agent.id}:${responsibleUserId}:${JSON.stringify(aiBinding)}` },
});
}
const savedAiAccount = parseObject(run.contextSnapshot?.aiConnection);
if (persistedNativeExecutionInput && !managedAiSessionIdentityCompatible(
savedAiAccount.sessionIdentity ?? savedAiAccount.identity,
managedAiRuntime.sessionIdentity,
managedAiRuntime.identity,
)) {
throw new ConfigurationIncompleteFailure("The AI account changed while this native run was suspended. Start a new execution.", { configurationIncomplete: { reason: "ai_connection_changed", actionUrl: `/agents/${agent.id}/runtime` } });
}
Object.assign(resolvedConfig, managedAiRuntime.config);
for (const key of AI_AUTH_ENV_KEYS) secretKeys.add(key);
context.aiConnection = { ...managedAiRuntime.attribution, accountName: managedAiRuntime.accountName, identity: managedAiRuntime.identity, sessionIdentity: managedAiRuntime.sessionIdentity };
await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ aiConnection: context.aiConnection, ...(context.aiRouterTaskKey ? { aiRouterTaskKey: context.aiRouterTaskKey } : {}), ...(context.aiRouterSelection ? { aiRouterSelection: context.aiRouterSelection } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id));
}
if (secretManifest.length > 0) {
context.paperclipSecrets = {
manifest: secretManifest,
};
} else {
delete context.paperclipSecrets;
}
const storedVersions = parseObject(run.contextSnapshot?.skillVersions);
const hasStoredVersions = Object.prototype.hasOwnProperty.call(run.contextSnapshot ?? {}, "skillVersions");
if (hasStoredVersions && (!run.contextSnapshot?.skillVersions || typeof run.contextSnapshot.skillVersions !== "object"
|| Array.isArray(run.contextSnapshot.skillVersions) || Object.values(storedVersions).some(value =>
typeof value !== "string" || !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value)))) {
throw new ConfigurationIncompleteFailure("The saved skill selection is incomplete. Recover its exact versions before retrying.", {
configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" },
});
}
const savedShadowedSkills = Array.isArray(run.contextSnapshot?.shadowedSkills)
? run.contextSnapshot.shadowedSkills.flatMap(value => {
const saved = parseObject(value);
return typeof saved.key === "string" && typeof saved.shadowedBy === "string" && Object.hasOwn(storedVersions, saved.shadowedBy)
? [{ key: saved.key, shadowedBy: saved.shadowedBy }] : [];
}) : [];
const selectedConfig = pinnedSkillTest
? writePaperclipSkillSyncPreference(resolvedConfig, [pinnedSkillTest])
: applyRunScopedMentionedSkillKeys(resolvedConfig, runScopedSkillKeys);
const { config: effectiveResolvedConfig, skillKeys: environmentSkills, shadowedSkills } = hasStoredVersions
? { config: selectedConfig, skillKeys: [], shadowedSkills: savedShadowedSkills }
: await environmentSkillConfig(db, agent.companyId, selectedEnvironmentForConfig, selectedConfig);
if (shadowedSkills?.length) {
context.shadowedSkills = shadowedSkills;
if (!hasStoredVersions) await appendRunEvent(run, { eventType: "skills.shadowed", stream: "system", level: "warn",
message: shadowedSkills.map(({ key, shadowedBy }) => `Skill "${key}" was skipped because required plugin skill "${shadowedBy}" uses the same name.`).join("\n"),
payload: { shadowedSkills } });
}
const runtimeSkillPreference = readPaperclipSkillSyncPreference(
effectiveResolvedConfig,
);
const nativeRunnerPreparationSpans: NativeRunHistoricalSpan[] = [];
const skillsPrepareStartedAtMs = Date.now();
let skillsPrepareEndedAtMs = skillsPrepareStartedAtMs;
let skillDeliveryConfig = effectiveResolvedConfig;
// A live remote Runner already holds the files of its admitted skill versions.
// Reuse that exact selection instead of re-downloading it on this server.
const reuseAdmittedSkills = input.task.adoptingLiveRunner === true && hasStoredVersions
&& persistedNativeExecutionInput != null && "runtimeContext" in persistedNativeExecutionInput;
const prepareSkills = async () => {
try {
const versionSelections = hasStoredVersions
? new Map(Object.entries(storedVersions).filter((entry): entry is [string, string | null] => entry[1] === null || typeof entry[1] === "string"))
: skillVersionSelectionMap(runtimeSkillPreference.desiredSkillEntries, {
versionPinsEnabled: resolvedInstanceSettings.experimental.enableBetaSkills === true,
});
// Studio always executes its saved revision, independently of assignment
// pins and the Beta skills preference. Recovery keeps the admitted map.
if (!hasStoredVersions && pinnedSkillTest) versionSelections.set(pinnedSkillTest.key, pinnedSkillTest.versionId);
return await withSkillActor(skillActor, () => companySkills.withRuntimeSkillEntries(agent.companyId, {
versionSelections, ...(reuseAdmittedSkills ? { admittedVersionsOnly: true } : { snapshotLatest: true }),
...(hasStoredVersions
? { onlyKeys: new Set(Object.keys(storedVersions)) }
: { selectionConfig: effectiveResolvedConfig, includeLegacy: agent.adapterType !== "paperclip_runner" }),
selectVersions: async entries => {
const available = hasStoredVersions ? entries.filter(entry => Object.prototype.hasOwnProperty.call(storedVersions, entry.key)) : entries;
const availableKeys = new Set(available.map(entry => entry.key));
if (!hasStoredVersions && pinnedSkillTest && (!availableKeys.has(pinnedSkillTest.key)
|| shadowedSkills?.some(skill => skill.key === pinnedSkillTest.key))) {
throw new ConfigurationIncompleteFailure("The skill test's pinned skill is unavailable or no longer permitted.", {
configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" },
});
}
if (hasStoredVersions && Object.keys(storedVersions).some(key => !availableKeys.has(key))) {
throw new ConfigurationIncompleteFailure("A pinned skill is unavailable or no longer permitted.", {
configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" },
});
}
const desiredKeys = hasStoredVersions
? Object.keys(storedVersions)
: agent.adapterType === "paperclip_runner"
? resolvePaperclipDesiredSkillNames(effectiveResolvedConfig, available)
: resolveLegacyPaperclipDesiredSkillNames(effectiveResolvedConfig, available);
const selectedKeys = new Set(desiredKeys.filter(key => availableKeys.has(key)));
const resolved = available.filter(entry => selectedKeys.has(entry.key));
skillDeliveryConfig = writePaperclipSkillSyncPreference(effectiveResolvedConfig,
resolved.map(entry => ({ key: entry.key, versionId: entry.versionId ?? null })));
if (!hasStoredVersions) {
const skillVersions = Object.fromEntries(resolved.map(entry => [entry.key, entry.versionId ?? entry.currentVersionId ?? null]));
context.skillVersions = skillVersions;
context.skillSelectionSnapshot = true;
await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions, skillSelectionSnapshot: true, ...(shadowedSkills?.length ? { shadowedSkills } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id));
}
return selectedKeys;
},
}, async entries => {
for (const key of [...runScopedMentionedSkillKeys.filter(key => !shadowedSkills?.some(skill => skill.key === key)), ...(hasStoredVersions ? [] : environmentSkills)]) {
const entry = entries.find(candidate => candidate.key === key);
if (!entry || entry.sourceStatus === "missing") {
throw new ConfigurationIncompleteFailure("A referenced skill is unavailable. Restore its installed version or update the task's skill reference.", {
configurationIncomplete: { reason: "skill_unavailable", skillKey: key, actionUrl: "/skills" },
});
}
}
return entries;
}));
} catch (error) {
if (agent.adapterType === "paperclip_runner") {
await recordFailedSkillPreparation({
runId: run.id,
startedAtMs: skillsPrepareStartedAtMs,
onEvent: async (event) => {
await appendRunEvent(run, event);
},
});
}
throw error;
}
};
const skillsPreparation = (preconfiguredHome ? Promise.resolve([])
: time(reuseAdmittedSkills ? "configuration_skills_adopted" : "configuration_skills", prepareSkills))
.finally(() => { skillsPrepareEndedAtMs = Date.now(); });
const assignmentsPreparation = preconfiguredHome ? Promise.resolve([]) : time("configuration_connector_assignments", () => resolveConnectorAssignments(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id, issueId: typeof context.issueId === "string" ? context.issueId : undefined }));
// Context that MCP Apps widgets on this task reported (`ui/update-model-context`).
// One indexed read beside the other preparation; it never fails or delays the run on error.
const mcpAppContextPreparation = typeof context.issueId === "string"
? time("configuration_mcp_app_context", () => mcpAppViewsService(db).runContextForIssue(agent.companyId, context.issueId as string))
.then(renderMcpAppRunContext)
.catch((error) => {
logger.warn({ err: error, runId: run.id }, "App context from task widgets could not be loaded; the run continues without it");
return null;
})
: Promise.resolve(null);
// Approved skills from connected apps load beside company skills. A skill
// that cannot load now is left out with a warning; it never fails the run.
const mcpSkillsPreparation = preconfiguredHome ? Promise.resolve(null)
: time("configuration_mcp_skills", () => resolveMcpConnectionSkillsForRun(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }))
.catch((error) => {
logger.warn({ err: error, runId: run.id }, "Skills from connected apps could not be prepared; the run continues without them");
return null;
});
const [preparedSkills, preparedAssignments, preparedInstructions] = await Promise.allSettled([
skillsPreparation,
assignmentsPreparation,
assignmentsPreparation.then(() => time("configuration_connection_instructions", () => resolveAssignedConnectionInstructionsForRun(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }))),
]);
if (preparedSkills.status === "rejected") throw preparedSkills.reason;
if (preparedAssignments.status === "rejected") throw preparedAssignments.reason;
if (preparedInstructions.status === "rejected") throw preparedInstructions.reason;
const runtimeSkillEntries = preparedSkills.value;
if (reuseAdmittedSkills) {
await appendRunEvent(run, { eventType: "lifecycle", stream: "system", level: "info",
message: "Reconnecting to the running agent with the skills it already has",
payload: { skillPreparation: "reused_admitted", skills: runtimeSkillEntries.length } });
}
nativeRunnerPreparationSpans.push({
name: "skills.prepare",
parentName: "task.prepare",
startedAtMs: skillsPrepareStartedAtMs,
endedAtMs: skillsPrepareEndedAtMs,
});
const connectorAssignments = preparedAssignments.value;
const appliedConnectorSkills = await time("configuration_connector_skills", () => applyConnectorSkills(skillDeliveryConfig, runtimeSkillEntries, connectorAssignments));
const mcpSkills = await mcpSkillsPreparation;
if (mcpSkills?.skipped.length) {
await appendRunEvent(run, { eventType: "skills.mcp_unavailable", stream: "system", level: "warn",
message: mcpSkills.skipped.map(({ skillName, appName, reason }) => skillName ? `Skill "${skillName}" from ${appName} was not loaded: ${reason}` : reason).join("\n"),
payload: { skipped: mcpSkills.skipped } });
}
const connectorSkillConfig = mcpSkills?.loaded.length
? await time("configuration_mcp_skill_bundles", () => applyMcpConnectionSkills(appliedConnectorSkills, mcpSkills.loaded))
: appliedConnectorSkills;
// Both CLI adapters and native context materialization use the same resolved set.
runtimeSkillEntries.splice(0, runtimeSkillEntries.length, ...connectorSkillConfig.paperclipRuntimeSkills);
const connectorDelivery = await time("configuration_connector_delivery", () => prepareConnectorSkillDelivery(connectorSkillConfig, agent.adapterType));
// Always replace this runtime-only field; caller wake data cannot supply skills.
// App context is server-owned too: a caller's wake data cannot supply it.
context.paperclipWake = {
...parseObject(context.paperclipWake),
connectorSkillInstructions: connectorDelivery.instructions,
mcpAppContext: await mcpAppContextPreparation,
};
let runtimeConfig = await prepareConnectionInstructionDelivery({
resolve: async () => preparedInstructions.value,
context, config: connectorDelivery.config, native: agent.adapterType === "paperclip_runner",
});
const agentIdentity = supportsManagedAgentIdentity(agent.adapterType, agent.adapterConfig, run.runtimeMode === "native" ? run.driverKind : undefined)
? await time("configuration_agent_identity", () => agentIdentityService(db).ensureAgentIdentity(agent.companyId, agent.id))
: undefined;
identityRedactor.values.push(...createAgentIdentityRedactor(agentIdentity?.privateKeyPem).values);
const harness = agentHarnessType(agent.adapterType, runtimeConfig);
const observability = unsupportedObservability !== "unsupported_home_mode" && observabilityKind(harness)
? agentObservabilityLaunch(observabilitySnapshot, { kind: observabilityKind(harness)!, companyId: agent.companyId, agentId: agent.id }) : undefined;
identityRedactor.values.push(...observabilitySecretValues(observability));
identityRedactor.values.sort((a, b) => b.length - a.length);
if (observabilitySnapshot.state === "unavailable") {
logger.warn({ runId: run.id, reason: observabilitySnapshot.reason }, "Agent observability unavailable; execution continues without export");
const reason = observabilitySnapshot.reason;
await appendRunEvent(run, { eventType: "agent_observability.unavailable", stream: "system", level: "warn",
message: reason === "unsupported_home_mode"
? "Company observability was not applied: automatic configuration of caller-owned Codex homes is unsupported."
: reason === "unsupported_harness" ? "Company observability was not applied: this harness is not supported."
: "Company observability is unavailable. Execution continues without company export.",
payload: { reason, revision: observabilitySnapshot.revision } }).catch(() => {
logger.warn({ runId: run.id }, "Could not record the agent observability availability diagnostic");
});
}
onIdentityRedactorPrepared(identityRedactor);
if (agentIdentity) secretKeys.add("PAPERCLIP_AGENT_PRIVATE_KEY");
return {
prepared: true as const,
mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub,
resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest,
runtimeSkillEntries, nativeRunnerPreparationSpans, runtimeConfig, agentIdentity, observability,
};
}