mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
## Thinking Path > - Paperclip gives teams durable tasks, agent execution, budgets, and approvals. > - People also use assistants in Codex, Claude, and other MCP clients. > - Those assistants need a scoped connection that preserves the person’s permissions and attribution. > - Delegating a task must not turn the assistant into the assigned agent. > - This PR adds opt-in user OAuth, ten first-party tools, browser consent, and workflow packages. > - Paid product evals verify the resulting tasks, documents, attribution, retries, and access boundaries. > - The team keeps working after the assistant conversation ends. ## Linked Issues or Issue Description **Problem or motivation** A person cannot connect an external assistant to an existing team through browser consent and safely delegate durable work as themselves. **Proposed solution** Expose an opt-in `/mcp/paperclip` endpoint with individually described first-party operations. Bind every connection to a person, client, company, resource, and scopes. Reuse domain authorization and scheduling. Package shared team-review, delegation, and follow-up workflows for OpenAI/Codex and Claude. **Alternatives considered** Related PRs #9393 and #12549 cover earlier remote MCP and board-operator approaches. This change uses user OAuth and a bounded public catalog. It does not expose a generic executor, operator administration, static shared board credentials, or external agent execution. Registry listing work in #9851 is a separate distribution step. **Roadmap alignment** This maintainer-requested implementation extends the governed MCP gateway, activity attribution, durable work products, and hosted deployment direction in `ROADMAP.md`. It implements the first release of the saved design plan; external agent participation and granted third-party tools remain later releases. ## What Changed - Add MCP 2.0 discovery and task status/comment/document Events on the same authenticated endpoint. Persist subscriptions and delivery receipts, verify HTTPS callbacks, sign Standard Webhooks, encrypt callback material, recheck permissions/Cloud membership, and bound retries/expiry. Older MCP clients keep their existing tools. - Add discovery, dynamic client registration, S256 PKCE, resource validation, rotating refresh tokens, revocation, and company consent. Store credentials as hashes and recheck membership at execution. - Add tools for connection identity, agents/projects, task search/read/create, human comments, documents/deliverables, and pending-approval links. Preserve current domain permissions and scheduling. - Add durable mutation receipts across reconnects. Matching retries replay results; uncertain outcomes keep the same request ID and require inspection. - Add consent and connection-management pages, OAuth log redaction, shared plugin workflows, and separate OpenAI/Codex and Claude package outputs. - Add eight paid Product E2E cases across three models, independent durable-state grading, usage evidence, cleanup, and report integration. Add task-document guidance and regenerate the runner capability inventories. - Add migrations 0301 and 0302, the dated implementation plan, result notes, and direct-client setup instructions in `doc/public-mcp.md`. ## Verification - Merge integration `e180b1948`: resolved conflicts with current master, preserved both eval registries, regenerated capability catalogs, and regenerated migrations as 0301/0302 while keeping the original replay-safe SQL byte-identical. Local migration safety/snapshot tests (26), MCP/OAuth tests (38), redaction/OpenAPI tests (71), and eval catalog/grading tests (198) pass. Token and capability gates pass. Full recursive typecheck passed. Fresh Greptile review is 5/5 with no unresolved findings. CI is green on this exact head (55 successes, two intentional skips, one neutral result): one unchanged Cursor sandbox test timed out at 10 seconds, then passed locally in 856 ms. A single retry of that failed shard and the aggregate workflow passed. Merge remains blocked on the repository code-owner approval rule. Earlier checks passed at `6aa0962d4fb715f2190bb7bb22efacab2e58495d`: 55 successes, two intentional skips and one neutral result. [The earlier CI run](https://github.com/paperclipai/paperclip/actions/runs/36901592350) includes all test shards, browser tests, typecheck, build and canary dry run. Greptile was 5/5 on that commit with no unresolved review threads. GitHub still requires code-owner review under the repository merge rules; passing checks do not bypass that approval. Paid source fingerprints remain separate below and in the dated result note. - Paid Events qualification passes **3/3**: GPT-5.4 Mini, Claude Haiku 4.5 and Claude Sonnet 4.6. Each uses a real public HTTPS callback, signature verification and report retrieval in a fresh conversation. A final Mini regression passes after the quota/status fixes. All evidence validates. Bounded tunnel startup retries occur before provider calls and remain visible; failed earlier attempts retain their original grades. - The earlier complete seven-case matrix passes **21/21**, with a separate **3/3** delegation regression. Two preceding matrices also passed 21/21 each. A complete 24-cell matrix including Events has not been run. [The dated results](doc/plans/2026-10-01-public-mcp-paid-eval-results.md) retain exact source fingerprints, failures, model IDs and partial costs. - Node 24: repository-wide `pnpm -r typecheck` and `pnpm build` pass after merging master. Server typecheck passes after the final quota/status changes. Eval typecheck and all 892 eval-support tests pass. - All 33 real MCP/OAuth tests pass. The preceding combined MCP, redaction, private-address and DNS-rebinding run passed 129 tests; two later MCP regressions cover quota reuse and unchanged-status suppression. All 28 adjacent issue-tree/stale-lock route tests pass. CI then found a null checkout result in the existing concurrent-workspace path; logging now uses optional status access. All 12 closed-workspace tests and all 33 MCP tests pass after that correction. The exact-start event calibration exposed a timestamp gap; scanning now includes the subscription start, with all 33 MCP tests and server typecheck passing. These two narrow corrections follow the paid regression. - A real Core → Cloud → Core authority round trip passes OAuth, MCP 2.0 subscription/delivery, current membership loss, unsubscribe, legacy SDK tools, refresh and revocation. Its callback transport is a fixture with independent HMAC verification. The paid Events campaigns separately prove public HTTPS delivery. - Earlier component qualification passed UI 7,117 tests, CLI 502, shared 832, skills catalog 20, database 160 and OpenAPI 10. Token gates, module boundaries, migration order and plugin regeneration passed. CI covers general/serialized suites, eight browser shards, runner checks, typecheck, build and canary dry run. - **Local full-suite limitation:** the earlier monolithic run was not clean. It encountered overlapping schema rebuilding, Mac database shared-memory limits and isolated CLI/fixture failures. Targeted reruns passed. The existing >32 MiB Git filename stress test still hit its 300-second Mac timeout. The additional serialized sweep stopped after 62 passing suites once CI passed. Original failures and partial logs remain; this PR does not claim a wholly green local monolithic run. - Local Codex CLI and Claude Code OAuth login and MCP SDK interoperability were verified. Public-store installation, actual ChatGPT Work Cloud Events UI, staging HTTPS client behavior and hosted newcomer provisioning remain release gates. Enablement is moving to **Settings → Experimental → Assistant connections (MCP)** in the stacked follow-up [#14933](https://github.com/paperclipai/paperclip/pull/14933). Merge both for the intended setup experience. This foundation branch alone still uses `PAPERCLIP_PUBLIC_MCP_ENABLED=true`. After deployment, set `PAPERCLIP_PUBLIC_URL` to the authenticated instance's HTTPS origin, and connect to `/mcp/paperclip`. Select a team and allow writes in browser consent. Configure an available agent and budget, then delegate and retrieve results later. For Events, rescan the deployed plugin catalog in ChatGPT Work Cloud; the host supplies its webhook credentials when the user asks to watch a task. See [the setup runbook](doc/public-mcp.md). ## Risks - Events are at-least-once and may arrive out of order. No replay cursor is advertised. Clients must refresh finite subscriptions, read current state and avoid comment feedback loops. Callback material uses the instance secrets master key; hosted subscriptions require the updated Cloud broker and are bounded to five minutes/the access proof expiry. - ChatGPT Work Cloud/dot event UI, plugin rescan and a hosted staging subscription remain deployment gates. Local signed-webhook and paid model evidence does not claim those surfaces have been exercised. - Disabled by default. Merging adds schema and opt-in code; it does not deploy a public endpoint, publish a store listing, create a team, or start paid agents. - Migrations 0301 and 0302 are additive and idempotent. Their SQL is unchanged from the earlier preview numbers, so hash-aware upgrade reconciliation preserves prior staging applications. Normal instance upgrades must apply it before enabling MCP. - Task creation and comments can schedule paid agent work. Consent and tool descriptions disclose that effect. Revocation blocks future calls but does not undo delegated work. - Public deployments need edge rate limits and credential-safe logging. Internal dispatch is restricted to the closed catalog and carries a request-local verified actor. - Hosted onboarding requires the companion Cloud broker, encryption-key configuration, and tenant rollout. Self-hosted direct connections can use this PR alone. - Store acceptance and agent-mode participation are not claimed. Checked-in plugin endpoints are development defaults; rebuild packages for a real deployment before installation. ## Model Used OpenAI GPT-6 in Codex, with reasoning, tool use, and code execution. A more specific serving version and context-window size were not exposed by the session. Paid eval models: `gpt-5.4-mini-2026-03-17`, `claude-haiku-4-5-20251001`, and `claude-sonnet-4-6`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted/component checks; full local-run limitations are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
827 lines
29 KiB
TypeScript
827 lines
29 KiB
TypeScript
import { z } from "zod";
|
|
import { randomUUID } from "node:crypto";
|
|
import type { Db } from "@paperclipai/db";
|
|
import type {
|
|
Environment,
|
|
EnvironmentDriver,
|
|
FakeSandboxEnvironmentConfig,
|
|
LocalEnvironmentConfig,
|
|
PluginEnvironmentConfig,
|
|
PluginSandboxEnvironmentConfig,
|
|
SandboxEnvironmentConfig,
|
|
SecretProvider,
|
|
SecretVersionSelector,
|
|
SshEnvironmentConfig,
|
|
} from "@paperclipai/shared";
|
|
import { unprocessable } from "../errors.js";
|
|
import { parseObject } from "../adapters/utils.js";
|
|
import { secretService } from "./secrets.js";
|
|
import {
|
|
resolvePluginSandboxProviderDriverByKey,
|
|
validatePluginEnvironmentDriverConfig,
|
|
validatePluginSandboxProviderConfig,
|
|
} from "./plugin-environment-driver.js";
|
|
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
|
import {
|
|
collectSecretRefPaths,
|
|
isUuidSecretRef,
|
|
parseSecretRefBindingObject,
|
|
readConfigValueAtPath,
|
|
writeConfigValueAtPath,
|
|
} from "./json-schema-secret-refs.js";
|
|
import { resolveActiveEnvironmentCustomImageTemplateForRuntime } from "./environment-custom-image-runtime.js";
|
|
|
|
const secretRefSchema = z.object({
|
|
type: z.literal("secret_ref"),
|
|
secretId: z.string().guid(),
|
|
version: z.union([z.literal("latest"), z.number().int().positive()]).optional().default("latest"),
|
|
}).strict();
|
|
|
|
const sshEnvironmentConfigSchema = z.object({
|
|
host: z.string({ error: "SSH environments require a host." }).trim().min(1, "SSH environments require a host."),
|
|
port: z.coerce.number().int().min(1).max(65535).default(22),
|
|
username: z.string({ error: "SSH environments require a username." }).trim().min(1, "SSH environments require a username."),
|
|
remoteWorkspacePath: z
|
|
.string({ error: "SSH environments require a remote workspace path." })
|
|
.trim()
|
|
.min(1, "SSH environments require a remote workspace path.")
|
|
.refine((value) => value.startsWith("/"), "SSH remote workspace path must be absolute."),
|
|
privateKey: z.null().optional().default(null),
|
|
privateKeySecretRef: secretRefSchema.optional().nullable().default(null),
|
|
knownHosts: z
|
|
.string()
|
|
.trim()
|
|
.optional()
|
|
.nullable()
|
|
.transform((value) => (value && value.length > 0 ? value : null)),
|
|
strictHostKeyChecking: z.boolean().optional().default(true),
|
|
}).strict();
|
|
|
|
const sshEnvironmentConfigProbeSchema = sshEnvironmentConfigSchema.extend({
|
|
privateKey: z
|
|
.string()
|
|
.trim()
|
|
.optional()
|
|
.nullable()
|
|
.transform((value) => (value && value.length > 0 ? value : null)),
|
|
}).strict();
|
|
|
|
const sshEnvironmentConfigPersistenceSchema = sshEnvironmentConfigProbeSchema;
|
|
|
|
const fakeSandboxEnvironmentConfigSchema = z.object({
|
|
provider: z.literal("fake").default("fake"),
|
|
image: z
|
|
.string()
|
|
.trim()
|
|
.min(1, "Fake sandbox environments require an image.")
|
|
.default("ubuntu:24.04"),
|
|
reuseLease: z.boolean().optional().default(false),
|
|
streamRunLogs: z.boolean().optional(),
|
|
runnerLifecycleMode: z.enum(["inherit", "per_turn", "warm"]).optional(),
|
|
runnerIdleTimeoutMs: z.coerce
|
|
.number()
|
|
.int()
|
|
.min(1_000)
|
|
.max(86_400_000)
|
|
.optional(),
|
|
archiveOnRelease: z.boolean().optional(),
|
|
}).strict();
|
|
|
|
const pluginSandboxProviderKeySchema = z.string()
|
|
.trim()
|
|
.min(1, "Sandbox provider is required.")
|
|
.regex(
|
|
/^[a-z0-9][a-z0-9._-]*$/,
|
|
"Sandbox provider key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores",
|
|
);
|
|
|
|
const pluginSandboxEnvironmentConfigSchema = z.object({
|
|
provider: pluginSandboxProviderKeySchema,
|
|
timeoutMs: z.coerce.number().int().min(1).max(86_400_000).optional(),
|
|
reuseLease: z.boolean().optional().default(false),
|
|
streamRunLogs: z.boolean().optional(),
|
|
runnerLifecycleMode: z.enum(["inherit", "per_turn", "warm"]).optional(),
|
|
runnerIdleTimeoutMs: z.coerce
|
|
.number()
|
|
.int()
|
|
.min(1_000)
|
|
.max(86_400_000)
|
|
.optional(),
|
|
archiveOnRelease: z.boolean().optional(),
|
|
}).catchall(z.unknown());
|
|
|
|
const pluginEnvironmentConfigSchema = z.object({
|
|
pluginKey: z.string().min(1),
|
|
driverKey: z.string().min(1).regex(
|
|
/^[a-z0-9][a-z0-9._-]*$/,
|
|
"Environment driver key must start with a lowercase alphanumeric and contain only lowercase letters, digits, dots, hyphens, or underscores",
|
|
),
|
|
driverConfig: z.record(z.string(), z.unknown()).optional().default({}),
|
|
}).strict();
|
|
|
|
export type ParsedEnvironmentConfig =
|
|
| { driver: "local"; config: LocalEnvironmentConfig }
|
|
| { driver: "ssh"; config: SshEnvironmentConfig }
|
|
| { driver: "sandbox"; config: SandboxEnvironmentConfig }
|
|
| { driver: "plugin"; config: PluginEnvironmentConfig };
|
|
|
|
function toErrorMessage(error: z.ZodError) {
|
|
const first = error.issues[0];
|
|
if (!first) return "Invalid environment config.";
|
|
return first.message;
|
|
}
|
|
|
|
function getSandboxProvider(raw: Record<string, unknown>) {
|
|
return typeof raw.provider === "string" && raw.provider.trim().length > 0 ? raw.provider.trim() : "fake";
|
|
}
|
|
|
|
// Operator flags removed when session-output streaming moved to the verified
|
|
// capability snapshot. A saved environment config can still carry a removed key.
|
|
// The server now decides session-output streaming from the effective capability
|
|
// snapshot alone, so no consumer reads these flags. Strip a removed key before
|
|
// validation so a strict schema (the fake sandbox) still loads an old config,
|
|
// and so the removed flag never reaches the parsed config.
|
|
const REMOVED_SANDBOX_CONFIG_KEYS = ["streamAgentSessionOutput"] as const;
|
|
|
|
function stripRemovedSandboxConfigKeys(raw: Record<string, unknown>): Record<string, unknown> {
|
|
if (!REMOVED_SANDBOX_CONFIG_KEYS.some((key) => key in raw)) {
|
|
return raw;
|
|
}
|
|
const next = { ...raw };
|
|
for (const key of REMOVED_SANDBOX_CONFIG_KEYS) {
|
|
delete next[key];
|
|
}
|
|
return next;
|
|
}
|
|
|
|
function parseSandboxEnvironmentConfig(
|
|
input: Record<string, unknown> | null | undefined,
|
|
) {
|
|
const raw = stripRemovedSandboxConfigKeys(parseObject(input));
|
|
const provider = getSandboxProvider(raw);
|
|
|
|
if (provider === "fake") {
|
|
const parsed = fakeSandboxEnvironmentConfigSchema.safeParse(raw);
|
|
return parsed.success
|
|
? ({ success: true as const, data: parsed.data satisfies FakeSandboxEnvironmentConfig })
|
|
: ({ success: false as const, error: parsed.error });
|
|
}
|
|
|
|
const parsed = pluginSandboxEnvironmentConfigSchema.safeParse(raw);
|
|
return parsed.success
|
|
? ({ success: true as const, data: parsed.data satisfies PluginSandboxEnvironmentConfig })
|
|
: ({ success: false as const, error: parsed.error });
|
|
}
|
|
|
|
async function getSandboxProviderConfigSchema(
|
|
db: Db,
|
|
provider: string,
|
|
): Promise<Record<string, unknown> | null> {
|
|
const resolved = await resolvePluginSandboxProviderDriverByKey({
|
|
db,
|
|
driverKey: provider,
|
|
});
|
|
const schema = resolved?.driver.configSchema;
|
|
return schema && typeof schema === "object" && !Array.isArray(schema)
|
|
? schema as Record<string, unknown>
|
|
: null;
|
|
}
|
|
|
|
export async function resolveSandboxProviderSecretRefPaths(
|
|
db: Db,
|
|
provider: string,
|
|
): Promise<Set<string>> {
|
|
return collectSecretRefPaths(await getSandboxProviderConfigSchema(db, provider));
|
|
}
|
|
|
|
function secretName(input: {
|
|
environmentName: string;
|
|
driver: EnvironmentDriver;
|
|
field: string;
|
|
}) {
|
|
const slug = input.environmentName
|
|
.toLowerCase()
|
|
.replace(/[^a-z0-9]+/g, "-")
|
|
.replace(/^-+|-+$/g, "")
|
|
.slice(0, 48) || "environment";
|
|
return `environment-${input.driver}-${slug}-${input.field}-${randomUUID().slice(0, 8)}`;
|
|
}
|
|
|
|
async function createEnvironmentSecret(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
environmentName: string;
|
|
driver: EnvironmentDriver;
|
|
field: string;
|
|
provider: SecretProvider;
|
|
value: string;
|
|
actor?: { userId?: string | null; agentId?: string | null };
|
|
}) {
|
|
const created = await secretService(input.db).create(
|
|
input.companyId,
|
|
{
|
|
name: secretName(input),
|
|
provider: input.provider,
|
|
value: input.value,
|
|
description: `Secret for ${input.environmentName} ${input.field}.`,
|
|
},
|
|
input.actor,
|
|
);
|
|
return {
|
|
type: "secret_ref" as const,
|
|
secretId: created.id,
|
|
version: "latest" as const,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Secret pickers submit `{ type: "secret_ref", secretId, version }` binding
|
|
* objects for `format: "secret-ref"` fields, while persisted configs store the
|
|
* bare secret id. Collapse binding objects to the secret id so every consumer
|
|
* downstream deals with one shape. Sandbox provider references always resolve
|
|
* the latest version, so pinned bindings are rejected rather than silently
|
|
* resolved to a different version than the caller asked for.
|
|
*/
|
|
function canonicalizeSecretRefValue(value: unknown, path: string): unknown {
|
|
const binding = parseSecretRefBindingObject(value);
|
|
if (!binding) return value;
|
|
if (binding.version !== "latest") {
|
|
throw unprocessable(
|
|
`Secret binding at ${path} pins version ${binding.version}; sandbox provider secret references always resolve the latest version.`,
|
|
);
|
|
}
|
|
return binding.secretId;
|
|
}
|
|
|
|
async function persistConfigSecretRefs(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
environmentName: string;
|
|
driver: EnvironmentDriver;
|
|
secretProvider: SecretProvider;
|
|
config: Record<string, unknown>;
|
|
schema: Record<string, unknown> | null;
|
|
actor?: { userId?: string | null; agentId?: string | null };
|
|
}): Promise<Record<string, unknown>> {
|
|
let nextConfig = { ...input.config };
|
|
for (const path of collectSecretRefPaths(input.schema)) {
|
|
const rawValue = canonicalizeSecretRefValue(readConfigValueAtPath(nextConfig, path), path);
|
|
if (typeof rawValue !== "string") continue;
|
|
const trimmed = rawValue.trim();
|
|
if (trimmed.length === 0) {
|
|
nextConfig = writeConfigValueAtPath(nextConfig, path, undefined);
|
|
continue;
|
|
}
|
|
if (isUuidSecretRef(trimmed)) {
|
|
nextConfig = writeConfigValueAtPath(nextConfig, path, trimmed);
|
|
continue;
|
|
}
|
|
const created = await createEnvironmentSecret({
|
|
db: input.db,
|
|
companyId: input.companyId,
|
|
environmentName: input.environmentName,
|
|
driver: input.driver,
|
|
field: path.replace(/[^a-z0-9]+/gi, "-").toLowerCase(),
|
|
provider: input.secretProvider,
|
|
value: trimmed,
|
|
actor: input.actor,
|
|
});
|
|
nextConfig = writeConfigValueAtPath(nextConfig, path, created.secretId);
|
|
}
|
|
return nextConfig;
|
|
}
|
|
|
|
async function resolveConfigSecretRefsForRuntime(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
config: Record<string, unknown>;
|
|
schema: Record<string, unknown> | null;
|
|
context: {
|
|
consumerId: string;
|
|
issueId?: string | null;
|
|
heartbeatRunId?: string | null;
|
|
};
|
|
}): Promise<Record<string, unknown>> {
|
|
const secrets = secretService(input.db);
|
|
let nextConfig = { ...input.config };
|
|
for (const path of collectSecretRefPaths(input.schema)) {
|
|
const current = canonicalizeSecretRefValue(readConfigValueAtPath(nextConfig, path), path);
|
|
if (typeof current !== "string") continue;
|
|
const trimmed = current.trim();
|
|
if (!isUuidSecretRef(trimmed)) continue;
|
|
if (!input.context.consumerId) {
|
|
throw unprocessable("Runtime secret resolution requires an environment id");
|
|
}
|
|
nextConfig = writeConfigValueAtPath(
|
|
nextConfig,
|
|
path,
|
|
await secrets.resolveSecretValue(input.companyId, trimmed, "latest", {
|
|
consumerType: "environment",
|
|
consumerId: input.context.consumerId,
|
|
actorType: "system",
|
|
actorId: null,
|
|
issueId: input.context.issueId ?? null,
|
|
heartbeatRunId: input.context.heartbeatRunId ?? null,
|
|
configPath: path,
|
|
}),
|
|
);
|
|
}
|
|
return nextConfig;
|
|
}
|
|
|
|
async function resolveConfigSecretRefsForProbe(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
config: Record<string, unknown>;
|
|
schema: Record<string, unknown> | null;
|
|
accessContext?: {
|
|
actorType: "agent" | "user";
|
|
actorId: string;
|
|
actorSource?: "local_implicit" | "session" | "board_key" | "agent_key" | "agent_jwt" | "mcp_oauth" | "cloud_tenant";
|
|
heartbeatRunId?: string | null;
|
|
};
|
|
}): Promise<Record<string, unknown>> {
|
|
const secrets = secretService(input.db);
|
|
let nextConfig = { ...input.config };
|
|
for (const path of collectSecretRefPaths(input.schema)) {
|
|
const current = canonicalizeSecretRefValue(readConfigValueAtPath(nextConfig, path), path);
|
|
if (typeof current !== "string") continue;
|
|
const trimmed = current.trim();
|
|
if (!isUuidSecretRef(trimmed)) continue;
|
|
// Unsaved draft probes do not have an environment record yet, so they
|
|
// cannot rely on environment-bound secret resolution. Resolve directly for
|
|
// this ephemeral board-only probe and never persist the plaintext value.
|
|
nextConfig = writeConfigValueAtPath(
|
|
nextConfig,
|
|
path,
|
|
await secrets.resolveSecretValueForEphemeralAccess(input.companyId, trimmed, "latest", {
|
|
consumerType: "system",
|
|
consumerId: "environment-probe-config",
|
|
configPath: path,
|
|
actorType: input.accessContext?.actorType ?? "system",
|
|
actorId: input.accessContext?.actorId ?? null,
|
|
actorSource: input.accessContext?.actorSource,
|
|
heartbeatRunId: input.accessContext?.heartbeatRunId ?? null,
|
|
}),
|
|
);
|
|
}
|
|
return nextConfig;
|
|
}
|
|
|
|
export async function collectEnvironmentSecretRefs(input: {
|
|
db: Db;
|
|
environment: Pick<Environment, "id" | "driver" | "config">;
|
|
}): Promise<Array<{ secretId: string; configPath: string; versionSelector?: SecretVersionSelector }>> {
|
|
const parsed = parseEnvironmentDriverConfig(input.environment);
|
|
if (parsed.driver === "ssh" && parsed.config.privateKeySecretRef) {
|
|
return [{
|
|
secretId: parsed.config.privateKeySecretRef.secretId,
|
|
configPath: "privateKeySecretRef",
|
|
versionSelector: parsed.config.privateKeySecretRef.version ?? "latest",
|
|
}];
|
|
}
|
|
if (parsed.driver === "sandbox" && parsed.config.provider !== "fake") {
|
|
const schema = await getSandboxProviderConfigSchema(input.db, parsed.config.provider);
|
|
const refs: Array<{ secretId: string; configPath: string; versionSelector?: SecretVersionSelector }> = [];
|
|
for (const path of collectSecretRefPaths(schema)) {
|
|
const current = readConfigValueAtPath(parsed.config as Record<string, unknown>, path);
|
|
const binding = parseSecretRefBindingObject(current);
|
|
if (binding) {
|
|
refs.push({ secretId: binding.secretId, configPath: path, versionSelector: binding.version });
|
|
continue;
|
|
}
|
|
if (typeof current === "string" && isUuidSecretRef(current.trim())) {
|
|
refs.push({ secretId: current.trim(), configPath: path, versionSelector: "latest" });
|
|
}
|
|
}
|
|
return refs;
|
|
}
|
|
return [];
|
|
}
|
|
|
|
export function stripSandboxProviderEnvelope(config: SandboxEnvironmentConfig): Record<string, unknown> {
|
|
const {
|
|
provider: _provider,
|
|
streamRunLogs: _streamRunLogs,
|
|
...driverConfig
|
|
} = config as Record<string, unknown>;
|
|
return driverConfig;
|
|
}
|
|
|
|
// The host owns these sandbox run-behavior flags, not the provider plugin. The
|
|
// host passes the whole config to the plugin, so a plugin that allowlists only
|
|
// its provider fields may drop them from its normalized config. Re-apply them
|
|
// from the parsed envelope after plugin normalization. Otherwise a saved
|
|
// environment can silently fall back to per-turn runner behavior even though
|
|
// the caller explicitly selected a warm lifecycle.
|
|
const HOST_OWNED_SANDBOX_FLAGS = [
|
|
"streamRunLogs",
|
|
"runnerLifecycleMode",
|
|
"runnerIdleTimeoutMs",
|
|
] as const;
|
|
|
|
function applyHostOwnedSandboxFlags(
|
|
normalizedConfig: Record<string, unknown>,
|
|
envelope: Record<string, unknown>,
|
|
): Record<string, unknown> {
|
|
const merged: Record<string, unknown> = { ...normalizedConfig };
|
|
for (const key of HOST_OWNED_SANDBOX_FLAGS) {
|
|
if (envelope[key] !== undefined) {
|
|
merged[key] = envelope[key];
|
|
}
|
|
}
|
|
return merged;
|
|
}
|
|
|
|
export function normalizeEnvironmentConfig(input: {
|
|
driver: EnvironmentDriver;
|
|
config: Record<string, unknown> | null | undefined;
|
|
}): Record<string, unknown> {
|
|
if (input.driver === "local") {
|
|
return { ...parseObject(input.config) };
|
|
}
|
|
|
|
if (input.driver === "ssh") {
|
|
const parsed = sshEnvironmentConfigSchema.safeParse(parseObject(input.config));
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
return parsed.data satisfies SshEnvironmentConfig;
|
|
}
|
|
|
|
if (input.driver === "sandbox") {
|
|
const parsed = parseSandboxEnvironmentConfig(input.config);
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
return parsed.data;
|
|
}
|
|
|
|
if (input.driver === "plugin") {
|
|
const parsed = pluginEnvironmentConfigSchema.safeParse(parseObject(input.config));
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
return parsed.data satisfies PluginEnvironmentConfig;
|
|
}
|
|
|
|
throw unprocessable(`Unsupported environment driver "${input.driver}".`);
|
|
}
|
|
|
|
export function normalizeEnvironmentConfigForProbe(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
driver: EnvironmentDriver;
|
|
config: Record<string, unknown> | null | undefined;
|
|
accessContext?: {
|
|
actorType: "agent" | "user";
|
|
actorId: string;
|
|
actorSource?: "local_implicit" | "session" | "board_key" | "agent_key" | "agent_jwt" | "mcp_oauth" | "cloud_tenant";
|
|
heartbeatRunId?: string | null;
|
|
};
|
|
pluginWorkerManager?: PluginWorkerManager;
|
|
}): Promise<Record<string, unknown>> | Record<string, unknown> {
|
|
if (input.driver === "ssh") {
|
|
const parsed = sshEnvironmentConfigProbeSchema.safeParse(parseObject(input.config));
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
return parsed.data satisfies SshEnvironmentConfig;
|
|
}
|
|
|
|
if (input.driver === "sandbox") {
|
|
const parsed = parseSandboxEnvironmentConfig(input.config);
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
if (parsed.data.provider === "fake") {
|
|
return parsed.data;
|
|
}
|
|
if (!input.pluginWorkerManager) {
|
|
throw unprocessable("Sandbox provider config validation requires a running plugin worker manager.");
|
|
}
|
|
return validatePluginSandboxProviderConfig({
|
|
db: input.db,
|
|
workerManager: input.pluginWorkerManager,
|
|
provider: parsed.data.provider,
|
|
config: stripSandboxProviderEnvelope(parsed.data),
|
|
}).then(async (validated) => ({
|
|
provider: parsed.data.provider,
|
|
...(await resolveConfigSecretRefsForProbe({
|
|
db: input.db,
|
|
companyId: input.companyId,
|
|
config: applyHostOwnedSandboxFlags(
|
|
validated.normalizedConfig,
|
|
parsed.data,
|
|
),
|
|
accessContext: input.accessContext,
|
|
schema:
|
|
validated.driver.configSchema &&
|
|
typeof validated.driver.configSchema === "object" &&
|
|
!Array.isArray(validated.driver.configSchema)
|
|
? validated.driver.configSchema as Record<string, unknown>
|
|
: null,
|
|
})),
|
|
}));
|
|
}
|
|
|
|
return normalizeEnvironmentConfig({
|
|
driver: input.driver,
|
|
config: input.config,
|
|
});
|
|
}
|
|
|
|
export async function normalizeEnvironmentConfigForPersistence(input: {
|
|
db: Db;
|
|
companyId: string;
|
|
environmentName: string;
|
|
driver: EnvironmentDriver;
|
|
secretProvider: SecretProvider;
|
|
config: Record<string, unknown> | null | undefined;
|
|
actor?: { userId?: string | null; agentId?: string | null };
|
|
pluginWorkerManager?: PluginWorkerManager;
|
|
}): Promise<Record<string, unknown>> {
|
|
if (input.driver === "ssh") {
|
|
const parsed = sshEnvironmentConfigPersistenceSchema.safeParse(parseObject(input.config));
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
const secrets = secretService(input.db);
|
|
const { privateKey, ...stored } = parsed.data;
|
|
let nextPrivateKeySecretRef = stored.privateKeySecretRef;
|
|
if (privateKey) {
|
|
nextPrivateKeySecretRef = await createEnvironmentSecret({
|
|
db: input.db,
|
|
companyId: input.companyId,
|
|
environmentName: input.environmentName,
|
|
driver: input.driver,
|
|
field: "private-key",
|
|
provider: input.secretProvider,
|
|
value: privateKey,
|
|
actor: input.actor,
|
|
});
|
|
if (
|
|
stored.privateKeySecretRef &&
|
|
stored.privateKeySecretRef.secretId !== nextPrivateKeySecretRef.secretId
|
|
) {
|
|
await secrets.remove(stored.privateKeySecretRef.secretId);
|
|
}
|
|
}
|
|
return {
|
|
...stored,
|
|
privateKey: null,
|
|
privateKeySecretRef: nextPrivateKeySecretRef,
|
|
} satisfies SshEnvironmentConfig;
|
|
}
|
|
|
|
if (input.driver === "sandbox") {
|
|
const parsed = parseSandboxEnvironmentConfig(input.config);
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
if (parsed.data.provider === "fake") {
|
|
throw unprocessable(
|
|
"Built-in fake sandbox environments are reserved for internal probes and cannot be saved.",
|
|
);
|
|
}
|
|
if (!input.pluginWorkerManager) {
|
|
throw unprocessable("Sandbox provider config validation requires a running plugin worker manager.");
|
|
}
|
|
const validated = await validatePluginSandboxProviderConfig({
|
|
db: input.db,
|
|
workerManager: input.pluginWorkerManager,
|
|
provider: parsed.data.provider,
|
|
config: stripSandboxProviderEnvelope(parsed.data),
|
|
});
|
|
return await persistConfigSecretRefs({
|
|
db: input.db,
|
|
companyId: input.companyId,
|
|
environmentName: input.environmentName,
|
|
driver: input.driver,
|
|
secretProvider: input.secretProvider,
|
|
config: {
|
|
provider: parsed.data.provider,
|
|
...applyHostOwnedSandboxFlags(validated.normalizedConfig, parsed.data),
|
|
},
|
|
schema:
|
|
validated.driver.configSchema && typeof validated.driver.configSchema === "object" && !Array.isArray(validated.driver.configSchema)
|
|
? validated.driver.configSchema as Record<string, unknown>
|
|
: null,
|
|
actor: input.actor,
|
|
});
|
|
}
|
|
|
|
if (input.driver === "plugin") {
|
|
const parsed = pluginEnvironmentConfigSchema.safeParse(parseObject(input.config));
|
|
if (!parsed.success) {
|
|
throw unprocessable(toErrorMessage(parsed.error), {
|
|
issues: parsed.error.issues,
|
|
});
|
|
}
|
|
if (!input.pluginWorkerManager) {
|
|
throw unprocessable("Plugin environment config validation requires a running plugin worker manager.");
|
|
}
|
|
return { ...(await validatePluginEnvironmentDriverConfig({
|
|
db: input.db,
|
|
workerManager: input.pluginWorkerManager,
|
|
config: parsed.data,
|
|
})) };
|
|
}
|
|
|
|
return normalizeEnvironmentConfig({
|
|
driver: input.driver,
|
|
config: input.config,
|
|
});
|
|
}
|
|
|
|
export async function resolveEnvironmentDriverConfigForRuntime(
|
|
db: Db,
|
|
companyId: string | null,
|
|
environment: Pick<Environment, "driver" | "config"> & Partial<Pick<Environment, "id">>,
|
|
context?: {
|
|
issueId?: string | null;
|
|
heartbeatRunId?: string | null;
|
|
// Force applying the active custom-image template even without a run/issue
|
|
// context. Operator-initiated `Test` probes have no issueId/heartbeatRunId
|
|
// but must still resolve the active custom image as prepared runtime
|
|
// configuration and tooling so the test reflects what real agent runs use.
|
|
applyCustomImageTemplate?: boolean;
|
|
},
|
|
): Promise<ParsedEnvironmentConfig> {
|
|
const parsed = parseEnvironmentDriverConfig(environment);
|
|
const secrets = secretService(db);
|
|
const environmentId = environment.id;
|
|
if (parsed.driver === "ssh" && parsed.config.privateKeySecretRef && !environmentId) {
|
|
throw unprocessable("Runtime secret resolution requires an environment id");
|
|
}
|
|
|
|
if (parsed.driver === "ssh" && parsed.config.privateKeySecretRef) {
|
|
if (!companyId) {
|
|
throw unprocessable("Runtime secret resolution requires a companyId context");
|
|
}
|
|
return {
|
|
driver: "ssh",
|
|
config: {
|
|
...parsed.config,
|
|
privateKey: await secrets.resolveSecretValue(
|
|
companyId,
|
|
parsed.config.privateKeySecretRef.secretId,
|
|
parsed.config.privateKeySecretRef.version ?? "latest",
|
|
{
|
|
consumerType: "environment",
|
|
consumerId: environmentId!,
|
|
actorType: "system",
|
|
actorId: null,
|
|
issueId: context?.issueId ?? null,
|
|
heartbeatRunId: context?.heartbeatRunId ?? null,
|
|
configPath: "privateKeySecretRef",
|
|
},
|
|
),
|
|
},
|
|
};
|
|
}
|
|
|
|
if (parsed.driver === "sandbox" && parsed.config.provider !== "fake") {
|
|
const schema = await getSandboxProviderConfigSchema(db, parsed.config.provider);
|
|
let runtimeConfig = parsed.config;
|
|
if (companyId) {
|
|
runtimeConfig = await resolveConfigSecretRefsForRuntime({
|
|
db,
|
|
companyId,
|
|
config: parsed.config as Record<string, unknown>,
|
|
schema,
|
|
context: {
|
|
consumerId: environmentId!,
|
|
issueId: context?.issueId ?? null,
|
|
heartbeatRunId: context?.heartbeatRunId ?? null,
|
|
},
|
|
}) as SandboxEnvironmentConfig;
|
|
} else {
|
|
for (const path of collectSecretRefPaths(schema)) {
|
|
const current = readConfigValueAtPath(parsed.config as Record<string, unknown>, path);
|
|
if (parseSecretRefBindingObject(current) || (typeof current === "string" && isUuidSecretRef(current.trim()))) {
|
|
throw unprocessable("Runtime secret resolution requires a companyId context");
|
|
}
|
|
}
|
|
}
|
|
return {
|
|
driver: "sandbox",
|
|
config: environmentId && (context?.issueId || context?.heartbeatRunId || context?.applyCustomImageTemplate)
|
|
? await resolveActiveEnvironmentCustomImageTemplateForRuntime(db, {
|
|
environmentId,
|
|
baseConfig: parsed.config,
|
|
runtimeConfig,
|
|
// Match the capture-time fingerprint exclusions: secret-ref paths
|
|
// are excluded when the template's source fingerprint is computed,
|
|
// so they must be excluded when re-checking it here.
|
|
secretRefExcludePaths: collectSecretRefPaths(schema),
|
|
})
|
|
: runtimeConfig,
|
|
};
|
|
}
|
|
|
|
return parsed;
|
|
}
|
|
|
|
/**
|
|
* Resolve the connection secrets of a recorded sandbox config for a durable
|
|
* orphan-sandbox teardown. The retry reads the recorded config from the durable
|
|
* `pending_cleanup` lease row, not from the current environment. So this
|
|
* resolver must not require the environment binding: a delete removed the
|
|
* environment, or a provider change replaced the binding. It resolves each
|
|
* schema-declared secret ref by id at the latest version through
|
|
* `resolveSecretValueForSandboxCleanup`, which authorizes the read from the
|
|
* durable orphan record and skips the binding check. The resolved values are
|
|
* used once for the teardown RPC and never persisted.
|
|
*/
|
|
export async function resolveSandboxCleanupConfigSecrets(
|
|
db: Db,
|
|
companyId: string,
|
|
config: SandboxEnvironmentConfig,
|
|
context?: { issueId?: string | null; heartbeatRunId?: string | null },
|
|
): Promise<SandboxEnvironmentConfig> {
|
|
if (config.provider === "fake") return config;
|
|
const schema = await getSandboxProviderConfigSchema(db, config.provider);
|
|
const secrets = secretService(db);
|
|
let nextConfig = { ...config } as Record<string, unknown>;
|
|
for (const path of collectSecretRefPaths(schema)) {
|
|
const current = canonicalizeSecretRefValue(readConfigValueAtPath(nextConfig, path), path);
|
|
if (typeof current !== "string") continue;
|
|
const trimmed = current.trim();
|
|
if (!isUuidSecretRef(trimmed)) continue;
|
|
nextConfig = writeConfigValueAtPath(
|
|
nextConfig,
|
|
path,
|
|
await secrets.resolveSecretValueForSandboxCleanup(companyId, trimmed, "latest", {
|
|
configPath: path,
|
|
issueId: context?.issueId ?? null,
|
|
heartbeatRunId: context?.heartbeatRunId ?? null,
|
|
}),
|
|
);
|
|
}
|
|
return nextConfig as SandboxEnvironmentConfig;
|
|
}
|
|
|
|
export function readSshEnvironmentPrivateKeySecretId(
|
|
environment: Pick<Environment, "driver" | "config">,
|
|
): string | null {
|
|
if (environment.driver !== "ssh") return null;
|
|
const parsed = sshEnvironmentConfigSchema.safeParse(parseObject(environment.config));
|
|
if (!parsed.success) return null;
|
|
return parsed.data.privateKeySecretRef?.secretId ?? null;
|
|
}
|
|
|
|
export function parseEnvironmentDriverConfig(
|
|
environment: Pick<Environment, "driver" | "config">,
|
|
): ParsedEnvironmentConfig {
|
|
if (environment.driver === "local") {
|
|
return {
|
|
driver: "local",
|
|
config: { ...parseObject(environment.config) },
|
|
};
|
|
}
|
|
|
|
if (environment.driver === "ssh") {
|
|
const parsed = sshEnvironmentConfigSchema.parse(parseObject(environment.config));
|
|
return {
|
|
driver: "ssh",
|
|
config: parsed,
|
|
};
|
|
}
|
|
|
|
if (environment.driver === "sandbox") {
|
|
const parsed = parseSandboxEnvironmentConfig(environment.config);
|
|
if (!parsed.success) {
|
|
throw parsed.error;
|
|
}
|
|
return {
|
|
driver: "sandbox",
|
|
config: parsed.data,
|
|
};
|
|
}
|
|
|
|
if (environment.driver === "plugin") {
|
|
const parsed = pluginEnvironmentConfigSchema.parse(parseObject(environment.config));
|
|
return {
|
|
driver: "plugin",
|
|
config: parsed,
|
|
};
|
|
}
|
|
|
|
throw new Error(`Unsupported environment driver "${environment.driver}".`);
|
|
}
|