diff --git a/doc/agent-observability.md b/doc/agent-observability.md index 2968ee6f507..c3b63085ba0 100644 --- a/doc/agent-observability.md +++ b/doc/agent-observability.md @@ -1,7 +1,7 @@ # Company agent observability Company Settings → General → Agent observability configures one company-owned -OTLP destination for Paperclip-managed Codex and Claude Code executions. Export +OTLP destination for Paperclip-managed Codex, Claude Code and native OpenCode executions. Export is disabled by default. Choose Braintrust (US) or a custom HTTP(S) OTLP base URL; use your own collector when you need fan-out. Braintrust EU and self-hosted installations can use Custom OTLP with their required routing/authentication @@ -16,11 +16,28 @@ This is separate from Paperclip's first-party Telemetry and its content-free The qualified versions are Codex 0.160.0 and Claude Code through ACP bridge 0.73.0 with Agent SDK 0.3.286. Both logs and traces use HTTP/protobuf. Prompts, assistant responses and tool content default on. Additional tool details are a -Claude Code control; Codex has no equivalent switch. Codex requests up to 64 KiB +Claude Code and native OpenCode control; Codex has no equivalent switch. Codex requests up to 64 KiB of tool output (or zero when disabled); native event availability, truncation and response caps still apply. Claude Code applies its own native limits. Metrics, raw API bodies and Codex Guardian assessments are disabled. +Native OpenCode 1.18.34 uses a runner-owned HTTP/JSON exporter. Each completed, +failed or cancelled turn produces a trace with Paperclip company, agent, run and +turn IDs. Tool outcomes and message-level token/cost receipts are child spans; +repeated usage updates replace the same message receipt. Missing prices remain +unknown. This projection excludes hidden reasoning, raw provider payloads and +system instructions. Prompt, response, tool-detail and tool-content switches +apply before export. Content is bounded to 8 KiB per field and 64 tools plus 64 +usage records per turn; truncation is marked. Tool inputs and full model request +bodies are not captured. OpenCode's performance-only OTLP sink stays separate. + +The exporter sends once at turn completion with a three-second deadline, at most +four requests in flight and no idle polling or persistent queue. Closing drains +pending requests and marks an unfinished turn failed. Export failures emit a +sanitized diagnostic and do not change task outcomes. Abrupt process termination +can lose an unexported turn. Recovered in-flight turns without a new submission +are not replayed just to produce an observability record. + The native runner and legacy local adapters receive transient launch material. Native Codex writes a private managed config.toml; legacy exec uses a unique invocation profile, preserving a selected source profile. Its profile injection @@ -37,7 +54,7 @@ them once to preserve literal percent sequences. Credentials remain literal values in the company secret; config-file environment interpolation is not used. A caller-owned/preconfigured Codex home cannot be automatically configured in this version: its files remain untouched and execution emits a diagnostic. -Other harnesses, personal Codex desktop sessions and hosted claude_managed are +Legacy OpenCode, other harnesses, personal Codex desktop sessions and hosted claude_managed are outside this feature. ## Changes and process lifetime @@ -53,8 +70,8 @@ never replays an uncertain turn merely to apply new export settings. Missing, disabled, invalid or unavailable configuration runs without company export and reports a sanitized availability diagnostic where relevant. Native export rejection/timeouts do not determine task success; exporter deadlines -are three seconds. Destination calls do not occur in usage, event-append, -tool-acknowledgment or settlement paths. +are three seconds. OpenCode starts its bounded asynchronous request when the terminal event is +observed; event delivery and task settlement do not wait for the receiver. ## Credentials and access @@ -72,8 +89,8 @@ row removes that header. API-key replacement is explicit, and Cancel clears entered secrets. Revisions prevent another administrator's edit being silently overwritten. Credential values are excluded from task environment manifests, run/config snapshots, argv, activity records, recovery configuration and query -cache data. Export credentials are necessarily accessible to the exporting -harness itself. Both Codex and Claude Code receive these headers through their +cache data. OpenCode keeps exporter credentials in the runner wrapper and excludes them +from its child environment and config. Codex and Claude Code receive headers through their process environment, which their tools can inherit or inspect. This feature does not isolate export credentials from agent code. Use a credential limited to telemetry ingestion for the selected destination; do not supply a broad diff --git a/maps/agent-access-map/companies.md b/maps/agent-access-map/companies.md index 9e5399a5e08..6e597c4eb10 100644 --- a/maps/agent-access-map/companies.md +++ b/maps/agent-access-map/companies.md @@ -215,6 +215,8 @@ Agent access: `operator-only` — configuration and tests require board context. Policy: Company access and Manage connections permission; non-managing board readers receive no settings. Credentials are write-only and stored through company secret bindings. +Native OpenCode's parent exports a bounded projection of normalized run events under these settings. +The destination headers stay out of the actual OpenCode child environment and configuration. Evidence: `source-only` — [routes](../../server/src/routes/agent-observability.ts). diff --git a/maps/feature-map/companies.md b/maps/feature-map/companies.md index 566a49319b0..c10d3ead5df 100644 --- a/maps/feature-map/companies.md +++ b/maps/feature-map/companies.md @@ -19,7 +19,7 @@ Agent access: [companion](../agent-access-map/companies.md). - `decision-models`: `companies/decision-models` — configure and test the company's decision model, then inspect invocation history and recorded usage. - `prompt-injection-prevention`: `companies/prompt-injection-prevention` — opt into a shared OpenRouter classifier for existing tool-result guards, test fixed samples, and recover unavailable selections. - `fast-responses`: `companies/fast-responses` — configure and test the experimental fast-response service, then inspect acknowledgement history, failures, and recorded usage. -- `agent-observability`: `companies/agent-observability` — configure one company OTLP destination for Codex and Claude Code, test synthetic delivery, and disable export. +- `agent-observability`: `companies/agent-observability` — configure one company OTLP destination for Codex, Claude Code and native OpenCode, test synthetic delivery, and disable export. - `archive`: `companies/archive` — archive and restore through the available controls without confusing archival with deletion. - `portability`: `companies/portability` — preview a company package and inspect imported agents, projects, skills, and conflicts. @@ -128,7 +128,7 @@ the test originates at the control plane and does not prove remote runner reacha Automated: shared validation, encrypted configuration service, receiver tests, runtime projection, and production settings stories cover their respective boundaries. Provider delivery requires a real run. -Manual: configure a disposable company, reload, test, run Codex and Claude Code locally and remotely, +Manual: configure a disposable company, reload, test, run Codex, Claude Code and native OpenCode locally and remotely, inspect their unique markers at the destination, then disable and verify a new marker is absent. Storybook verification: `agent-observability-01-settings`, `agent-observability-02-states`, diff --git a/packages/adapter-utils/src/agent-observability.test.ts b/packages/adapter-utils/src/agent-observability.test.ts index 22fb580541e..26d59bf1f91 100644 --- a/packages/adapter-utils/src/agent-observability.test.ts +++ b/packages/adapter-utils/src/agent-observability.test.ts @@ -113,3 +113,13 @@ it.each([true, false])("preserves refreshed Codex credentials across repeated an expect(JSON.parse(await readFile(join(prepared, "auth.json"), "utf8")).tokens.refresh_token).toBe("refreshed-fixture-token"); } }); + +it("keeps OpenCode export opt-in and credentials out of ordinary task environment projections", () => { + const launch = buildProviderObservabilityLaunch({ ...input, kind: "opencode" }); + expect(launch.environment).toMatchObject({ PAPERCLIP_OBSERVABILITY_OPENCODE: "1", OTEL_LOG_USER_PROMPTS: "1", OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: input.tracesEndpoint }); + expect(observabilitySecretValues(launch)).toContain(input.headers.authorization); + expect(JSON.stringify(buildInvocationEnvForLogs({ ...launch.environment }))).not.toContain("a%2Cb"); + const disabled = buildProviderObservabilityLaunch({ ...input, kind: "opencode", enabled: false }); + expect(disabled.environment).toMatchObject({ PAPERCLIP_OBSERVABILITY_OPENCODE: "0" }); + expect(disabled.environment).not.toHaveProperty("OTEL_EXPORTER_OTLP_TRACES_HEADERS"); +}); diff --git a/packages/adapter-utils/src/agent-observability.ts b/packages/adapter-utils/src/agent-observability.ts index 051e7cb2c78..3ce728ac175 100644 --- a/packages/adapter-utils/src/agent-observability.ts +++ b/packages/adapter-utils/src/agent-observability.ts @@ -2,7 +2,7 @@ import type { AgentObservabilityCapture } from "@paperclipai/shared"; /** Secret-bearing, controller-owned launch material. Never persist or merge into adapter config. */ export type ProviderObservabilityLaunch = { - kind: "codex" | "claude"; + kind: "codex" | "claude" | "opencode"; compatibilityKey: string; otelConfig?: Record; environment: Readonly>; @@ -17,7 +17,7 @@ export const AGENT_OBSERVABILITY_ENV_KEYS = [ "OTEL_EXPORTER_OTLP_LOGS_HEADERS", "OTEL_EXPORTER_OTLP_TRACES_HEADERS", "OTEL_EXPORTER_OTLP_LOGS_TIMEOUT", "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT", "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT", "OTEL_EXPORTER_OTLP_METRICS_HEADERS", "OTEL_EXPORTER_OTLP_METRICS_PROTOCOL", - "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG", + "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG", "PAPERCLIP_OBSERVABILITY_OPENCODE", "OTEL_LOG_USER_PROMPTS", "OTEL_LOG_ASSISTANT_RESPONSES", "OTEL_LOG_TOOL_DETAILS", "OTEL_LOG_TOOL_CONTENT", "OTEL_LOG_RAW_API_BODIES", ] as const; @@ -33,7 +33,7 @@ export function applyObservabilityEnvironment(env: NodeJS.ProcessEnv, launch: Pr export function observabilitySecretValues(launch: ProviderObservabilityLaunch | undefined): string[] { if (!launch) return []; const headers = ((launch.otelConfig?.exporter as Record | undefined)?.["otlp-http"] as { headers?: Record } | undefined)?.headers; - const encoded = launch.environment.OTEL_EXPORTER_OTLP_LOGS_HEADERS; + const encoded = [launch.environment.OTEL_EXPORTER_OTLP_LOGS_HEADERS, launch.environment.OTEL_EXPORTER_OTLP_TRACES_HEADERS].filter(Boolean).join(","); const decoded = (value: string) => { try { return decodeURIComponent(value); } catch { return value; } }; const values = [...Object.values(headers ?? {}).flatMap(value => [value, decoded(value)]), ...(encoded ? [encoded, ...encoded.split(",").flatMap(item => { const value = item.slice(item.indexOf("=") + 1); @@ -43,7 +43,7 @@ export function observabilitySecretValues(launch: ProviderObservabilityLaunch | } export function buildProviderObservabilityLaunch(input: { - kind: "codex" | "claude"; companyId: string; agentId: string; revision: string | null; + kind: "codex" | "claude" | "opencode"; companyId: string; agentId: string; revision: string | null; compatibilityKey: string; enabled: boolean; capture: AgentObservabilityCapture; logsEndpoint?: string; tracesEndpoint?: string; headers?: Readonly>; }): ProviderObservabilityLaunch { @@ -65,6 +65,20 @@ export function buildProviderObservabilityLaunch(input: { span_attributes: enabled ? attributes : {}, } }; } + if (kind === "opencode") { + // The runner owns this exporter. Never pass its credentials to OpenCode or tools. + environment.PAPERCLIP_OBSERVABILITY_OPENCODE = enabled ? "1" : "0"; + if (enabled) Object.assign(environment, { + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: input.tracesEndpoint, + OTEL_EXPORTER_OTLP_TRACES_HEADERS: encodeObservabilityHeaders(input.headers ?? {}), + OTEL_RESOURCE_ATTRIBUTES: Object.entries(attributes).map(([key, value]) => `${key}=${encodeURIComponent(value)}`).join(","), + OTEL_LOG_USER_PROMPTS: capture.userPrompts ? "1" : "0", + OTEL_LOG_ASSISTANT_RESPONSES: capture.assistantResponses ? "1" : "0", + OTEL_LOG_TOOL_DETAILS: capture.toolDetails ? "1" : "0", + OTEL_LOG_TOOL_CONTENT: capture.toolContent ? "1" : "0", + }); + return { kind, compatibilityKey, environment }; + } Object.assign(environment, { CLAUDE_CODE_ENABLE_TELEMETRY: enabled ? "1" : "0", CLAUDE_CODE_ENHANCED_TELEMETRY_BETA: enabled ? "1" : "0", diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs index 3ec890aa246..88a9c40b064 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs @@ -32,6 +32,14 @@ const WARM_ATTACHMENT_TAIL_DRAIN_LIMIT: usize = 256; const WARM_ATTACHMENT_QUIET_WINDOW: Duration = Duration::from_millis(10); const WARM_ATTACHMENT_DRAIN_DEADLINE: Duration = Duration::from_millis(100); const OPENCODE_PROVIDER_ENVIRONMENT_KEYS: &[&str] = &[ + "PAPERCLIP_OBSERVABILITY_OPENCODE", + "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", + "OTEL_EXPORTER_OTLP_TRACES_HEADERS", + "OTEL_RESOURCE_ATTRIBUTES", + "OTEL_LOG_USER_PROMPTS", + "OTEL_LOG_ASSISTANT_RESPONSES", + "OTEL_LOG_TOOL_DETAILS", + "OTEL_LOG_TOOL_CONTENT", "PAPERCLIP_AI_PROVIDER_KEY", "PAPERCLIP_AI_PROVIDER_URL", "PAPERCLIP_AGENT_KEY_ID", @@ -1745,7 +1753,7 @@ impl CodexProvider { cwd: &str, options: CodexTurnOptions<'_>, ) -> Result { - self.start_turn_with_options_and_effort(message, cwd, options, None) + self.start_turn_with_options_and_effort(message, cwd, options, None, None) } pub(crate) fn start_turn_with_options_and_effort( @@ -1754,6 +1762,7 @@ impl CodexProvider { cwd: &str, options: CodexTurnOptions<'_>, effort: Option<&Value>, + run_id: Option<&str>, ) -> Result { let CodexTurnOptions { skills, @@ -1825,6 +1834,13 @@ impl CodexProvider { let turn_params_object = turn_params .as_object_mut() .expect("Codex turn parameters are an object"); + // The retained proxy's launch environment belongs to its first run. + // Only runnerd's rotated authority supplies the current run identity. + if self.config.provider == "opencode" { + if let Some(run_id) = run_id { + turn_params_object.insert("paperclipRunId".to_owned(), json!(run_id)); + } + } if let Some(effort) = effort { turn_params_object.insert("effort".to_owned(), effort.clone()); } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/configured_environment.rs b/packages/paperclip-runner/runner/crates/runner-core/src/configured_environment.rs index aba51dbbcde..337afa929d5 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/configured_environment.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/configured_environment.rs @@ -5,6 +5,7 @@ use std::process::Command; const MARKER: &str = "PAPERCLIP_CONFIGURED_ENV_KEYS"; const RESERVED: &[&str] = &[ "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG", + "PAPERCLIP_OBSERVABILITY_OPENCODE", "CLAUDE_CODE_ENABLE_TELEMETRY", "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA", "ENABLE_ENHANCED_TELEMETRY_BETA", diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs index d898f581897..0587b9537dc 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs @@ -3025,6 +3025,10 @@ impl CodexCommandExecutor { state.ambiguous_turn_start_pending = true; } self.save_state()?; + let run_id = self + .event_identity + .as_ref() + .map(|identity| identity.run_id.clone()); let ( start_result, completion_authority_retained, @@ -3040,6 +3044,7 @@ impl CodexCommandExecutor { reasoning_mode, }, payload.get("effort"), + run_id.as_deref(), ); ( result, diff --git a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts index 7737ef2ee20..67963ec4995 100644 --- a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts +++ b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts @@ -53,6 +53,7 @@ let eventPump: Promise | null = null; let cwd = ""; let activeModel = ""; let activeTurnId: string | null = null; +let activeRunId = process.env.PAPERCLIP_RUN_ID ?? "runnerd-run"; const announcedTurnIds = new Set(); const launchBinding = trustedOpenCodeLaunchBinding(process.argv.slice(2)); const usageLedger = new OpenCodeProxyUsageLedger(); @@ -151,7 +152,7 @@ async function open( driverKind: "opencode_server", driverSessionId: threadId, providerSessionId: threadId, - runId: process.env.PAPERCLIP_RUN_ID ?? "runnerd-run", + runId: activeRunId, normalizedSessionId: process.env.PAPERCLIP_NORMALIZED_SESSION_ID ?? threadId, activeTurnId: null, @@ -163,7 +164,7 @@ async function open( session = recovered.session; } else { session = await driver.openSession({ - runId: process.env.PAPERCLIP_RUN_ID ?? "runnerd-run", + runId: activeRunId, normalizedSessionId: process.env.PAPERCLIP_NORMALIZED_SESSION_ID ?? `runnerd-${Date.now()}`, workingDirectory: cwd, @@ -365,6 +366,16 @@ async function handle(message: RpcMessage): Promise { case "turn/start": { if (!session) throw new Error("OpenCode thread is not open"); assertOpenCodeProxyCollaborationMode(params); + if (params.paperclipRunId !== undefined) { + const runId = params.paperclipRunId; + if (typeof runId !== "string" || !/^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/.test(runId)) + throw new Error("Invalid Paperclip run identity"); + if (runId !== activeRunId) { + if (!session.attachRun) throw new Error("OpenCode run attachment unavailable"); + await session.attachRun({ runId }); + activeRunId = runId; + } + } const inputItems = Array.isArray(params.input) ? params.input.map(record) : []; diff --git a/packages/paperclip-runner/src/cli/opencode-proxy-input.test.ts b/packages/paperclip-runner/src/cli/opencode-proxy-input.test.ts index d74eb069391..47170c7dbf8 100644 --- a/packages/paperclip-runner/src/cli/opencode-proxy-input.test.ts +++ b/packages/paperclip-runner/src/cli/opencode-proxy-input.test.ts @@ -82,8 +82,21 @@ it.each([true, false])("settles a real proxy interrupt with controller feedback await writeFile(proxy, execFileSync(process.execPath, ["--input-type=module", "-e", `import { bundleVerifiedProviderEntrypoints } from "./scripts/build-verified-provider-entrypoints.mjs"; const entries = await bundleVerifiedProviderEntrypoints({ write: false }); process.stdout.write(entries.find(({ entrypoint }) => entrypoint.name === "opencode-app-server-proxy").verifiedResult.outputFiles[0].contents);`], { maxBuffer: 16 * 1024 * 1024 })); const fd = process.platform === "linux" ? openSync(executable, "r") : undefined; const runtime = join(root, "runtime"); + const bodies: Array<{ resourceSpans: Array<{ scopeSpans: Array<{ spans: Array<{ attributes: Array<{ key: string; value: { stringValue?: string } }> }> }> }> }> = []; + const { createServer } = await import("node:http"); + const collector = createServer((request, response) => { + const chunks: Buffer[] = []; + request.on("data", chunk => chunks.push(chunk)); + request.on("end", () => { bodies.push(JSON.parse(Buffer.concat(chunks).toString("utf8"))); response.end("{}"); }); + }); + await new Promise(resolve => collector.listen(0, "127.0.0.1", resolve)); + const address = collector.address(); + if (!address || typeof address === "string") throw new Error("collector did not bind"); const child = spawn(process.execPath, [proxy, "--paperclip-trusted-opencode-executable", fd === undefined ? executable : "/proc/self/fd/3"], { - env: { ...process.env, PAPERCLIP_OPENCODE_RUNTIME_DIR: runtime, OPENROUTER_API_KEY: "fixture-key" }, + env: { ...process.env, PAPERCLIP_OPENCODE_RUNTIME_DIR: runtime, OPENROUTER_API_KEY: "fixture-key", + PAPERCLIP_RUN_ID: "stale-launch-run", PAPERCLIP_OBSERVABILITY_OPENCODE: "1", + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: `http://127.0.0.1:${address.port}/v1/traces`, + OTEL_RESOURCE_ATTRIBUTES: "paperclip.company.id=fixture-company" }, stdio: fd === undefined ? ["pipe", "pipe", "pipe"] : ["pipe", "pipe", "pipe", fd], detached: process.platform !== "win32", }); @@ -106,7 +119,7 @@ it.each([true, false])("settles a real proxy interrupt with controller feedback await response(1); send({ id: 2, method: "thread/start", params: { cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731", completionContract: { revision: "proxy-v1", criterionIds: ["objective"] }, conversationMode: "prepared" } }); await response(2); - send({ id: 3, method: "turn/start", params: { input: [{ type: "text", text: JSON.stringify({ schema: "paperclip.native-model-envelope.v3", task: { prompt: "invalid-tool-feedback completion-feedback" }, completionContract: { revision: "proxy-v1", criteria: [{ id: "objective", requirement: "Complete the fixture." }] } }) }] } }); + send({ id: 3, method: "turn/start", params: { paperclipRunId: "current-run-1", input: [{ type: "text", text: JSON.stringify({ schema: "paperclip.native-model-envelope.v3", task: { prompt: "invalid-tool-feedback completion-feedback" }, completionContract: { revision: "proxy-v1", criteria: [{ id: "objective", requirement: "Complete the fixture." }] } }) }] } }); await response(3); await expect.poll(() => messages.find(message => message.method === "item/tool/call"), { timeout: 5_000 }).toBeDefined(); const call = messages.find(message => message.method === "item/tool/call")!; @@ -132,11 +145,20 @@ it.each([true, false])("settles a real proxy interrupt with controller feedback await expect.poll(async () => JSON.parse(await readFile(join(runtime, sessions[0]!.name, "data/fake-completion-feedback.json"), "utf8"))).toMatchObject([ { result: { content: [{ text: expect.stringContaining(feedback) }] } }, ]); + await expect.poll(() => bodies.length, { timeout: 5_000 }).toBe(1); + send({ id: 5, method: "turn/start", params: { paperclipRunId: "current-run-2", input: [{ type: "text", text: JSON.stringify({ schema: "paperclip.native-model-envelope.v3", task: { prompt: "completion-feedback" }, completionContract: { revision: "proxy-v1", criteria: [{ id: "objective", requirement: "Complete the fixture." }] } }) }] } }); + await response(5); + await expect.poll(() => messages.find(message => message.method === "item/tool/call" && message.id !== call.id), { timeout: 5_000 }).toBeDefined(); + const warmCall = messages.find(message => message.method === "item/tool/call" && message.id !== call.id)!; + send({ id: warmCall.id, result: { success: true, contentItems: [{ type: "inputText", text: "Accepted." }] } }); + await expect.poll(() => bodies.length, { timeout: 5_000 }).toBe(2); + expect(bodies.map(body => body.resourceSpans[0]!.scopeSpans[0]!.spans[0]!.attributes + .find(attribute => attribute.key === "paperclip.run.id")?.value.stringValue)).toEqual(["current-run-1", "current-run-2"]); child.stdin!.end(); } await expect.poll(() => exited, { timeout: 5_000 }).toBe(true); await exit; - expect(messages.filter(message => message.method === "paperclip/runResult")).toHaveLength(respond ? 1 : 0); + expect(messages.filter(message => message.method === "paperclip/runResult")).toHaveLength(respond ? 2 : 0); } finally { child.kill("SIGTERM"); if (!exited) { @@ -145,6 +167,7 @@ it.each([true, false])("settles a real proxy interrupt with controller feedback await exit; } input.close(); + await new Promise(resolve => { collector.close(() => resolve()); collector.closeAllConnections(); }); await rm(root, { recursive: true, force: true }); } }, 30_000); diff --git a/packages/paperclip-runner/src/drivers/agent-observability.ts b/packages/paperclip-runner/src/drivers/agent-observability.ts index 7ee8159df14..dd1df2eccf8 100644 --- a/packages/paperclip-runner/src/drivers/agent-observability.ts +++ b/packages/paperclip-runner/src/drivers/agent-observability.ts @@ -5,7 +5,7 @@ import { randomUUID } from "node:crypto"; /** Controller-owned, secret-bearing options; excluded from all durable descriptors. */ export interface ProviderObservabilityLaunch { - kind: "codex" | "claude"; + kind: "codex" | "claude" | "opencode"; compatibilityKey: string; otelConfig?: Record; environment: Readonly>; @@ -20,7 +20,7 @@ export const AGENT_OBSERVABILITY_ENV_KEYS = [ "OTEL_EXPORTER_OTLP_LOGS_TIMEOUT", "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT", "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT", "OTEL_EXPORTER_OTLP_METRICS_HEADERS", "OTEL_EXPORTER_OTLP_METRICS_PROTOCOL", "OTEL_LOG_USER_PROMPTS", "OTEL_LOG_ASSISTANT_RESPONSES", "OTEL_LOG_TOOL_DETAILS", "OTEL_LOG_TOOL_CONTENT", "OTEL_LOG_RAW_API_BODIES", - "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG", + "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG", "PAPERCLIP_OBSERVABILITY_OPENCODE", ] as const; export function providerObservabilityEnvironment(source: NodeJS.ProcessEnv | undefined, agent: string): NodeJS.ProcessEnv { diff --git a/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts b/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts new file mode 100644 index 00000000000..e6729350f9f --- /dev/null +++ b/packages/paperclip-runner/src/drivers/opencode/company-observability.test.ts @@ -0,0 +1,119 @@ +import { expect, it, vi } from "vitest"; +import type { PrpEvent } from "../../protocol/replay-contract.js"; +import { createOpenCodeCompanyObservability } from "./company-observability.js"; + +const environment = { + PAPERCLIP_OBSERVABILITY_OPENCODE: "1", OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://collector.example/v1/traces", + OTEL_EXPORTER_OTLP_TRACES_HEADERS: "authorization=Bearer%20secret-canary", + OTEL_RESOURCE_ATTRIBUTES: "paperclip.company.id=company,paperclip.agent.id=agent,private.extra=must-not-export", + OTEL_LOG_USER_PROMPTS: "1", OTEL_LOG_ASSISTANT_RESPONSES: "1", OTEL_LOG_TOOL_DETAILS: "1", OTEL_LOG_TOOL_CONTENT: "1", +}; +const event = (eventType: PrpEvent["eventType"], payload: Record = {}, runId = "run-one", itemId?: string): PrpEvent => ({ + schema: "paperclip.prp.event.v1", sourceEventId: "event", sourceSeq: 1, sourceInstanceId: "runner", sourceKind: "runner", + runId, normalizedSessionId: "session", turnId: `turn-${runId}`, ...(itemId ? { itemId } : {}), eventType, schemaVersion: 1, + priority: 1, emittedAt: new Date().toISOString(), payload, +}); +const spanAttributes = (span: { attributes: { key: string; value: { stringValue?: string; doubleValue?: number; boolValue?: boolean } }[] }) => + Object.fromEntries(span.attributes.map(a => [a.key, a.value.stringValue ?? a.value.doubleValue ?? a.value.boolValue])); +function setup(env = environment, response = new Response("{}")) { + const bodies: any[] = []; + const transport = vi.fn(async (_url, init) => { bodies.push(JSON.parse(String(init?.body))); return response; }); + const diagnostic = vi.fn(); + const observer = createOpenCodeCompanyObservability({ environment: env, model: "openrouter/test-model", sensitiveValues: ["provider-secret"], fetch: transport, onDiagnostic: diagnostic }); + return { observer: observer!, bodies, transport, diagnostic }; +} +it("requires an explicit controller opt-in even when ambient OTLP is configured", () => { + expect(setup({ ...environment, PAPERCLIP_OBSERVABILITY_OPENCODE: "0" }).observer).toBeUndefined(); + expect(createOpenCodeCompanyObservability({ environment: undefined, model: "test", sensitiveValues: [] })).toBeUndefined(); +}); +it("exports turn and tool content without raw payloads, reasoning, credentials or duplicate usage", async () => { + const { observer, bodies, transport } = setup(); + observer.observe(event("turn.submitted", { text: "question secret-canary provider-secret", raw: "private" })); + observer.observe(event("item.completed", { kind: "reasoning", text: "HIDDEN REASONING" })); + observer.observe(event("tool.execution.started", { name: "read" }, "run-one", "tool-one")); + observer.observe(event("tool.execution.completed", { name: "read", output: "answer secret-canary", status: "failed", raw: "private" }, "run-one", "tool-one")); + observer.observe(event("item.completed", { kind: "usage", usageMessageId: "message", usage: { input: 10, output: 2, costUsd: 0.1 } })); + observer.observe(event("item.completed", { kind: "usage", usageMessageId: "message", usage: { input: 12, output: 3, cache: { read: 4 }, costUsd: 0.2 } })); + observer.observe(event("item.completed", { kind: "agentMessage", text: "final answer" })); + observer.observe(event("turn.completed")); + await observer.close(); + expect(transport).toHaveBeenCalledOnce(); + expect(transport.mock.calls[0]?.[1]).toMatchObject({ redirect: "error", headers: { authorization: "Bearer secret-canary" } }); + const wire = JSON.stringify(bodies); + for (const excluded of ["secret-canary", "provider-secret", "private", "HIDDEN REASONING"]) expect(wire).not.toContain(excluded); + const spans = bodies[0].resourceSpans[0].scopeSpans[0].spans; + expect(spans).toHaveLength(3); + expect(spanAttributes(spans[0])).toMatchObject({ "braintrust.input": "question [REDACTED] [REDACTED]", "braintrust.output": "final answer", "paperclip.run.id": "run-one", "paperclip.company.id": "company" }); + expect(spans[1].status.code).toBe(2); + expect(spanAttributes(spans[2])).toMatchObject({ "gen_ai.usage.input_tokens": 12, "gen_ai.usage.cache_read.input_tokens": 4, "paperclip.cost.usd": 0.2 }); +}); +it("honors all capture switches while retaining usage and the correct warm-run identity", async () => { + const { observer, bodies } = setup({ ...environment, OTEL_LOG_USER_PROMPTS: "0", OTEL_LOG_ASSISTANT_RESPONSES: "0", OTEL_LOG_TOOL_CONTENT: "0" }); + for (const runId of ["first", "second"]) { + observer.observe(event("turn.submitted", { text: "private-prompt" }, runId)); + observer.observe(event("tool.execution.completed", { name: "read", output: "private-tool" }, runId, "tool")); + observer.observe(event("item.completed", { kind: "agentMessage", text: "private-response" }, runId)); + observer.observe(event("item.completed", { kind: "usage", usageMessageId: "reused", usage: { input: 3, costUsd: null } }, runId)); + observer.observe(event("turn.completed", {}, runId)); + } + await observer.close(); + expect(bodies).toHaveLength(2); + expect(JSON.stringify(bodies)).not.toContain("private-"); + const spans = bodies.map(b => b.resourceSpans[0].scopeSpans[0].spans); + expect(spanAttributes(spans[0][0])["paperclip.run.id"]).toBe("first"); + expect(spanAttributes(spans[1][0])["paperclip.run.id"]).toBe("second"); + expect(spans[0][0].traceId).not.toBe(spans[1][0].traceId); + expect(spanAttributes(spans[1][2])).not.toHaveProperty("paperclip.cost.usd"); +}); +it.each([new Response('{"partialSuccess":{"rejectedSpans":"1"}}'), new Response("secret-canary", { status: 403 }), new Response("[]")])( + "reports export failure without exposing response bodies or disrupting close", async response => { + const { observer, diagnostic } = setup(environment, response); + observer.observe(event("turn.submitted")); observer.observe(event("turn.failed")); + await expect(observer.close()).resolves.toBeUndefined(); + expect(diagnostic).toHaveBeenCalledWith("Company observability export unavailable (export_failed)."); + }, +); +it("bounds rows and content and marks an interrupted active turn on close", async () => { + const { observer, bodies } = setup(); + observer.observe(event("turn.submitted", { text: "x".repeat(100000) })); + for (let i = 0; i < 100; i++) observer.observe(event("tool.execution.completed", { name: "read", output: "x".repeat(100000) }, "run-one", String(i))); + await observer.close(); + const spans = bodies[0].resourceSpans[0].scopeSpans[0].spans; + expect(spans).toHaveLength(65); + expect(spans[0].status.code).toBe(2); + expect(spanAttributes(spans[0])["paperclip.observability.truncated"]).toBe(true); + expect(Buffer.byteLength(JSON.stringify(bodies[0]))).toBeLessThan(1048576); +}); +it.each(["prompt", "answer", "tool", "upstream-tool"])("marks a shortened %s even below the span limit", async kind => { + const { observer, bodies } = setup(); + const longText = "😀".repeat(3000); + observer.observe(event("turn.submitted", { text: kind === "prompt" ? longText : "short" })); + if (kind === "answer") observer.observe(event("item.completed", { kind: "agentMessage", text: longText })); + if (kind === "tool" || kind === "upstream-tool") observer.observe(event("tool.execution.completed", { + name: "read", output: kind === "tool" ? longText : "excerpt", outputTruncated: kind === "upstream-tool", + }, "run-one", "tool")); + observer.observe(event("turn.completed")); + await observer.close(); + const spans = bodies[0].resourceSpans[0].scopeSpans[0].spans; + expect(spanAttributes(spans[0])["paperclip.observability.truncated"]).toBe(true); + for (const span of spans) for (const [key, value] of Object.entries(spanAttributes(span))) { + if (key.startsWith("braintrust.") && typeof value === "string") { + expect(Buffer.byteLength(value)).toBeLessThanOrEqual(8192); + expect(value).not.toContain("\uFFFD"); + } + } +}); +it("does not block event delivery when a receiver stalls and cancels the request at its deadline", async () => { + const diagnostics: string[] = []; + let aborted = false; + const transport: typeof fetch = async (_url, init) => new Promise((_resolve, reject) => { + init?.signal?.addEventListener("abort", () => { aborted = true; reject(new Error("private receiver failure")); }, { once: true }); + }); + const observer = createOpenCodeCompanyObservability({ environment, model: "test", sensitiveValues: [], fetch: transport, onDiagnostic: text => diagnostics.push(text) })!; + const start = performance.now(); + observer.observe(event("turn.submitted")); observer.observe(event("turn.completed")); + expect(performance.now() - start).toBeLessThan(100); + await observer.close(); + expect(aborted).toBe(true); + expect(diagnostics).toEqual(["Company observability export unavailable (export_failed)."]); +}, 5000); diff --git a/packages/paperclip-runner/src/drivers/opencode/company-observability.ts b/packages/paperclip-runner/src/drivers/opencode/company-observability.ts new file mode 100644 index 00000000000..4ea7cda93c6 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/opencode/company-observability.ts @@ -0,0 +1,163 @@ +import { randomBytes } from "node:crypto"; +import type { PrpEvent } from "../../protocol/replay-contract.js"; +import { redactObservabilityDiagnostic } from "../agent-observability.js"; + +type Attributes = Record; +interface Span { id: string; name: string; start: number; end: number; failed: boolean; attributes: Attributes } +interface Turn { + runId: string; turnId: string; traceId: string; root: Span; + tools: Map; usage: Map; truncated: boolean; +} +const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +const finite = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value) && value >= 0; +const id = () => randomBytes(8).toString("hex"); +const maxRows = 64; +const maxTextBytes = 8192; +const attributes = (values: Attributes) => Object.entries(values).map(([key, value]) => ({ key, value: + typeof value === "string" ? { stringValue: value } : typeof value === "boolean" ? { boolValue: value } : { doubleValue: value } })); +function pairs(value: string | undefined): Record { + return Object.fromEntries((value ?? "").split(",").filter(Boolean).map(entry => { + const index = entry.indexOf("="); + if (index < 1) throw new Error("invalid_configuration"); + return [entry.slice(0, index), decodeURIComponent(entry.slice(index + 1))]; + })); +} + +/** A controller opt-in, not OpenCode's raw OTLP. Projected PRP events never include reasoning or raw provider payloads. */ +export function createOpenCodeCompanyObservability(input: { + environment: NodeJS.ProcessEnv | undefined; model: string; sensitiveValues: readonly string[]; + fetch?: typeof globalThis.fetch; onDiagnostic?: (message: string) => void; +}) { + const env = input.environment; + if (env?.PAPERCLIP_OBSERVABILITY_OPENCODE !== "1") return undefined; + const diagnostic = (reason: string) => input.onDiagnostic?.(`Company observability export unavailable (${reason}).`); + let endpoint: string, headers: Record, resource: Attributes; + try { + const url = new URL(env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT ?? ""); + if (!["http:", "https:"].includes(url.protocol) || url.username || url.password || url.hash) throw new Error(); + endpoint = url.href; + headers = pairs(env.OTEL_EXPORTER_OTLP_TRACES_HEADERS); + const supplied = pairs(env.OTEL_RESOURCE_ATTRIBUTES); + resource = { "service.name": "paperclip-opencode", ...Object.fromEntries( + ["paperclip.company.id", "paperclip.agent.id", "paperclip.observability.revision"] + .filter(key => supplied[key]).map(key => [key, supplied[key]])) }; + } catch { diagnostic("configuration_invalid"); return undefined; } + const capture = (key: string) => env[key] === "1"; + const clean = (value: unknown, limit = maxTextBytes): string => { + if (typeof value !== "string") return ""; + let text = redactObservabilityDiagnostic(value, env); + for (const secret of input.sensitiveValues.filter(Boolean).sort((a, b) => b.length - a.length)) text = text.replaceAll(secret, "[REDACTED]"); + const bytes = Buffer.from(text); + if (bytes.length > limit && current) current.truncated = true; + let end = Math.min(bytes.length, limit); + while (end > 0 && end < bytes.length && (bytes[end]! & 0xc0) === 0x80) end--; + return bytes.subarray(0, end).toString("utf8"); + }; + const pending = new Set>(); + let current: Turn | undefined; + let closed = false; + const makeSpan = (name: string, time: number, values: Attributes): Span => ({ id: id(), name, start: time, end: time, failed: false, attributes: values }); + const finish = (failed: boolean, time: number) => { + const turn = current; + current = undefined; + if (!turn) return; + turn.root.end = Math.max(turn.root.start, time); + turn.root.failed = failed; + turn.root.attributes["paperclip.observability.truncated"] = turn.truncated; + const spans = [turn.root, ...turn.tools.values(), ...turn.usage.values()].map(span => ({ + traceId: turn.traceId, spanId: span.id, ...(span !== turn.root ? { parentSpanId: turn.root.id } : {}), + name: span.name, kind: 0, startTimeUnixNano: String(BigInt(Math.trunc(span.start)) * 1_000_000n), + endTimeUnixNano: String(BigInt(Math.trunc(Math.max(span.start, span.end))) * 1_000_000n), + status: { code: span.failed ? 2 : 1 }, attributes: attributes({ ...resource, ...span.attributes, + "paperclip.run.id": turn.runId, "paperclip.turn.id": turn.turnId }), + })); + // One bounded request per terminal turn, no idle worker, retries or durable secret-bearing queue. + if (pending.size >= 4) { diagnostic("export_busy"); return; } + const body = JSON.stringify({ resourceSpans: [{ resource: { attributes: attributes(resource) }, + scopeSpans: [{ scope: { name: "paperclip.opencode", version: "1" }, spans }] }] }); + if (Buffer.byteLength(body) > 1_048_576) { diagnostic("export_too_large"); return; } + const job = (async () => { + try { + const response = await (input.fetch ?? globalThis.fetch)(endpoint, { + method: "POST", redirect: "error", signal: AbortSignal.timeout(3000), + headers: { ...headers, "content-type": "application/json", accept: "application/json" }, body, + }); + const reader = response.body?.getReader(); + try { + if (!response.ok) throw new Error("receiver_rejected"); + const chunks: Uint8Array[] = []; let size = 0; + while (reader) { + const next = await reader.read(); if (next.done) break; + size += next.value.length; if (size > 65536) throw new Error("invalid_response"); + chunks.push(next.value); + } + if (size) { + const decoded: unknown = JSON.parse(Buffer.concat(chunks).toString("utf8")); + if (!decoded || typeof decoded !== "object" || Array.isArray(decoded)) throw new Error("invalid_response"); + const partial = record(decoded).partialSuccess; + if (partial !== undefined && (!partial || typeof partial !== "object" || Array.isArray(partial))) throw new Error("invalid_response"); + const rejected = record(partial).rejectedSpans ?? 0; + if (!/^\d+$/.test(String(rejected)) || Number(rejected) !== 0) throw new Error("partial_rejection"); + } + } finally { await reader?.cancel().catch(() => {}); } + } catch { diagnostic("export_failed"); } + })(); + pending.add(job); + void job.finally(() => pending.delete(job)); + }; + return { + observe(event: PrpEvent) { + if (closed) return; + const time = Date.parse(event.emittedAt); + if (!Number.isFinite(time)) return; + if (event.eventType === "turn.submitted" && event.turnId) { + if (current) finish(true, time); + current = { runId: event.runId, turnId: event.turnId, traceId: randomBytes(16).toString("hex"), + root: makeSpan("paperclip.opencode.turn", time, { "gen_ai.operation.name": "invoke_agent", "gen_ai.request.model": input.model }), + tools: new Map(), usage: new Map(), truncated: false }; + if (capture("OTEL_LOG_USER_PROMPTS")) current.root.attributes["braintrust.input"] = clean(record(event.payload).text); + } + const turn = current; + if (!turn || event.runId !== turn.runId || event.turnId !== turn.turnId) return; + const payload = record(event.payload); + if (event.eventType === "item.completed" && payload.kind === "agentMessage" && capture("OTEL_LOG_ASSISTANT_RESPONSES")) { + turn.root.attributes["braintrust.output"] = clean(payload.text); + } + if (event.eventType.startsWith("tool.execution.") && event.itemId && capture("OTEL_LOG_TOOL_DETAILS")) { + let span = turn.tools.get(event.itemId); + if (!span && turn.tools.size < maxRows) { + const name = clean(payload.name, 128); + span = makeSpan(`tool ${name || "execution"}`, time, { + "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name }); + turn.tools.set(event.itemId, span); + } + if (span) { + span.end = time; span.failed = payload.status === "failed"; + if (capture("OTEL_LOG_TOOL_CONTENT")) { + span.attributes["braintrust.output"] = clean(payload.output); + if (payload.outputTruncated === true) turn.truncated = true; + } + } else turn.truncated = true; + } + if (event.eventType === "item.completed" && payload.kind === "usage" && typeof payload.usageMessageId === "string") { + const key = payload.usageMessageId; + if (turn.usage.has(key) || turn.usage.size < maxRows) { + const usage = record(payload.usage), cache = record(usage.cache); + const values: Attributes = { "gen_ai.operation.name": "chat", "gen_ai.request.model": input.model }; + for (const [name, value] of Object.entries({ "gen_ai.usage.input_tokens": usage.input, "gen_ai.usage.output_tokens": usage.output, + "gen_ai.usage.cache_read.input_tokens": cache.read, "gen_ai.usage.cache_creation.input_tokens": cache.write, + "paperclip.cost.usd": usage.costUsd })) if (finite(value)) values[name] = value; + const previous = turn.usage.get(key); + turn.usage.set(key, { ...makeSpan("opencode.model.usage", previous?.start ?? time, values), id: previous?.id ?? id(), end: time }); + } else turn.truncated = true; + } + if (["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(event.eventType)) finish(event.eventType !== "turn.completed", time); + }, + async close() { + if (closed) return; + closed = true; + finish(true, Date.now()); + await Promise.all(pending); + }, + }; +} diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index 5f171927977..6862bccfc39 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -153,6 +153,46 @@ afterAll(async () => { }); describe("OpenCodeServerDriver", () => { + it("exports company traces across warm runs without giving the ingestion key to the provider", async () => { + await chmod(fixture, 0o755); + const bodies: Array> = []; + const collector = createHttpServer((request, response) => { + const chunks: Buffer[] = []; + request.on("data", chunk => chunks.push(chunk)); + request.on("end", () => { bodies.push(JSON.parse(Buffer.concat(chunks).toString("utf8"))); response.end("{}"); }); + }); + await new Promise(resolve => collector.listen(0, "127.0.0.1", resolve)); + const address = collector.address(); + if (!address || typeof address === "string") throw new Error("collector did not bind"); + const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-company-export-")); roots.push(root); + const driver = new OpenCodeServerDriver({ model: "openrouter/test-model", command: fixture, runtimeDirectory: root, + environment: { PATH: process.env.PATH, PAPERCLIP_OBSERVABILITY_OPENCODE: "1", + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: `http://127.0.0.1:${address.port}/v1/traces`, + OTEL_EXPORTER_OTLP_TRACES_HEADERS: "authorization=Bearer%20ingestion-canary", + OTEL_RESOURCE_ATTRIBUTES: "paperclip.company.id=fixture-company", OTEL_LOG_USER_PROMPTS: "1", OTEL_LOG_ASSISTANT_RESPONSES: "1" } }); + let session: Awaited> | undefined; + try { + session = await driver.openSession({ runId: "first", normalizedSessionId: "export", workingDirectory: root }); + const child = await readFile(join(root, "export", "data", "fake-environment.json"), "utf8"); + expect(child).not.toContain("OTEL_EXPORTER_OTLP_TRACES_HEADERS"); + expect(child).not.toContain("ingestion-canary"); + for (const runId of ["first", "second"]) { + if (runId === "second") session.attachRun!({ runId }); + await session.startTurn({ message: { role: "user", text: "finish" } }); + const events = await collectTurnEvents(session.events()); + expect(events.at(-1)).toMatchObject({ runId, eventType: "turn.completed" }); + } + await session.close({ reason: "test" }); session = undefined; + expect(bodies).toHaveLength(2); + const roots = bodies.map(body => body.resourceSpans[0].scopeSpans[0].spans[0]); + expect(roots.map(span => span.attributes.find((a: any) => a.key === "paperclip.run.id").value.stringValue)).toEqual(["first", "second"]); + expect(JSON.stringify(bodies)).not.toContain("ingestion-canary"); + } finally { + await session?.close({ reason: "test" }); + await new Promise(resolve => { collector.close(() => resolve()); collector.closeAllConnections(); }); + } + }); + it("advertises within-turn plans as unsupported", async () => { const driver = new OpenCodeServerDriver({ model: "openrouter/deepseek/deepseek-v4-flash-0731", diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index 825812a0d09..2694d75fdf9 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -24,6 +24,7 @@ import { pipeline } from "node:stream/promises"; import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { parseOpenCodeReasoningMode, type OpenCodeReasoningMode } from "./reasoning-mode.js"; +import { createOpenCodeCompanyObservability } from "./company-observability.js"; import { startOpenCodePerformanceDiagnostics } from "./performance-diagnostics.js"; import { PROVIDER_PERFORMANCE_ENV } from "../../contracts/provider-performance.js"; import { annotateOmittedProviderText, parseOpenCodeSseFrames } from "./sse-frames.js"; @@ -443,6 +444,8 @@ export class OpenCodeServerDriver implements HarnessDriver { completionFeedback: this.#options.completionFeedback, snapshot, now: this.#options.now ?? (() => new Date()), + observabilityEnvironment: this.#options.environment, + onDiagnostic: this.#options.onDiagnostic, }); session.startEventPump(); return session; @@ -534,6 +537,7 @@ class OpenCodeHarnessSession implements HarnessSession { #usage: Record | null = null; readonly #conversationMode: "task" | "prepared"; #sendFullContext: boolean; + readonly #companyObservability: ReturnType; #closed = false; #completionSettlement: Promise | null = null; #abort = new AbortController(); @@ -554,8 +558,14 @@ class OpenCodeHarnessSession implements HarnessSession { completionFeedback?: OpenCodeCompletionFeedback; snapshot: PersistedHarnessSession | null; now: () => Date; + observabilityEnvironment?: NodeJS.ProcessEnv; + onDiagnostic?: (message: string) => void; }) { this.#runtime = input.runtime; + this.#companyObservability = createOpenCodeCompanyObservability({ + environment: input.observabilityEnvironment, model: input.model, sensitiveValues: input.runtime.sensitiveValues, + onDiagnostic: input.onDiagnostic ?? ((message) => console.warn(message)), + }); this.#fetch = input.fetcher; this.#runId = input.runId; this.#sourceSequence = 0; @@ -679,7 +689,7 @@ class OpenCodeHarnessSession implements HarnessSession { this.#emit("turn.submitted", { envelopeSchema: this.#taskEnvelope.schema, text: input.message.text, - }); + }, { turnId }); this.#emit("turn.accepted", { turnId }, { turnId }); this.#emit("turn.started", { status: "inProgress" }, { turnId }); const [providerID, ...modelParts] = this.#model.split("/"); @@ -991,7 +1001,7 @@ class OpenCodeHarnessSession implements HarnessSession { for (const requestId of [...this.#pendingRuntimeRequests.keys()]) this.#settlePendingRuntimeRequest(requestId); this.#events.close(); - await this.#runtime.close(); + await Promise.all([this.#runtime.close(), this.#companyObservability?.close()]); } // A request can outlive its own turn: the turn can fail or get cancelled @@ -2162,6 +2172,7 @@ class OpenCodeHarnessSession implements HarnessSession { if (this.#activeTraceFrameId !== null) { this.#activeTraceEmittedEventIds.push(event.sourceEventId); } + this.#companyObservability?.observe(event); this.#transcript.push(structuredClone(event)); this.#events.push(event); } diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index c8fea0a9b05..ba2a3195557 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -64435,7 +64435,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }, ); - it.each(["contained", "prior_denied_intent", "observability_warning", "observability_reason", "observability_revision", "observability_provenance", "observability_extra", "other_reason", "wrong_issue", "provider_started", "result_extra", "profile", "usage", "output", + it.each(["contained", "prior_denied_intent", "observability_opencode", "observability_warning", "observability_reason", "observability_revision", "observability_provenance", "observability_extra", "other_reason", "wrong_issue", "provider_started", "result_extra", "profile", "usage", "output", "process", "controller", "active_control", "lease", "accepted_result", "event_source", "extra_event", "source_edited", "principal_revoked"] as const)( "retries an authenticated GitHub request after isolated-directory validation failed before allocation: %s", async mode => { const f = await failedChatRetryFixture("github", "42", true, false, true); @@ -64455,7 +64455,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await db.insert(companyObservability).values({ companyId, revision: randomUUID(), enabled: true, destination: { provider: "otlp", endpointBaseUrl: "https://collector.example/otel" }, capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } }); - const [agent] = await db.update(agents).set({ adapterConfig: { provider: "opencode", model: "openrouter/synthetic", homeMode: "preconfigured" } }) + const [agent] = await db.update(agents).set({ adapterConfig: mode === "observability_opencode" + ? { provider: "opencode", model: "openrouter/synthetic", homeMode: "preconfigured" } + : { provider: "acpx", acpxAgent: "pi", model: "synthetic", homeMode: "preconfigured" } }) .where(eq(agents.id, agentId)).returning(); const [run] = await readHeartbeatRunsById(f.runId); const prepared = await prepareHeartbeatExecutionConfiguration(db, { run, agent, @@ -64473,8 +64475,17 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { onFailureSecretsResolved: () => {}, onManagedAiRuntimeAcquired: () => {}, onIdentityRedactorPrepared: () => {}, }); expect(prepared.prepared).toBe(true); - expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId))) - .toMatchObject([{ eventType: "agent_observability.unavailable", payload: { reason: "unsupported_harness" } }]); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId)); + if (mode === "observability_opencode") { + if (!prepared.prepared) throw new Error("Configuration preparation did not finish"); + expect(prepared.observability?.kind).toBe("opencode"); + expect(prepared.observability?.environment).toMatchObject({ PAPERCLIP_OBSERVABILITY_OPENCODE: "1", + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "https://collector.example/otel/v1/traces" }); + expect(events.filter(event => event.eventType === "agent_observability.unavailable")).toHaveLength(0); + expect(JSON.stringify(prepared.runtimeConfig)).not.toContain("collector.example"); + } else { + expect(events).toMatchObject([{ eventType: "agent_observability.unavailable", payload: { reason: "unsupported_harness" } }]); + } } await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId, eventType: "error", stream: "system", level: "error", message: diagnostic }); @@ -64552,7 +64563,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed after Retry acceptance" }).where(eq(issueComments.id, String(f.action.payload.commentId))); await f.service.processFailedChatRunRetry(response.body.chatActionId); } - if (!["contained", "prior_denied_intent", "observability_warning"].includes(mode)) { + if (!["contained", "prior_denied_intent", "observability_opencode", "observability_warning"].includes(mode)) { expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0); return; } diff --git a/server/src/services/agent-observability-launch.ts b/server/src/services/agent-observability-launch.ts index 739abbb7892..000cbdd23dc 100644 --- a/server/src/services/agent-observability-launch.ts +++ b/server/src/services/agent-observability-launch.ts @@ -2,7 +2,7 @@ import { DEFAULT_AGENT_OBSERVABILITY_CAPTURE } from "@paperclipai/shared"; import { buildProviderObservabilityLaunch } from "@paperclipai/adapter-utils/agent-observability"; import type { ResolvedAgentObservability } from "./agent-observability.js"; -export function agentObservabilityLaunch(snapshot: ResolvedAgentObservability, input: { kind: "codex" | "claude"; companyId: string; agentId: string }) { +export function agentObservabilityLaunch(snapshot: ResolvedAgentObservability, input: { kind: "codex" | "claude" | "opencode"; companyId: string; agentId: string }) { return buildProviderObservabilityLaunch({ ...input, revision: snapshot.revision, compatibilityKey: snapshot.compatibilityKey, enabled: snapshot.state === "enabled", capture: snapshot.state === "enabled" ? snapshot.capture : DEFAULT_AGENT_OBSERVABILITY_CAPTURE, ...(snapshot.state === "enabled" ? { logsEndpoint: snapshot.logsEndpoint, tracesEndpoint: snapshot.tracesEndpoint, headers: snapshot.headers } : {}) }); diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts index ec531455fed..0e10cf59b5a 100644 --- a/server/src/services/heartbeat/execution-configuration.ts +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -96,13 +96,16 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear : input.config.sourceWorkspaceConfig; const preconfiguredHome = agent.adapterType === "paperclip_runner" && sourceWorkspaceConfig.homeMode === "preconfigured"; const initialHarness = agentHarnessType(agent.adapterType, sourceWorkspaceConfig); - const unsupportedObservability = !["codex_local", "claude_local"].includes(initialHarness) + const observabilityKind = (harness: string) => harness === "codex_local" ? "codex" as const + : harness === "claude_local" ? "claude" as const + : harness === "opencode_local" && agent.adapterType === "paperclip_runner" ? "opencode" as const : undefined; + const unsupportedObservability = !observabilityKind(initialHarness) ? "unsupported_harness" as const : preconfiguredHome && initialHarness === "codex_local" ? "unsupported_home_mode" as const : undefined; const observabilitySnapshot: ResolvedAgentObservability = await agentObservabilityService(db).resolve(agent.companyId, { runId: run.id, agentId: agent.id, responsibleUserId, issueId, unavailableReason: unsupportedObservability }) .catch(() => unavailableAgentObservability(agent.companyId, "configuration_invalid")); const identityRedactor = createAgentIdentityRedactor(); - const earlyObservability = unsupportedObservability === undefined && (initialHarness === "codex_local" || initialHarness === "claude_local") - ? agentObservabilityLaunch(observabilitySnapshot, { kind: initialHarness === "codex_local" ? "codex" : "claude", companyId: agent.companyId, agentId: agent.id }) : undefined; + const earlyObservability = unsupportedObservability === undefined && observabilityKind(initialHarness) + ? agentObservabilityLaunch(observabilitySnapshot, { kind: observabilityKind(initialHarness)!, companyId: agent.companyId, agentId: agent.id }) : undefined; identityRedactor.values.push(...observabilitySecretValues(earlyObservability)); onIdentityRedactorPrepared(identityRedactor); // The executor keeps these values in its failure redactor before any later @@ -335,8 +338,8 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear : undefined; identityRedactor.values.push(...createAgentIdentityRedactor(agentIdentity?.privateKeyPem).values); const harness = agentHarnessType(agent.adapterType, runtimeConfig); - const observability = unsupportedObservability !== "unsupported_home_mode" && (harness === "codex_local" || harness === "claude_local") - ? agentObservabilityLaunch(observabilitySnapshot, { kind: harness === "codex_local" ? "codex" : "claude", companyId: agent.companyId, agentId: agent.id }) : undefined; + const observability = unsupportedObservability !== "unsupported_home_mode" && observabilityKind(harness) + ? agentObservabilityLaunch(observabilitySnapshot, { kind: observabilityKind(harness)!, companyId: agent.companyId, agentId: agent.id }) : undefined; identityRedactor.values.push(...observabilitySecretValues(observability)); identityRedactor.values.sort((a, b) => b.length - a.length); if (observabilitySnapshot.state === "unavailable") { diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index 767e2f247a9..24393e099ee 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -23,6 +23,8 @@ never start a server or depend on a tenant filesystem. | Journey | Required starting state | Observed outcome | | --- | --- | --- | +| `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work | +| `ask-question` | Company, configured agent capability, positive model allowance and exactly one permitted work run | Reuse the existing cross-model Ask question prompt; require one successful run with matching company/task/agent/model/environment, one saved exact answer, and visible readback after reload | | `task-properties` | Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run | | `cloud-onboarding` | Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact | @@ -77,6 +79,24 @@ server proves that no provider work started. A scheduled replacement remains pending and consumes the run limit. It cannot satisfy completion; the child must still finish through native execution and pass the independent artifact checks. +The attached `ask-question` recipe requires a prepared native general agent and +an explicit execution environment. Legacy execution is unsupported until its +saved model/provider identity can be verified. A registered +agent avoids a new setup probe; a disposable QA agent uses the existing native +fixture defaults and its setup remains separately unpriced. A restricted review +bot is not a general-work fixture. +The private provider supplies its exact execution identity through the `agent` +capability and collects normalized run usage. The existing Cloud provider still +owns authentication, conditional leases, fixture ownership, and cleanup. +A model allowance is an accounting threshold; reused direct-provider routes do +not acquire the onboarding gateway's request-level cash guarantee. The question +observer allows one run and waits at most four minutes. Cleanup stops only +run-owned question work; shared agent configuration stays outside ownership. +When the QA adapter explicitly selects terminal-history retention, it terminates +its disposable agent, makes owned tasks terminal, verifies no active runs, retains +the records for decision-history integrity, and releases both QA leases. This is +reported as retained data, rather than confirmed deletion. + ## Evidence and lifecycle Results use the existing campaign/result schemas and dashboard. Each result @@ -115,7 +135,8 @@ contract before successful cleanup. Missing assertions or paid accounting proof turn the campaign into a failure while preserving valid cost receipts, so an invalid success cannot trigger disposable-stack deletion. -Only post-journey screenshots are captured. Browser state, login codes, raw +Only post-journey screenshots are captured. Failed task/search screens are +retained only after tenant setup; authentication pages are excluded. Browser state, login codes, raw errors, credentials, request headers, and auth screenshots are excluded. All Cloud evidence is private; the public history publisher rejects these bundles. The private provider uses conditional S3 identity/stack leases and intent/receipt diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 009445d2723..d50c16df3c4 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -59,7 +59,7 @@ describe("runner E2E catalog", () => { }); it("exports deployment journeys separately from the model matrix and default selection", () => { - expect(productJourneys.map(journey => journey.id)).toEqual(["task-properties", "cloud-onboarding"]); + expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "ask-question", "task-properties", "cloud-onboarding"]); expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false); expect(runnerMatrix.some(execution => execution.suite.id === "product-journeys")).toBe(false); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "product-journeys")).toBe(false); diff --git a/tests/runner-e2e/journey-features.ts b/tests/runner-e2e/journey-features.ts index 30c489ad928..de90df3e026 100644 --- a/tests/runner-e2e/journey-features.ts +++ b/tests/runner-e2e/journey-features.ts @@ -18,7 +18,8 @@ function associate(suites: readonly string[], cases: Record { + it("accepts the bound saved answer and rejects plausible false passes", () => { + expect(questionEvidencePasses(evidence())).toBe(true); + const changes = [ + (e: ReturnType) => { e.issue.companyId = "other"; }, + (e: ReturnType) => { e.issue.workMode = "standard"; }, + (e: ReturnType) => { e.runs[0]!.issueId = "other"; }, + (e: ReturnType) => { e.runs[0]!.status = "failed"; }, + (e: ReturnType) => { e.runs.push(e.runs[0]!); }, + (e: ReturnType) => { e.runs[0]!.contextSnapshot.paperclipEnvironment.id = "local"; }, + (e: ReturnType) => { e.runs[0]!.runnerProfileJson.nativeExecutionInput.provider.model = "other"; }, + (e: ReturnType) => { e.runtimeMode = "legacy"; e.runs[0]!.runtimeMode = "legacy"; }, + (e: ReturnType) => { e.environmentId = ""; }, + (e: ReturnType) => { e.runs[0]!.runnerProfileJson.nativeExecutionInput.provider.kind = "other"; }, + (e: ReturnType) => { e.comments[0]!.authorAgentId = "other"; }, + (e: ReturnType) => { e.comments[0]!.body = "wrong answer"; }, + (e: ReturnType) => { e.comments = []; }, + ]; + for (const change of changes) { const e = evidence(); change(e); expect(questionEvidencePasses(e)).toBe(false); } + }); + it("requires an explicit single work run for the attached question", () => { + const c = { version: 1, id: "qa", mode: "existing-cloud", adapterModule: "/private/provider.mjs", journeys: ["task-create", "ask-question", "task-properties"], + executionEnvironment: "daytona", baseline: { features: {}, connections: [] }, limits: { durationMs: 600_000, modelBudgetCents: 100, maxRuns: 1 } }; + expect(parseTargetConfig(c).journeys).toEqual(c.journeys); + expect(() => parseTargetConfig({ ...c, limits: { ...c.limits, maxRuns: 2 } })).toThrow("question_requires_one_bounded_run"); + expect(() => parseTargetConfig({ ...c, limits: { ...c.limits, modelBudgetCents: 0 } })).toThrow("question_requires_one_bounded_run"); + }); +}); diff --git a/tests/runner-e2e/product-journeys/basic-evidence.ts b/tests/runner-e2e/product-journeys/basic-evidence.ts new file mode 100644 index 00000000000..6616b18fef9 --- /dev/null +++ b/tests/runner-e2e/product-journeys/basic-evidence.ts @@ -0,0 +1,19 @@ +/** Independent saved-state oracle shared by attached and managed targets. */ +export function questionEvidencePasses(input: { + companyId: string; agentId: string; environmentId?: string; marker: string; model: string; provider: string; runtimeMode: string; + issue: { id: string; companyId: string; workMode: string; status: string; assigneeAgentId: string | null }; + runs: Array>; + comments: Array<{ authorAgentId?: string | null; body: string }>; +}) { + const { issue, runs } = input; + const run = runs[0]; + const provider = run?.runnerProfileJson?.nativeExecutionInput?.provider; + return issue.companyId === input.companyId && issue.workMode === "ask" && issue.status === "done" + && issue.assigneeAgentId === input.agentId && runs.length === 1 + && run?.companyId === input.companyId && run.agentId === input.agentId && run.issueId === issue.id + && run.status === "succeeded" && run.runtimeMode === input.runtimeMode + && input.runtimeMode === "native" && Boolean(input.environmentId && input.provider && input.model) + && run.contextSnapshot?.paperclipEnvironment?.id === input.environmentId + && provider?.kind === input.provider && provider?.model === input.model + && input.comments.filter(comment => comment.authorAgentId === input.agentId && comment.body.trim() === input.marker).length === 1; +} diff --git a/tests/runner-e2e/product-journeys/basic-flows.ts b/tests/runner-e2e/product-journeys/basic-flows.ts new file mode 100644 index 00000000000..0d1a19dc385 --- /dev/null +++ b/tests/runner-e2e/product-journeys/basic-flows.ts @@ -0,0 +1,56 @@ +import { expect } from "@playwright/test"; +import { runnerTasks } from "../catalog.js"; +import { createTaskThroughUi } from "../user-actions.js"; +import { questionEvidencePasses } from "./basic-evidence.js"; +import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; + +export async function taskCreate(context: JourneyContext, session: TargetSession, campaignId: string, evidence: JourneyEvidence) { + const title = `Journey QA create ${campaignId}`; + const prompt = "Keep this task for me to review. No agent work is required."; + const created = await createTaskThroughUi({ page: context.page, origin: context.api.origin, issuePrefix: context.company.issuePrefix, + agentName: "Me", title, prompt, workMode: "standard", requireExplicitTitle: true, + beforeDispatch: () => session.own({ kind: "issue", name: title }) }); + await session.own({ kind: "issue", name: title, id: created.issueId }); + const route = `/api/issues/${created.issueId}`; + const issue = await context.api.get(route); + if (issue.companyId !== context.company.id || issue.title !== title || issue.description !== prompt + || issue.assigneeUserId !== context.user.id || issue.assigneeAgentId !== null) throw new JourneyError("failed", "created_task_mismatch"); + await context.page.goto(`${context.api.origin}/${context.company.issuePrefix}/issues`); + await context.page.getByRole("link").filter({ hasText: title }).first().click(); + await context.page.reload(); + await expect(context.page.getByText(title, { exact: true }).first()).toBeVisible(); + const saved = await context.api.get(route); + if (saved.title !== title || saved.description !== prompt || saved.assigneeUserId !== context.user.id + || (await context.api.get(`${route}/runs`)).length) throw new JourneyError("failed", "created_task_not_persisted"); + evidence.checks.push({ id: "browser-created-task-persisted", passed: true }, { id: "no-agent-work", passed: true }); +} + +export async function askQuestion(context: JourneyContext, session: TargetSession, campaignId: string, evidence: JourneyEvidence) { + const agent = context.agent; + if (!agent) throw new JourneyError("blocked", "configured_agent_required"); + if (agent.runtimeMode !== "native" || !context.environmentId || !agent.model || !agent.provider) throw new JourneyError("unsupported", "question_execution_identity_unavailable"); + // Reuse the exact existing cross-model Ask fixture, including its terminal marker. + const fixture = runnerTasks.find(task => task.id === "ask-question")!; + const nonce = campaignId.replace(/-/g, "_"); + const title = `Journey QA ask ${campaignId}`; + const marker = fixture.buildVisibleMarker(nonce); + const created = await createTaskThroughUi({ page: context.page, origin: context.api.origin, issuePrefix: context.company.issuePrefix, + agentName: agent.name, title, prompt: fixture.buildPrompt(nonce), workMode: "ask", requireExplicitTitle: true, + beforeDispatch: () => session.own({ kind: "issue", name: title }) }); + await session.own({ kind: "issue", name: title, id: created.issueId }); + const route = `/api/issues/${created.issueId}`; + await expect.poll(async () => { + const issue = await context.api.get(route); + const summaries = await context.api.get>(`${route}/runs`); + const runs = await Promise.all(summaries.map(run => context.api.get>(`/api/heartbeat-runs/${run.runId}`))); + if (runs.length > 1 || runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status))) throw new JourneyError("failed", "question_run_failed"); + const comments = await context.api.get(`${route}/comments`); + return questionEvidencePasses({ companyId: context.company.id, agentId: agent.id, environmentId: context.environmentId, + marker, model: agent.model, provider: agent.provider, runtimeMode: agent.runtimeMode, issue, runs, comments }); + }, { timeout: 240_000, intervals: [1000, 2000, 3000] }).toBe(true); + const issue = await context.api.get(route); + await context.page.goto(`${context.api.origin}/${context.company.issuePrefix}/issues/${issue.identifier}`); + await context.page.reload(); + await expect(context.page.getByText(marker, { exact: true }).last()).toBeVisible(); + evidence.checks.push({ id: "question-answer-persisted", passed: true }, { id: "one-bound-successful-run", passed: true }, { id: "configured-model-and-environment", passed: true }); +} diff --git a/tests/runner-e2e/product-journeys/campaign.test.ts b/tests/runner-e2e/product-journeys/campaign.test.ts index e039157c8f1..b1a111df3cc 100644 --- a/tests/runner-e2e/product-journeys/campaign.test.ts +++ b/tests/runner-e2e/product-journeys/campaign.test.ts @@ -45,7 +45,7 @@ async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preA await writeFile(module, `import {readFile,writeFile} from 'node:fs/promises'; const identity={kind:'cloud',targetId:'qa',origin:'https://qa.example.test',commit:'${"a".repeat(40)}',cloudRelease:'v1',runtimeImage:'image',configurationHash:'config'}; let onboarded=false; -const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{async screenshot(){} }}; +const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{origin:identity.origin,post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{url:()=>identity.origin+'/QA/issues/QA-1',locator:()=>({isVisible:async()=>true,screenshot:async()=>{}}) }}; export const provider={version:1, async acquire(input){return { async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}}, async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}}, diff --git a/tests/runner-e2e/product-journeys/catalog.ts b/tests/runner-e2e/product-journeys/catalog.ts index 6077ff48c36..30f1d516faa 100644 --- a/tests/runner-e2e/product-journeys/catalog.ts +++ b/tests/runner-e2e/product-journeys/catalog.ts @@ -1,5 +1,7 @@ import type { JourneyId, TargetConfig } from "./types.js"; export const productJourneys = [ + { id: "task-create", label: "Create a task for yourself and reopen it", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/create"], paid: false }, + { id: "ask-question", label: "Ask an agent a question and read its answer", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true }, { id: "task-properties", label: "Assign a task to the signed-in user", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["tasks/properties"], paid: false }, { id: "cloud-onboarding", label: "Create a Cloud workspace and complete a first task", requires: ["cloud-account", "fresh-workspace", "provider-key"], capabilities: ["onboarding/first-task"], paid: true }, ] as const; @@ -38,6 +40,7 @@ export function parseTargetConfig(value: unknown): TargetConfig { if (c.mode === "new-cloud" && c.baseline.connections.length) throw new Error("onboarding_requires_ui_connection_setup"); if (c.mode !== "new-cloud" && c.journeys.includes("cloud-onboarding")) throw new Error("onboarding_requires_fresh_cloud_workspace"); if (c.journeys.includes("cloud-onboarding") && (!c.limits.modelBudgetCents || !c.limits.maxRuns)) throw new Error("paid_journey_requires_limits"); + if (c.journeys.includes("ask-question") && (!c.limits.modelBudgetCents || c.limits.maxRuns !== 1)) throw new Error("question_requires_one_bounded_run"); if (c.mode !== "managed-self-hosted" && !c.adapterModule) throw new Error("cloud_target_requires_private_adapter"); if (c.mode === "managed-self-hosted" && c.executionEnvironment !== "local") throw new Error("unsupported_self_hosted_environment"); return { ...c, ...(c.journeys.includes("cloud-onboarding") ? { modelProfileId: c.modelProfileId ?? "runner-codex" as const } : {}), headed: c.headed ?? false, journeys: c.journeys as JourneyId[] }; diff --git a/tests/runner-e2e/product-journeys/fixtures.ts b/tests/runner-e2e/product-journeys/fixtures.ts index 8a13eee0c02..efd632b761e 100644 --- a/tests/runner-e2e/product-journeys/fixtures.ts +++ b/tests/runner-e2e/product-journeys/fixtures.ts @@ -11,7 +11,7 @@ export function journeyFixtures(session: TargetSession, campaignId?: string) { id: "ready-context", async setup() { return ready ??= await (preparation ??= session.prepare()); }, }); - for (const id of ["company", "board-user", "task-write"]) registry.register({ + for (const id of ["company", "board-user", "task-write", "agent"]) registry.register({ id, dependencies: ["ready-context"], async setup(resolved) { const context = resolved.get("ready-context") as JourneyContext; diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index 003d998ab4c..f98430df099 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -9,12 +9,14 @@ import { runnerSuites } from "../catalog.js"; import { productJourneys, parseTargetConfig, preparationPreview } from "./catalog.js"; import { journeyFixtures } from "./fixtures.js"; import { taskProperties } from "./task-properties.js"; +import { askQuestion, taskCreate } from "./basic-flows.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js"; import type { RunnerE2EResult } from "../types.js"; import { regenerateRunnerDashboard } from "../dashboard-regenerate.js"; import { resolveRunnerE2ESource } from "../source.js"; import { validateRetainedRunnerResult } from "../result-validation.js"; import { validateJourneyCollection } from "./collection.js"; +import { captureProductScreenshot } from "./product-screenshot.js"; import { validateOnboardingArtifact } from "./artifact-evidence.js"; const cleanupGraceMs = 600_000; @@ -99,9 +101,11 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["catalog.ts", "task-properties.ts", "basic-flows.ts", "basic-evidence.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + definitionHash.update(await readFile(path.join(import.meta.dirname, "../user-actions.ts"))); + definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition")); const suiteDefinitionHash = definitionHash.digest("hex"); - const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`); + const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : id === "ask-question" ? "configured-agent" : "browser"}.${config.executionEnvironment}.${id}`); let reportPersisted = false; const failFinalization = (code: string) => { for (const result of results) { @@ -160,7 +164,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri const profile = id === "cloud-onboarding" ? runnerSuites.flatMap(suite => suite.profiles).find(p => p.id === config.modelProfileId) : undefined; const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: expected[index]!, suiteId: "product-journeys", suiteDefinitionHash, - source, attempt: 1, status: "failed", profileId: profile?.id ?? "browser", environmentId: config.executionEnvironment, caseId: id, + source, attempt: 1, status: "failed", profileId: id === "ask-question" ? "configured-agent" : profile?.id ?? "browser", environmentId: config.executionEnvironment, caseId: id, provider: profile?.provider ?? null, model: profile?.model ?? null, runtimeMode: profile?.generation ?? "none", startedAt, finishedAt: startedAt, durationMs: 0, cleanup: "not_started", productJourney: evidence, }; results.push(result); @@ -187,7 +191,15 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri evidence.startingState = ready.startingState; await persist(); phase = "journey"; - await taskProperties(ready, session!, campaignId, evidence); + if (id === "ask-question") { + if (!ready.agent) throw new JourneyError("blocked", "configured_agent_required"); + result.profileId = "configured-agent"; + result.provider = ready.agent.provider; + result.model = ready.agent.model; + result.runtimeMode = ready.agent.runtimeMode; + await askQuestion(ready, session!, campaignId, evidence); + } else if (id === "task-create") await taskCreate(ready, session!, campaignId, evidence); + else await taskProperties(ready, session!, campaignId, evidence); } evidence.deployment = ready.identity; evidence.startingState = ready.startingState; @@ -197,7 +209,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri result.status = "passed"; const evidenceDir = path.join(outputDir, "evidence", result.executionId, "attempt-1"); await mkdir(evidenceDir, { recursive: true, mode: 0o700 }); - await ready.page.screenshot({ path: path.join(evidenceDir, `${id}.png`) }); + await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}.png`)); result.screenshots = [{ id, label: productJourneys.find(j => j.id === id)!.label, file: `${id}.png` }]; } catch (error) { const failure = journeyFailure(error); @@ -206,6 +218,18 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri result.status = "failed"; result.error = failure.code; result.failureClass = failure.outcome === "failed" ? "candidate_failure" : "permanent_infrastructure"; + // Retain failed product screens only after tenant setup, never login screens. + if (ready && phase === "journey") { + try { + const url = new URL(ready.page.url()); + if (url.origin === ready.api.origin && /^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname)) { + const evidenceDir = path.join(outputDir, "evidence", result.executionId, "attempt-1"); + await mkdir(evidenceDir, { recursive: true, mode: 0o700 }); + await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}-failed.png`)); + result.screenshots = [{ id: `${id}-failed`, label: "Failed journey screen", file: `${id}-failed.png` }]; + } + } catch { /* Preserve the original failure when its page is unavailable. */ } + } if (id === "cloud-onboarding" || !ready) prerequisiteFailure = failure; } result.finishedAt = new Date().toISOString(); result.durationMs = Date.parse(result.finishedAt) - Date.parse(startedAt); diff --git a/tests/runner-e2e/product-journeys/product-screenshot.test.ts b/tests/runner-e2e/product-journeys/product-screenshot.test.ts new file mode 100644 index 00000000000..9d1cfd911cb --- /dev/null +++ b/tests/runner-e2e/product-journeys/product-screenshot.test.ts @@ -0,0 +1,20 @@ +import { describe, expect, it, vi } from "vitest"; +import type { Page } from "@playwright/test"; +import { captureProductScreenshot } from "./product-screenshot.js"; + +describe("product screenshot boundary", () => { + it("excludes a sign-in screen rendered at an unchanged task URL", async () => { + const screenshot = vi.fn(); + const page = { url: () => "https://qa.example.test/QA/issues/QA-1", locator: () => ({ isVisible: async () => false, screenshot }) } as unknown as Page; + await expect(captureProductScreenshot(page, "https://qa.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable"); + expect(screenshot).not.toHaveBeenCalled(); + }); + it("captures only the signed-in product region and refuses other origins", async () => { + const screenshot = vi.fn(), locator = vi.fn(() => ({ isVisible: async () => true, screenshot })); + const page = { url: () => "https://qa.example.test/QA/issues/QA-1", locator } as unknown as Page; + await captureProductScreenshot(page, "https://qa.example.test", "shot.png"); + expect(locator).toHaveBeenCalledWith("main#main-content"); expect(screenshot).toHaveBeenCalledTimes(1); + await expect(captureProductScreenshot(page, "https://other.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable"); + expect(screenshot).toHaveBeenCalledTimes(1); + }); +}); diff --git a/tests/runner-e2e/product-journeys/product-screenshot.ts b/tests/runner-e2e/product-journeys/product-screenshot.ts new file mode 100644 index 00000000000..14ae3d0a9c8 --- /dev/null +++ b/tests/runner-e2e/product-journeys/product-screenshot.ts @@ -0,0 +1,12 @@ +import type { Page } from "@playwright/test"; +import { JourneyError } from "./types.js"; + +export async function captureProductScreenshot(page: Page, origin: string, file: string) { + const url = new URL(page.url()); + if (url.origin !== origin || !/^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname)) throw new JourneyError("failed", "product_screen_unavailable"); + // CloudAccessGate replaces Layout on expiry without necessarily changing the + // URL. Capture Layout's main element, so auth screens can never be included. + const product = page.locator("main#main-content"); + if (!await product.isVisible()) throw new JourneyError("failed", "product_screen_unavailable"); + await product.screenshot({ path: file, timeout: 2000 }); +} diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index 00bdeb5c164..54b2f53eb0c 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -2,7 +2,7 @@ import type { BrowserContext, Page } from "@playwright/test"; import type { ConnectionApi } from "../connection-target.js"; export type JourneyOutcome = "passed" | "failed" | "blocked" | "unsupported"; -export type JourneyId = "task-properties" | "cloud-onboarding"; +export type JourneyId = "task-properties" | "task-create" | "ask-question" | "cloud-onboarding"; export type TargetMode = "managed-self-hosted" | "existing-cloud" | "new-cloud"; export interface DeploymentIdentity { kind: "self-hosted" | "cloud"; @@ -80,6 +80,7 @@ export interface JourneyContext { identity: DeploymentIdentity; capabilities: string[]; startingState: "recipe" | "onboarding"; + agent?: { id: string; name: string; provider: string; model: string; runtimeMode: "native" | "legacy" }; fixtures?: { backlogTask?: PreparedBacklogTask }; } export interface PreparedBacklogTask { diff --git a/tests/runner-e2e/report-catalog.test.ts b/tests/runner-e2e/report-catalog.test.ts index 0cb16af378c..d205e95191b 100644 --- a/tests/runner-e2e/report-catalog.test.ts +++ b/tests/runner-e2e/report-catalog.test.ts @@ -85,6 +85,9 @@ describe("trusted report catalog discovery", () => { const known = runnerMatrix.find((execution) => execution.id === "core-compatibility.legacy-codex.local.ask-question")!; expect(journeyCapabilities(known).map((capability) => capability.id)).toEqual(["questions-and-approvals/questions"]); expect(journeyCapabilities({ ...known, suite: { ...known.suite, id: "branch-only" } })).toEqual([]); + const product = { ...known, suite: { ...known.suite, id: "product-journeys" } }; + expect(journeyCapabilities(product).map(capability => capability.id)).toEqual(["questions-and-approvals/questions"]); + expect(journeyCapabilities({ ...product, task: { ...known.task, id: "task-create" } }).map(capability => capability.id)).toEqual(["tasks/create"]); }); it("aggregates unknown suites and renders their cases, missing results, and screenshot gallery", () => { diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 59c7685f349..1ca685369a7 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -429,9 +429,13 @@ export function validateRetainedRunnerResult( if (checked.suiteId === "product-journeys" && checked.status === "passed") { const evidence = checked.productJourney; if (!evidence) return invalid("result.productJourney"); - const required = checked.caseId === "task-properties" ? ["assignment-persisted", "no-agent-work"] : ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"]; + const required = checked.caseId === "task-properties" ? ["assignment-persisted", "no-agent-work"] + : checked.caseId === "task-create" ? ["browser-created-task-persisted", "no-agent-work"] + : checked.caseId === "ask-question" ? ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"] + : ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"]; if (evidence.outcome !== "passed" || !evidence.deployment || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney"); - if (checked.caseId === "task-properties" && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly"); + if (["task-properties", "task-create"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly"); + if (checked.caseId === "ask-question" && (checked.runIds?.length !== 1 || !checked.model || !checked.provider || checked.runtimeMode === "none")) invalid("result.questionRun"); if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt"); } if (checked.suiteId === "provider-connections" && checked.status === "passed" && (!checked.providerConnection || !connectionEvidencePasses(checked.providerConnection))) invalid("result.providerConnection"); diff --git a/tests/runner-e2e/user-actions.ts b/tests/runner-e2e/user-actions.ts index 7ba0d18ecff..e9611348dc4 100644 --- a/tests/runner-e2e/user-actions.ts +++ b/tests/runner-e2e/user-actions.ts @@ -20,6 +20,7 @@ export async function fillTaskPrompt(editor: Locator, prompt: string): Promise void; + beforeDispatch?: () => void | Promise; }) { // Search's public creation action exposes the explicit title field. Strict // native-operation fixtures use it so automatic task naming is not an extra @@ -42,7 +43,7 @@ export async function createTaskThroughUi(input: { let bootstrapError: unknown; for (let bootstrapAttempt = 1; bootstrapAttempt <= 3; bootstrapAttempt += 1) { try { - await input.page.goto(issuesUrl, { + await input.page.goto(input.origin ? new URL(issuesUrl, input.origin).href : issuesUrl, { waitUntil: "domcontentloaded", timeout: 30_000, }); @@ -106,7 +107,7 @@ export async function createTaskThroughUi(input: { } // The prompt-only composer generates a provisional title which the provider // can rename immediately. Bind the fixture to the actual creation response. - input.beforeDispatch?.(); + await input.beforeDispatch?.(); const submittedAtMs = Date.now(); const [response] = await Promise.all([ input.page.waitForResponse(response => response.request().method() === "POST" diff --git a/ui/src/components/observability/AgentObservabilitySettings.tsx b/ui/src/components/observability/AgentObservabilitySettings.tsx index 88ce70645e2..ae063a61061 100644 --- a/ui/src/components/observability/AgentObservabilitySettings.tsx +++ b/ui/src/components/observability/AgentObservabilitySettings.tsx @@ -94,7 +94,7 @@ export function AgentObservabilitySettingsView(props: AgentObservabilitySettings return

Agent observability

-

Send Codex and Claude Code logs and traces to your observability service.

+

Send Codex, Claude Code and native OpenCode execution data to your observability service.

{!canManage ?

Manage connections permission is required to configure agent observability.

: <>