diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 3855bea4ae1..2aea0368662 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -180,6 +180,16 @@ checkout, including a non-primary checkout. Changing projects clears the previous worktree choice. Switching from reuse to a new worktree or the project workspace restores the project default checkout. Task creation uses the selected company and has no separate heading or settings control. + +Task execution-policy controls apply the shared schema defaults when stages or +participants are omitted. An invalid policy shows an unavailable notice and +disables policy edits while leaving other task properties usable. Refresh to +retry. Optional browser error monitoring reports only a fixed field category, +once per mounted control; it never includes the policy or task identifiers. +Edits in the reviewer and monitor controls retain an explicitly configured +review-round limit when the last reviewer or monitor is removed, so adding a +reviewer later uses the saved limit. + Fresh tasks start with an empty request and the last task assignee chosen in that company, including a human. If that assignee is unavailable, the CEO is the default, or the first eligible agent when the company has no CEO. Explicit launch @@ -1232,6 +1242,15 @@ all service instances share in-flight materialization. Existing public helpers and the workspace validation error class remain available from `heartbeat.ts`. Keep workspace policy changes separate from scheduling and run execution changes. +Run preparation is in `server/src/services/heartbeat/run-preparation.ts`. It owns +issue and wake context, responsible-user resolution, routine environment snapshots, +skill mentions, adapter environment configuration, and MCP/tool access setup. +`createHeartbeatRunPreparation(db)` binds the context loaders to a service's database +without doing database work during construction. `heartbeat.ts` keeps queueing, +dispatch, retries, cancellation, and execution order, and re-exports the existing +public helpers and configuration-incomplete error class. Keep preparation policy +changes in this module and its tests. + ## Wake Context Delivery Built-in adapters deliver wake context through the run prompt, including structured diff --git a/doc/observability.md b/doc/observability.md index 2114c16eb79..b62a1263309 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -448,9 +448,45 @@ explicit `local_path` or `non_git_path` project workspace has no repository URL, Git confirms the selected directory is not a repository, and the task requests a Git worktree. The run must fail during setup before provider work starts. Its existing validation error, blocked task and board recovery action remain. -Git command failures, permissions, corrupt repositories, failed materialization, -fallback paths, and generic missing or unrestorable workspaces remain reportable; -the error text or `workspace_validation_failed` code alone never suppresses them. +Local Git command failures, permissions, corrupt repositories, fallback paths, +and generic missing or unrestorable workspaces remain reportable. The error text +or `workspace_validation_failed` code alone never suppresses them. + +Recognized external connection failures also stay in the local run result instead +of producing application-error events in Sentry. Their producer records a closed +`connectionFailure` descriptor: schema version, provider, operation, and reason. +It contains no URL, host, path, credential, or remote response. This classification +does not change the run's failed state, task recovery action, retry policy, or +whether remote work may have been accepted. + +- Managed remote Git clones classify known authentication, repository access, + invalid URL, DNS (including machine-specific SSH aliases), connection, TLS, + rate-limit, and upstream failures from the actual clone subprocess. Setup must + fail before provider work starts. Every failed workspace candidate must have + proven connection evidence; a mixed or unknown failure remains reportable. + Local/file clones, credential-provider exceptions, local checkout or copy + failures, process signals, and repository corruption remain reportable. Remote + fetch failures already return workspace warnings; their classification is not + transferred to a later failure resolving a local base ref. +- Hermes gateway configuration validation and the initial create-run request use + exact config, HTTP, or structured transport evidence. Missing configuration, + rejected credentials, unavailable endpoints, rate limits, and upstream outages + remain visible to the run owner. Unknown protocol/parser errors, missing run + identifiers, untyped terminal failures, and timeouts after accepted work remain + reportable. Stream/poll errors retain their existing retry and logging behavior. + +Expected pre-provider waits for a shared project workspace, a credential rotation +lock, or an exhausted AI connection pool carry a server-owned expected +cancellation receipt. They retain existing retry and lock behavior without a +Sentry cancellation event. A later failed retry still reports its actual error; +the wait code alone is not sufficient to suppress an event. + +MCP tool discovery records recognized outbound HTTP/transport failures through +server-only provenance. These failures preserve the response status, stored +connection health, audit record, local context, and first-party telemetry while +skipping only Sentry. A proven missing OAuth scope returns an actionable 422 +response with its reconnect guidance. Unknown responses, parsing errors, local +callbacks, database errors, and unproven network-looking messages still report. The `run_failure` context also includes the recorded process `exitCode` and `signal`, so a generic adapter error can still distinguish a nonzero exit from diff --git a/packages/adapter-utils/src/connection-failure.test.ts b/packages/adapter-utils/src/connection-failure.test.ts new file mode 100644 index 00000000000..602e01fd9ec --- /dev/null +++ b/packages/adapter-utils/src/connection-failure.test.ts @@ -0,0 +1,105 @@ +import { describe, expect, it } from "vitest"; +import { readConnectionFailure } from "./connection-failure.js"; + +describe("readConnectionFailure", () => { + const git = { schemaVersion: 1, provider: "git", operation: "clone", reason: "authentication_failed" }; + const gateway = { schemaVersion: 1, provider: "hermes_gateway", operation: "create_run", reason: "connection_refused" }; + const mcp = { schemaVersion: 1, provider: "mcp_http", operation: "discover_tools", reason: "connection_refused" }; + + it.each([ + "authentication_failed", "repository_unavailable", "invalid_remote", "dns_failure", "connection_refused", + "network_unreachable", "connection_reset", "connection_timeout", "tls_failure", "rate_limited", "remote_unavailable", + ])("accepts the Git clone reason %s", (reason) => { + const input = { ...git, reason }; + expect(readConnectionFailure(input)).toEqual(input); + }); + + it.each(["endpoint_missing", "endpoint_invalid", "insecure_transport", "credentials_missing"])( + "accepts the gateway configuration reason %s", (reason) => { + const input = { ...gateway, operation: "configuration", reason }; + expect(readConnectionFailure(input)).toEqual(input); + }, + ); + + it.each([ + "authentication_failed", "endpoint_not_found", "rate_limited", "remote_unavailable", "connection_refused", + "dns_failure", "network_unreachable", "connection_reset", "connection_timeout", "tls_failure", + ])("accepts the gateway request reason %s", (reason) => { + const input = { ...gateway, reason }; + expect(readConnectionFailure(input)).toEqual(input); + }); + + it.each([ + "authentication_failed", "endpoint_not_found", "rate_limited", "remote_unavailable", "connection_refused", + "dns_failure", "network_unreachable", "connection_reset", "connection_timeout", "tls_failure", + ])("accepts the MCP tool discovery request reason %s", (reason) => { + const input = { ...mcp, reason }; + expect(readConnectionFailure(input)).toEqual(input); + }); + + it.each([ + { ...git, operation: "fetch" }, + { ...git, operation: "create_run" }, + { ...git, reason: "endpoint_not_found" }, + { ...gateway, operation: "clone" }, + { ...gateway, operation: "poll_status" }, + { ...gateway, operation: "stop_run" }, + { ...gateway, operation: "configuration" }, + { ...gateway, reason: "credentials_missing" }, + { ...gateway, reason: "repository_unavailable" }, + { ...gateway, reason: "timeout" }, + { ...gateway, reason: "unknown" }, + { ...gateway, provider: "other_gateway" }, + { ...gateway, provider: "hermes_local" }, + { ...mcp, operation: "create_run" }, + { ...mcp, operation: "clone" }, + { ...mcp, operation: "configuration" }, + { ...mcp, operation: "call_tool" }, + { ...mcp, reason: "credentials_missing" }, + { ...mcp, reason: "repository_unavailable" }, + { ...gateway, operation: "discover_tools" }, + { ...git, operation: "discover_tools" }, + ])("rejects an unsupported provider/operation/reason combination: %j", (input) => { + expect(readConnectionFailure(input)).toBeNull(); + }); + + it.each([ + undefined, null, false, 1, "connection_refused", [], [gateway], {}, + { ...gateway, schemaVersion: 0 }, { ...gateway, schemaVersion: 2 }, { ...gateway, schemaVersion: "1" }, + { ...gateway, schemaVersion: null }, { ...gateway, schemaVersion: undefined }, + { ...gateway, provider: null }, { ...gateway, provider: ["hermes_gateway"] }, + { ...gateway, operation: null }, { ...gateway, operation: 1 }, + { ...gateway, reason: null }, { ...gateway, reason: 1 }, { ...gateway, reason: ["connection_refused"] }, + { ...gateway, reason: { code: "ECONNREFUSED" } }, + { provider: "hermes_gateway", operation: "create_run", reason: "connection_refused" }, + { ...gateway, reason: "connection_refused " }, { ...gateway, reason: "CONNECTION_REFUSED" }, + ])("rejects unknown versions or malformed values: %j", (input) => { + expect(readConnectionFailure(input)).toBeNull(); + }); + + it.each([ + { url: "https://private.example.test" }, + { account: "private-account" }, + { token: "synthetic-secret" }, + { remoteRunAccepted: false }, + { providerWorkStarted: false }, + { details: { message: "private diagnostics" } }, + { extra: undefined }, + ])("rejects extra metadata instead of broadening the bounded marker: %j", (extra) => { + expect(readConnectionFailure({ ...gateway, ...extra })).toBeNull(); + }); + + it("rejects unknown own fields even when a prototype supplies a valid descriptor", () => { + const input = Object.assign(Object.create(gateway), { a: 1, b: 2, c: 3, d: 4 }); + expect(readConnectionFailure(input)).toBeNull(); + }); + + it("returns an independent bounded value without mutating its producer's input", () => { + const input = Object.freeze({ ...gateway }); + const output = readConnectionFailure(input); + expect(output).toEqual(gateway); + expect(output).not.toBe(input); + expect(input).toEqual(gateway); + expect(Object.keys(output ?? {}).sort()).toEqual(["operation", "provider", "reason", "schemaVersion"]); + }); +}); diff --git a/packages/adapter-utils/src/connection-failure.ts b/packages/adapter-utils/src/connection-failure.ts new file mode 100644 index 00000000000..a5a3e5895d2 --- /dev/null +++ b/packages/adapter-utils/src/connection-failure.ts @@ -0,0 +1,58 @@ +/** + * A bounded producer-owned explanation of an external connection failure. + * This changes error reporting only: it never proves remote acceptance, + * authorizes a retry, or turns a failed run into a success. + */ +const gitReasons = [ + "authentication_failed", "repository_unavailable", "invalid_remote", + "dns_failure", "connection_refused", "network_unreachable", "connection_reset", + "connection_timeout", "tls_failure", "rate_limited", "remote_unavailable", +] as const; +const hermesConfigurationReasons = [ + "endpoint_missing", "endpoint_invalid", "insecure_transport", "credentials_missing", +] as const; +const hermesRequestReasons = [ + "authentication_failed", "endpoint_not_found", "rate_limited", "remote_unavailable", + "connection_refused", "dns_failure", "network_unreachable", "connection_reset", + "connection_timeout", "tls_failure", +] as const; + +export type GitConnectionFailure = { + schemaVersion: 1; + provider: "git"; + operation: "clone"; + reason: typeof gitReasons[number]; +}; + +export type ConnectionFailure = GitConnectionFailure | { + schemaVersion: 1; + provider: "hermes_gateway"; +} & ({ + operation: "configuration"; + reason: typeof hermesConfigurationReasons[number]; +} | { + operation: "create_run"; + reason: typeof hermesRequestReasons[number]; +}) | { + schemaVersion: 1; + provider: "mcp_http"; + operation: "discover_tools"; + reason: typeof hermesRequestReasons[number]; +}; + +/** Unknown versions, combinations and extra payload fields remain reportable. */ +export function readConnectionFailure(value: unknown): ConnectionFailure | null { + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const record = value as Record; + const keys = Object.keys(record); + if (keys.length !== 4 || !keys.every((key) => ["schemaVersion", "provider", "operation", "reason"].includes(key)) || + record.schemaVersion !== 1 || typeof record.reason !== "string") return null; + let reasons: readonly string[]; + if (record.provider === "git" && record.operation === "clone") reasons = gitReasons; + else if (record.provider === "hermes_gateway" && record.operation === "configuration") reasons = hermesConfigurationReasons; + else if (record.provider === "hermes_gateway" && record.operation === "create_run") reasons = hermesRequestReasons; + else if (record.provider === "mcp_http" && record.operation === "discover_tools") reasons = hermesRequestReasons; + else return null; + if (!reasons.includes(record.reason)) return null; + return { schemaVersion: 1, provider: record.provider, operation: record.operation, reason: record.reason } as ConnectionFailure; +} diff --git a/packages/adapters/hermes/src/gateway/server/execute.test.ts b/packages/adapters/hermes/src/gateway/server/execute.test.ts index efe4df66660..39393432a73 100644 --- a/packages/adapters/hermes/src/gateway/server/execute.test.ts +++ b/packages/adapters/hermes/src/gateway/server/execute.test.ts @@ -633,7 +633,9 @@ describe("execute", () => { expect(result).toMatchObject({ exitCode: 1, signal: null, timedOut: false, errorCode: "hermes_gateway_connect_failed", errorFamily: "transient_upstream", retryNotBefore: null }); expect(result.executionRecovery).toBeUndefined(); - expect(result.resultJson).toBeUndefined(); + expect(result.resultJson).toEqual({ connectionFailure: { + schemaVersion: 1, provider: "hermes_gateway", operation: "create_run", reason: "connection_refused", + } }); expect(ctx.onDispatch).toHaveBeenCalledOnce(); expect(fetchMock).toHaveBeenCalledOnce(); expect(fetchMock.mock.calls[0]?.[1]?.redirect).toBeUndefined(); @@ -738,6 +740,189 @@ describe("execute", () => { }); }); +describe("gateway connection failure reporting", () => { + const config = { apiBaseUrl: "https://gateway.example.test", apiKey: "gateway-private-key" }; + const marker = (operation: string, reason: string) => ({ + schemaVersion: 1, provider: "hermes_gateway", operation, reason, + }); + + it.each([ + [{}, "api_base_url_missing", "endpoint_missing"], + [{ apiBaseUrl: "not a URL" }, "api_base_url_invalid", "endpoint_invalid"], + [{ apiBaseUrl: "ftp://gateway.example.test" }, "api_base_url_invalid", "endpoint_invalid"], + [{ apiBaseUrl: "http://gateway.example.test" }, "plain_http_remote_denied", "insecure_transport"], + [{ apiBaseUrl: config.apiBaseUrl }, "api_key_missing", "credentials_missing"], + ])("classifies configuration failure %s without dispatch", async (settings, code, reason) => { + const fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); + const ctx = makeCtx(settings); + ctx.onDispatch = vi.fn(); + const result = await execute(ctx); + expect(result).toMatchObject({ exitCode: 1, signal: null, timedOut: false, + errorCode: `hermes_gateway_${code}`, resultJson: { connectionFailure: marker("configuration", reason) } }); + expect(result.errorMessage).toBeTruthy(); + expect(ctx.onDispatch).not.toHaveBeenCalled(); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it.each([ + [401, "auth_failed", "authentication_failed", null], + [403, "auth_failed", "authentication_failed", null], + [404, "runs_unsupported", "endpoint_not_found", null], + [429, "rate_limited", "rate_limited", "transient_upstream"], + [500, "upstream_error", "remote_unavailable", "transient_upstream"], + [503, "upstream_error", "remote_unavailable", "transient_upstream"], + [599, "upstream_error", "remote_unavailable", "transient_upstream"], + ])("classifies HTTP %i without changing retry or dispatch behavior", async (status, code, reason, errorFamily) => { + const fetchMock = vi.fn(async () => new Response(JSON.stringify({ detail: "remote failure" }), { + status, headers: { "retry-after": "7" }, + })); + vi.stubGlobal("fetch", fetchMock); + const ctx = makeCtx(config); + ctx.onDispatch = vi.fn(); + const result = await execute(ctx); + expect(result).toMatchObject({ exitCode: 1, signal: null, timedOut: false, + errorCode: `hermes_gateway_${code}`, errorFamily, retryNotBefore: "7", + errorMeta: { status, body: { detail: "remote failure" } }, + resultJson: { connectionFailure: marker("create_run", reason) } }); + expect(result.errorMessage).toContain(`Hermes gateway HTTP ${status}`); + expect(result.executionRecovery).toBeUndefined(); + expect(ctx.onDispatch).toHaveBeenCalledOnce(); + expect(fetchMock).toHaveBeenCalledOnce(); + expect(Object.keys(result.resultJson ?? {})).toEqual(["connectionFailure"]); + }); + + it.each([ + ["ECONNREFUSED", "connection_refused"], ["ENOTFOUND", "dns_failure"], ["EAI_AGAIN", "dns_failure"], + ["EHOSTUNREACH", "network_unreachable"], ["ENETUNREACH", "network_unreachable"], + ["ECONNRESET", "connection_reset"], ["EPIPE", "connection_reset"], ["UND_ERR_SOCKET", "connection_reset"], + ["ETIMEDOUT", "connection_timeout"], ["UND_ERR_CONNECT_TIMEOUT", "connection_timeout"], + ["UND_ERR_HEADERS_TIMEOUT", "connection_timeout"], ["UND_ERR_BODY_TIMEOUT", "connection_timeout"], + ["CERT_HAS_EXPIRED", "tls_failure"], ["CERT_NOT_YET_VALID", "tls_failure"], + ["DEPTH_ZERO_SELF_SIGNED_CERT", "tls_failure"], ["SELF_SIGNED_CERT_IN_CHAIN", "tls_failure"], + ["UNABLE_TO_VERIFY_LEAF_SIGNATURE", "tls_failure"], ["UNABLE_TO_GET_ISSUER_CERT_LOCALLY", "tls_failure"], + ["ERR_TLS_CERT_ALTNAME_INVALID", "tls_failure"], + ])("classifies structured fetch cause %s", async (code, reason) => { + vi.stubGlobal("fetch", vi.fn(async () => { + throw new TypeError("fetch failed", { cause: Object.assign(new Error("external transport failed"), { code }) }); + })); + const result = await execute(makeCtx(config)); + expect(result).toMatchObject({ exitCode: 1, signal: null, timedOut: false, + errorCode: "hermes_gateway_connect_failed", errorFamily: "transient_upstream", retryNotBefore: null, + resultJson: { connectionFailure: marker("create_run", reason) } }); + expect(result.errorMessage).toContain(code); + expect(result.executionRecovery).toBeUndefined(); + }); + + it("classifies a network failure reading the create response without rewriting its error", async () => { + const response = new Response("unread"); + vi.spyOn(response, "text").mockRejectedValue(new TypeError("terminated", { cause: { code: "UND_ERR_SOCKET" } })); + vi.stubGlobal("fetch", vi.fn(async () => response)); + const result = await execute(makeCtx(config)); + expect(result).toMatchObject({ exitCode: 1, signal: null, timedOut: false, + errorCode: "hermes_gateway_protocol_error", errorFamily: null, errorMessage: "terminated", + resultJson: { connectionFailure: marker("create_run", "connection_reset") } }); + expect(result.executionRecovery).toBeUndefined(); + }); + + it("requires every aggregate member to prove the same external failure", async () => { + vi.stubGlobal("fetch", vi.fn(async () => { throw new TypeError("fetch failed", { cause: { + code: "ECONNREFUSED", errors: [{ code: "ECONNREFUSED" }, { code: "ECONNREFUSED" }], + } }); })); + const result = await execute(makeCtx(config)); + expect(result.resultJson).toEqual({ connectionFailure: marker("create_run", "connection_refused") }); + }); + + it.each([ + undefined, { message: "ECONNREFUSED" }, { code: "ERR_INTERNAL_ASSERTION" }, { code: "toString" }, + { code: "ECONNREFUSED", errors: [] }, { code: "ECONNREFUSED", errors: "invalid" }, + { code: "ECONNREFUSED", errors: Array.from({ length: 9 }, () => ({ code: "ECONNREFUSED" })) }, + { code: "ECONNREFUSED", errors: [{ code: "ECONNREFUSED" }, { code: "ETIMEDOUT" }] }, + { code: "ECONNREFUSED", errors: [{ code: "ECONNREFUSED" }, new TypeError("internal bug")] }, + { code: "ERR_INTERNAL_ASSERTION", errors: [{ code: "ECONNREFUSED" }] }, + { code: "ECONNRESET", name: "AbortError" }, + ])("keeps unknown, mixed, or internal transport causes reportable: %j", async (cause) => { + vi.stubGlobal("fetch", vi.fn(async () => { throw new TypeError("ECONNREFUSED", { cause }); })); + const result = await execute(makeCtx(config)); + expect(result.errorCode).toBe("hermes_gateway_connect_failed"); + expect(result.resultJson).toBeUndefined(); + }); + + it("keeps abort failures reportable even if their cause looks like a transport error", async () => { + const aborted = Object.assign(new Error("cancelled", { cause: { code: "ECONNRESET" } }), { name: "AbortError" }); + vi.stubGlobal("fetch", vi.fn(async () => { throw aborted; })); + const result = await execute(makeCtx(config)); + expect(result.resultJson).toBeUndefined(); + expect(result.errorCode).toBe("hermes_gateway_connect_failed"); + }); + + it.each([400, 409, 422])("does not trust HTTP %i body claims about connection failures", async (status) => { + vi.stubGlobal("fetch", vi.fn(async () => new Response(JSON.stringify({ + error: "ECONNREFUSED", connectionFailure: marker("create_run", "connection_refused"), + }), { status }))); + const result = await execute(makeCtx(config)); + expect(result.errorCode).toBe("hermes_gateway_protocol_error"); + expect(result.resultJson).toBeUndefined(); + }); + + it.each(["not json", JSON.stringify({ status: "running", connectionFailure: marker("create_run", "connection_refused") })])( + "keeps malformed successful create responses reportable", async (body) => { + vi.stubGlobal("fetch", vi.fn(async () => new Response(body))); + const result = await execute(makeCtx(config)); + expect(result.errorCode).toBe("hermes_gateway_protocol_error"); + expect(result.resultJson).toBeUndefined(); + }, + ); + + it("does not classify a local response reader TypeError", async () => { + const response = new Response("unread"); + vi.spyOn(response, "text").mockRejectedValue(new TypeError("cannot read property of undefined")); + vi.stubGlobal("fetch", vi.fn(async () => response)); + const result = await execute(makeCtx(config)); + expect(result.errorCode).toBe("hermes_gateway_protocol_error"); + expect(result.resultJson).toBeUndefined(); + }); + + it("does not trust a callback error with forged gateway codes or markers", async () => { + const fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); + const ctx = makeCtx(config); + ctx.onDispatch = () => { throw Object.assign(new Error("callback bug"), { + code: "hermes_gateway_auth_failed", status: 401, cause: { code: "ECONNREFUSED" }, + connectionFailure: marker("create_run", "authentication_failed"), + }); }; + const result = await execute(ctx); + expect(result.resultJson).toBeUndefined(); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it.each(["completed", "failed", "cancelled"])("does not classify remote terminal %s from text or a forged marker", (status) => { + const result = mapFinalResultForTest({ + terminal: { runId: "run-remote", status, payload: { + error: "ECONNREFUSED", code: "ECONNREFUSED", + connectionFailure: marker("create_run", "connection_refused"), + } }, outputChunks: [], sessionKey: null, strategy: "none", + }); + expect(result.resultJson?.connectionFailure).toBeUndefined(); + expect(result.errorCode).toBe(status === "completed" ? undefined : status === "failed" ? "hermes_gateway_run_failed" : "hermes_gateway_cancelled"); + }); + + it("retains an ambiguous accepted-run timeout after stream, poll, and stop connection failures", async () => { + const fetchMock = vi.fn(async (input: RequestInfo | URL) => { + if (String(input).endsWith("/v1/runs")) return new Response(JSON.stringify({ run_id: "run-accepted" })); + throw new TypeError("fetch failed", { cause: { code: "ECONNRESET" } }); + }); + vi.stubGlobal("fetch", fetchMock); + const ctx = makeCtx({ ...config, timeoutSec: 0.3, pollIntervalMs: 250, eventReconnectMs: 250 }); + const result = await execute(ctx); + expect(result).toMatchObject({ timedOut: true, errorCode: "hermes_gateway_timeout" }); + expect(result.resultJson?.connectionFailure).toBeUndefined(); + expect(result.resultJson?.run_id).toBe("run-accepted"); + expect(fetchMock.mock.calls.some(([input]) => String(input).endsWith("/stop"))).toBe(true); + expect(ctx.onLog).toHaveBeenCalledWith("stderr", expect.stringContaining("status poll failed")); + }); +}); + describe("testEnvironment", () => { it("fails remote plain HTTP before probing health", async () => { const fetchMock = vi.fn(async () => new Response("{}", { status: 200 })); diff --git a/packages/adapters/hermes/src/gateway/server/execute.ts b/packages/adapters/hermes/src/gateway/server/execute.ts index 6942af59fc0..e4551b1cb6d 100644 --- a/packages/adapters/hermes/src/gateway/server/execute.ts +++ b/packages/adapters/hermes/src/gateway/server/execute.ts @@ -3,6 +3,7 @@ import type { AdapterExecutionResult, UsageSummary, } from "@paperclipai/adapter-utils"; +import type { ConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; import { asNumber, asString, @@ -61,6 +62,72 @@ type ExecutionState = { type TextRedactor = (value: string) => string; +type GatewayConfigurationFailure = Extract; +type GatewayRequestFailure = Extract; + +// Only errors observed at our outbound request boundary acquire this provenance. +// A callback or remote payload cannot opt itself out of product error reporting. +const requestConnectionFailures = new WeakMap(); +const NETWORK_FAILURE_REASONS: Readonly> = { + ECONNREFUSED: "connection_refused", + ENOTFOUND: "dns_failure", + EAI_AGAIN: "dns_failure", + EHOSTUNREACH: "network_unreachable", + ENETUNREACH: "network_unreachable", + ECONNRESET: "connection_reset", + EPIPE: "connection_reset", + UND_ERR_SOCKET: "connection_reset", + ETIMEDOUT: "connection_timeout", + UND_ERR_CONNECT_TIMEOUT: "connection_timeout", + UND_ERR_HEADERS_TIMEOUT: "connection_timeout", + UND_ERR_BODY_TIMEOUT: "connection_timeout", + CERT_HAS_EXPIRED: "tls_failure", + CERT_NOT_YET_VALID: "tls_failure", + DEPTH_ZERO_SELF_SIGNED_CERT: "tls_failure", + SELF_SIGNED_CERT_IN_CHAIN: "tls_failure", + UNABLE_TO_VERIFY_LEAF_SIGNATURE: "tls_failure", + UNABLE_TO_GET_ISSUER_CERT_LOCALLY: "tls_failure", + ERR_TLS_CERT_ALTNAME_INVALID: "tls_failure", +}; + +function gatewayConfigurationFailure(reason: GatewayConfigurationFailure["reason"]): GatewayConfigurationFailure { + return { schemaVersion: 1, provider: "hermes_gateway", operation: "configuration", reason }; +} + +function networkFailureReason(value: unknown, depth = 0): GatewayRequestFailure["reason"] | null { + const cause = asRecord(value); + if (!cause || depth > 2 || cause.name === "AbortError") return null; + const reason = typeof cause.code === "string" && Object.hasOwn(NETWORK_FAILURE_REASONS, cause.code) + ? NETWORK_FAILURE_REASONS[cause.code]! + : null; + if (cause.code !== undefined && !reason) return null; + if (cause.errors !== undefined) { + if (!Array.isArray(cause.errors) || cause.errors.length === 0 || cause.errors.length > 8) return null; + const reasons = cause.errors.map((error) => networkFailureReason(error, depth + 1)); + const first = reasons[0]; + return first && reasons.every((item) => item === first) && (!reason || reason === first) ? first : null; + } + return reason; +} + +function rememberNetworkFailure(target: Error, error: unknown): void { + if (!(error instanceof Error) || error.name === "AbortError") return; + // Native fetch wraps transport errors in cause. Do not classify an arbitrary + // TypeError, its text, or remote JSON that happens to name a network error. + const reason = networkFailureReason(error.cause); + if (reason) requestConnectionFailures.set(target, { + schemaVersion: 1, provider: "hermes_gateway", operation: "create_run", reason, + }); +} + +function httpFailureReason(status: number): GatewayRequestFailure["reason"] | null { + if (status === 401 || status === 403) return "authentication_failed"; + if (status === 404) return "endpoint_not_found"; + if (status === 429) return "rate_limited"; + if (status >= 500 && status <= 599) return "remote_unavailable"; + return null; +} + const CRITICAL_HEADERS = new Set([ "authorization", "content-type", @@ -380,9 +447,16 @@ async function fetchJson(input: RequestInfo | URL, init: RequestInit): Promise rmSync(root, { recursive: true, force: true })); + const pack = join(root, "temporary pack"); + mkdirSync(join(pack, "node_modules", ".bin"), { recursive: true }); + writeFileSync(join(pack, "package.json"), '{"type":"module"}'); + return { root, pack }; +} + +for (const platform of ["darwin-arm64", "darwin-x64", "linux-x64"]) { + test(`relocates the installed ${platform} Copilot command and preserves arguments and exit status`, (t) => { + const { root, pack } = fixture(t); + const name = `copilot-${platform}`; + const installed = join(pack, "node_modules", "@github", name); + mkdirSync(installed, { recursive: true }); + writeFileSync(join(installed, "package.json"), JSON.stringify({ + name: `@github/${name}`, exports: { ".": "./copilot" }, bin: { [name]: "copilot" }, + })); + // A provider-free executable proves relocation without invoking a real CLI. + writeFileSync(join(installed, "copilot"), '#!/bin/sh\nprintf "%s\\n" "$@"\nexit 17\n', { mode: 0o755 }); + const shim = join(pack, "node_modules", ".bin", name); + writeFileSync(shim, `#!/bin/sh\nexport NODE_PATH="${pack}/node_modules"\nexit 99\n`); + + writePortableCopilotShims(pack); + assert.equal(readFileSync(shim, "utf8").includes(root), false); + const relocated = join(root, "relocated pack"); + renameSync(pack, relocated); + const args = ["argument with spaces", "", "$(not-a-command)"]; + const result = spawnSync(join(relocated, "node_modules", ".bin", name), args, { + cwd: root, env: { PATH: "/usr/bin:/bin" }, encoding: "utf8", timeout: 5_000, + }); + assert.equal(result.error, undefined); + assert.equal(result.status, 17); + assert.equal(result.stdout, args.join("\n") + "\n"); + assert.equal(result.stderr, ""); + }); +} + +test("does not create commands for absent optional platform packages", (t) => { + const { pack } = fixture(t); + writePortableCopilotShims(pack); + for (const platform of ["darwin-arm64", "darwin-x64", "linux-x64"]) { + assert.equal(existsSync(join(pack, "node_modules", ".bin", `copilot-${platform}`)), false); + } +}); + +test("fails when an installed command has no matching package", (t) => { + const { pack } = fixture(t); + writeFileSync(join(pack, "node_modules", ".bin", "copilot-linux-x64"), "invalid old shim"); + assert.throws(() => writePortableCopilotShims(pack), { code: "MODULE_NOT_FOUND" }); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index fdda7fe759d..f60cc365a75 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -174,7 +174,7 @@ importers: dependencies: '@agentclientprotocol/claude-agent-acp': specifier: ^0.73.0 - version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.30.0(zod@4.6.5)) + version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)) '@anthropic-ai/sdk': specifier: 0.129.0 version: 0.129.0(zod@4.6.5) @@ -426,8 +426,8 @@ importers: packages/google-sheets-mcp-server: dependencies: '@modelcontextprotocol/sdk': - specifier: ^1.30.0 - version: 1.30.0(zod@4.4.3) + specifier: ^1.31.0 + version: 1.31.0(zod@4.4.3) googleapis: specifier: ^182.0.0 version: 182.0.0 @@ -448,8 +448,8 @@ importers: packages/kv-demo-mcp-server: dependencies: '@modelcontextprotocol/sdk': - specifier: ^1.30.0 - version: 1.30.0(zod@4.4.3) + specifier: ^1.31.0 + version: 1.31.0(zod@4.4.3) zod: specifier: ^4.4.3 version: 4.4.3 @@ -467,8 +467,8 @@ importers: packages/mcp-server: dependencies: '@modelcontextprotocol/sdk': - specifier: ^1.30.0 - version: 1.30.0(zod@4.4.3) + specifier: ^1.31.0 + version: 1.31.0(zod@4.4.3) '@paperclipai/shared': specifier: workspace:* version: link:../shared @@ -499,7 +499,7 @@ importers: dependencies: '@agentclientprotocol/claude-agent-acp': specifier: 0.73.0 - version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.30.0(zod@4.6.5)) + version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)) '@agentclientprotocol/codex-acp': specifier: 1.6.2 version: 1.6.2(patch_hash=ilkykmzaax6atpgtip6bxikv6i) @@ -912,8 +912,8 @@ importers: specifier: 1.14.5 version: 1.14.5 '@modelcontextprotocol/sdk': - specifier: ^1.30.0 - version: 1.30.0(zod@4.4.3) + specifier: ^1.31.0 + version: 1.31.0(zod@4.4.3) '@openrouter/ai-sdk-provider': specifier: 3.1.0 version: 3.1.0(ai@7.0.130(zod@4.4.3))(zod@4.4.3) @@ -973,7 +973,7 @@ importers: version: 2.1.0(@grpc/grpc-js@1.14.5)(nice-grpc-common@2.0.4)(nice-grpc@2.1.17) '@vercel/connect': specifier: 2.3.3 - version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.30.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))) + version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))) acorn: specifier: 8.18.0 version: 8.18.0 @@ -2662,8 +2662,8 @@ packages: cpu: [x64] os: [win32] - '@hono/node-server@2.1.0': - resolution: {integrity: sha512-XovyyCCnBzW+zKu+z/zq8hwNs4KOR5rEMAOxo2f40Q5xoOI37IMm6MIg2COOUtUApo0i6850MTBKH2u4QLGIqg==} + '@hono/node-server@2.1.3': + resolution: {integrity: sha512-TA//nWMqPhbfdfneACk6t5a9eqbS9lABEPyKn0/xZTah3H3U2XaVg85rJFl0/Fyit0I552YDHgXGVSf3GwqbUw==} engines: {node: '>=20'} peerDependencies: hono: ^4 @@ -3141,8 +3141,8 @@ packages: resolution: {integrity: sha512-tEdVCj76gEdgvxoYkBH3li+mr8Tfe94pp1cSOUMQNwOnsuKYvretKMZfAnt9adJCgGzGH2MYZeDDtgwmOBufGQ==} engines: {node: '>=20'} - '@modelcontextprotocol/sdk@1.30.0': - resolution: {integrity: sha512-xKd8OIzlqNzcqcNumGAa6g+PW2kjD5vrpcKOnfldAUPP3j7lnqMPwlTXQm8gF+UwH72z0lqaRbjr9hqGz0eITA==} + '@modelcontextprotocol/sdk@1.31.0': + resolution: {integrity: sha512-UvTMgnNlnIBO/22ob2RcVGDlcvOslQs8T59+FTGdA0L27a39fdGF/EDETNtDVK4DZGpwomlsYpRdA8UXcVL/pw==} engines: {node: '>=18'} peerDependencies: '@cfworker/json-schema': ^4.1.1 @@ -6057,8 +6057,8 @@ packages: resolution: {integrity: sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==} engines: {node: '>= 0.6'} - content-type@3.0.0: - resolution: {integrity: sha512-AIi5H6p0xk5uknXcN3/rmhP8jgp69OfSe/JuKiQAFprJ7UGw7mwj7m4XcmDzlrnJDG+cGpphAINGdU3g3g7kDw==} + content-type@3.1.1: + resolution: {integrity: sha512-GW4qUsfFo59d0HbUibDlWv5wPz+vAAcaTWbKIuKCf0JkC7wWkSyf8f13IpXn5JkeMlB2P8iTSSIjwXljorg2vA==} engines: {node: '>=22'} convert-source-map@2.0.0: @@ -6653,8 +6653,8 @@ packages: resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==} engines: {node: '>=12.0.0'} - express-rate-limit@8.6.2: - resolution: {integrity: sha512-YH4ru+eOJxQABscKFfRCy9R7x9QFGdezclVMwwgFFndzS2Xnm0uo6B0ABZsLhcpeptGv2qvuJVWlQr9gQZoC3A==} + express-rate-limit@8.7.1: + resolution: {integrity: sha512-2FXbqkygil3mZiXzEtM0IjE/UBmBNot0J1GEVrJT4JbrvwOSoEtqtO+wHbiMQELWe1eNvHoYRg2dkjOid+6JkA==} engines: {node: '>= 16'} peerDependencies: express: '>= 4.11' @@ -6877,8 +6877,8 @@ packages: help-me@5.0.0: resolution: {integrity: sha512-7xgomUX6ADmcYzFik0HzAxh/73YlKR9bmFzf51CZwR+b6YtzU2m0u49hQCqV6SvlqIqsaxovfwdvbnsw3b/zpg==} - hono@4.13.2: - resolution: {integrity: sha512-JydRilDRkYBQMt9qR9U92mXxmbGqsqSn/IKOrh4e7/gEbn+0zSr8igTu0obwJoNGN4sez28DIql7FBHWydoJpA==} + hono@4.13.13: + resolution: {integrity: sha512-CQ46U0ZkAGmbT/4UxdzzGJpacP2IeKgY4a5/tOI9AABbpOMfK739wfDXmv1usCk+3RkKj1hQy4/fjhiwa2xlrA==} engines: {node: '>=16.9.0'} html-encoding-sniffer@6.0.0: @@ -6950,8 +6950,8 @@ packages: resolution: {integrity: sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==} engines: {node: '>=12'} - ip-address@10.5.0: - resolution: {integrity: sha512-R5SnVLJmgYYvf2F2ZgwSBnelz5G4q5AxIC277GDfUaNbrZKNANcBC7RHqYYePlszf4kBolVkJauG0ZjHHFh55g==} + ip-address@10.7.3: + resolution: {integrity: sha512-A1kdq/tSb5QjvKvAMgIoEvDBIgL7qaqVP/jkvSwYYRZ9iEzvPpopxp2wQfu3SuZRHtpHNxMn8Fs0bS+gf5Xmwg==} engines: {node: '>= 12'} ipaddr.js@1.9.1: @@ -7039,8 +7039,8 @@ packages: jose@6.2.10: resolution: {integrity: sha512-iiW7J9qRFlGxvCOIBDBDxFePQSn7ZMAnrYGhrrOo6siO/MIqwfyilLR27pkfDgUk+raLuzADS8A3S/KLBisc0g==} - jose@6.2.8: - resolution: {integrity: sha512-Bsdjwm3Qsd/P0jR+BHDe3LytDfY7WBq2HmCCLIwuVRHMuEC9ae7/R474GIUdF1NgCyZjzVo/A9DOiOBtXq8ZoQ==} + jose@6.2.12: + resolution: {integrity: sha512-9NiFmJEex0sy2Dk58j2UGBSHgUs2ypF9eZSu4L6vjOX3Dp96Sw1F3uL+H+D1sx02jZZdzUT0HgvCy59CuvXcWw==} joycon@3.1.1: resolution: {integrity: sha512-34wB/Y7MW7bzjKRjUKTa46I2Z7eV62Rkhva+KkopW7Qvv/OSWBqvkSY7vusOPrNuZcUG3tApvdVgNB8POj3SPw==} @@ -9110,10 +9110,10 @@ snapshots: '@adobe/css-tools@4.5.0': {} - '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.30.0(zod@4.6.5))': + '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))': dependencies: '@agentclientprotocol/sdk': 1.4.0(zod@4.4.3) - '@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.30.0(zod@4.6.5))(zod@4.4.3) + '@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))(zod@4.4.3) zod: 4.4.3 transitivePeerDependencies: - '@anthropic-ai/sdk' @@ -9209,10 +9209,10 @@ snapshots: '@anthropic-ai/claude-agent-sdk-win32-x64@0.3.286': optional: true - '@anthropic-ai/claude-agent-sdk@0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.30.0(zod@4.6.5))(zod@4.4.3)': + '@anthropic-ai/claude-agent-sdk@0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))(zod@4.4.3)': dependencies: '@anthropic-ai/sdk': 0.129.0(zod@4.6.5) - '@modelcontextprotocol/sdk': 1.30.0(zod@4.6.5) + '@modelcontextprotocol/sdk': 1.31.0(zod@4.6.5) zod: 4.4.3 optionalDependencies: '@anthropic-ai/claude-agent-sdk-darwin-arm64': 0.3.286 @@ -10558,9 +10558,9 @@ snapshots: '@heif2jpeg/heif2jpeg-win32-x64-msvc@0.1.6': optional: true - '@hono/node-server@2.1.0(hono@4.13.2)': + '@hono/node-server@2.1.3(hono@4.13.13)': dependencies: - hono: 4.13.2 + hono: 4.13.13 '@iconify/types@2.0.0': {} @@ -11160,9 +11160,9 @@ snapshots: - debug - supports-color - '@modelcontextprotocol/sdk@1.30.0(zod@4.4.3)': + '@modelcontextprotocol/sdk@1.31.0(zod@4.4.3)': dependencies: - '@hono/node-server': 2.1.0(hono@4.13.2) + '@hono/node-server': 2.1.3(hono@4.13.13) ajv: 8.20.0 ajv-formats: 3.0.1(ajv@8.20.0) content-type: 1.0.5 @@ -11171,9 +11171,9 @@ snapshots: eventsource: 3.0.7 eventsource-parser: 3.1.1 express: 5.2.1 - express-rate-limit: 8.6.2(express@5.2.1) - hono: 4.13.2 - jose: 6.2.8 + express-rate-limit: 8.7.1(express@5.2.1) + hono: 4.13.13 + jose: 6.2.12 json-schema-typed: 8.0.2 pkce-challenge: 5.0.1 raw-body: 3.0.2 @@ -11182,9 +11182,9 @@ snapshots: transitivePeerDependencies: - supports-color - '@modelcontextprotocol/sdk@1.30.0(zod@4.6.5)': + '@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)': dependencies: - '@hono/node-server': 2.1.0(hono@4.13.2) + '@hono/node-server': 2.1.3(hono@4.13.13) ajv: 8.20.0 ajv-formats: 3.0.1(ajv@8.20.0) content-type: 1.0.5 @@ -11193,9 +11193,9 @@ snapshots: eventsource: 3.0.7 eventsource-parser: 3.1.1 express: 5.2.1 - express-rate-limit: 8.6.2(express@5.2.1) - hono: 4.13.2 - jose: 6.2.8 + express-rate-limit: 8.7.1(express@5.2.1) + hono: 4.13.13 + jose: 6.2.12 json-schema-typed: 8.0.2 pkce-challenge: 5.0.1 raw-body: 3.0.2 @@ -11320,7 +11320,7 @@ snapshots: '@octokit/endpoint': 11.0.5 '@octokit/request-error': 7.1.2 '@octokit/types': 18.0.0 - content-type: 3.0.0 + content-type: 3.1.1 json-with-bigint: 3.5.12 universal-user-agent: 7.0.3 @@ -13387,12 +13387,12 @@ snapshots: dependencies: execa: 5.1.1 - '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.30.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))': + '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))': dependencies: '@vercel/oidc': 3.8.9 optionalDependencies: '@chat-adapter/slack': 4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3) - '@modelcontextprotocol/sdk': 1.30.0(zod@4.4.3) + '@modelcontextprotocol/sdk': 1.31.0(zod@4.4.3) ai: 7.0.130(zod@4.4.3) better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))) @@ -13900,7 +13900,7 @@ snapshots: content-type@1.0.5: {} - content-type@3.0.0: {} + content-type@3.1.1: {} convert-source-map@2.0.0: {} @@ -14482,11 +14482,11 @@ snapshots: expect-type@1.4.0: {} - express-rate-limit@8.6.2(express@5.2.1): + express-rate-limit@8.7.1(express@5.2.1): dependencies: debug: 4.4.3 express: 5.2.1 - ip-address: 10.5.0 + ip-address: 10.7.3 transitivePeerDependencies: - supports-color @@ -14780,7 +14780,7 @@ snapshots: help-me@5.0.0: {} - hono@4.13.2: {} + hono@4.13.13: {} html-encoding-sniffer@6.0.0(@noble/hashes@2.4.0): dependencies: @@ -14846,7 +14846,7 @@ snapshots: internmap@2.0.3: {} - ip-address@10.5.0: {} + ip-address@10.7.3: {} ipaddr.js@1.9.1: {} @@ -14907,7 +14907,7 @@ snapshots: jose@6.2.10: {} - jose@6.2.8: {} + jose@6.2.12: {} joycon@3.1.1: {} diff --git a/server/package.json b/server/package.json index 5d905feaba0..81cd24e5f11 100644 --- a/server/package.json +++ b/server/package.json @@ -56,7 +56,7 @@ "@chat-adapter/telegram": "4.39.0", "@discordjs/ws": "1.2.3", "@grpc/grpc-js": "1.14.5", - "@modelcontextprotocol/sdk": "^1.30.0", + "@modelcontextprotocol/sdk": "^1.31.0", "@opentelemetry/api": "^1.9.0", "@paperclipai/adapter-claude-local": "workspace:*", "@paperclipai/adapter-codex-local": "workspace:*", diff --git a/server/src/__tests__/agent-hire-ai-connections.test.ts b/server/src/__tests__/agent-hire-ai-connections.test.ts index c8fcab742b5..e1be98a92b0 100644 --- a/server/src/__tests__/agent-hire-ai-connections.test.ts +++ b/server/src/__tests__/agent-hire-ai-connections.test.ts @@ -8,7 +8,7 @@ import express from "express"; import request from "supertest"; import { and, eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { activityLog, agents, companies, companyMemberships, createDb, heartbeatRuns, issues, plugins, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db"; +import { activityLog, agents, companies, companyMemberships, createDb, closeRegisteredClients, heartbeatRuns, issues, plugins, principalPermissionGrants, toolConnectionInstalls } from "@paperclipai/db"; import { type AiConnectionBinding, type AiConnectionPoolMember, type PaperclipPluginManifestV1 } from "@paperclipai/shared"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { agentRoutes } from "../routes/agents.js"; @@ -21,8 +21,16 @@ import { secretService } from "../services/secrets.js"; import { aiConnectionRouterService } from "../services/ai-connection-router.js"; import { instanceSettingsService } from "../services/instance-settings.js"; import { toolAccessService } from "../services/tool-access.js"; +import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; + +const captureRunFailure = vi.hoisted(() => vi.fn()); +vi.mock("../sentry.js", async (importOriginal) => ({ + ...await importOriginal(), + captureRunFailure, +})); let database: Awaited>; +const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); let db: ReturnType; let home: string; @@ -31,11 +39,17 @@ beforeAll(async () => { vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "hire-ai"); vi.stubEnv("PAPERCLIP_IN_WORKTREE", "false"); - database = await startEmbeddedPostgresTestDatabase("paperclip-hire-ai-db-"); - db = createDb(database.connectionString); + if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl); + else { + database = await startEmbeddedPostgresTestDatabase("paperclip-hire-ai-db-"); + db = createDb(database.connectionString); + } }, 90_000); afterAll(async () => { + await waitForPendingRunFailureReports(); + if (externalTestDatabaseUrl) await closeRegisteredClients(externalTestDatabaseUrl); + await db?.$client.end(); await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); @@ -488,10 +502,21 @@ describe("hired agents sharing a subscription", () => { expect(run).not.toBeNull(); await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("cancelled"); await heartbeat.drainActiveRunExecutions(); - expect(await heartbeat.getRun(run!.id)).toMatchObject({ errorCode: "ai_connection_busy", resultJson: { executionRecovery: { providerWorkStarted: false } } }); + const cancelled = await heartbeat.getRun(run!.id); + expect(cancelled).toMatchObject({ errorCode: "ai_connection_busy", resultJson: { + executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false }, + cancellation: { + source: "control_plane", expected: true, initiator: { type: "system" }, + reason: "Waiting for shared AI credentials", + recordedAt: cancelled!.finishedAt!.toISOString(), + }, + } }); + await waitForPendingRunFailureReports(); + expect(captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(0); const retries = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, run!.id)); expect(retries).toHaveLength(1); expect(retries[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy" }); + expect(retries[0].resultJson?.cancellation).toBeUndefined(); const [savedIssue] = await db.select().from(issues).where(eq(issues.id, issue.id)); expect(savedIssue.status).not.toBe("blocked"); expect(savedIssue.executionRunId).toBe(retries[0].id); diff --git a/server/src/__tests__/ai-connection-router.test.ts b/server/src/__tests__/ai-connection-router.test.ts index b2e13a92e8a..1a1f3f29bd2 100644 --- a/server/src/__tests__/ai-connection-router.test.ts +++ b/server/src/__tests__/ai-connection-router.test.ts @@ -4,7 +4,7 @@ import { mkdtemp, rm } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { and, eq } from "drizzle-orm"; -import { createDb, companies, agents, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues } from "@paperclipai/db"; +import { createDb, closeRegisteredClients, companies, agents, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres"; import type { AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, PaperclipPluginManifestV1 } from "@paperclipai/shared"; import { aiConnectionRouterService, AiConnectionPoolExhausted, poolMemberRuntimeConfig, applyAiConnectionRouterTaskSettings } from "../services/ai-connection-router.js"; @@ -24,8 +24,16 @@ import { heartbeatService, buildEffectiveRunSessionConfigMetadata, resolveTaskSe import { connectionIntentService } from "../services/connection-intents.js"; import { legacyExecutionNeedsReconciliation } from "../services/legacy-execution-recovery.js"; import { getServerAdapter, registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js"; +import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; + +const captureRunFailure = vi.hoisted(() => vi.fn()); +vi.mock("../sentry.js", async (importOriginal) => ({ + ...await importOriginal(), + captureRunFailure, +})); let database: Awaited>; +const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); let db: ReturnType; let home: string; const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID(), secondAgentId = randomUUID(), pluginId = randomUUID(); @@ -46,7 +54,8 @@ const makePool = async (config = {}) => service().save(pluginKey, { companyId, c beforeAll(async () => { home = await mkdtemp(path.join(os.tmpdir(), "paperclip-router-tests-")); vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "router-fixture"); - database = await startEmbeddedPostgresTestDatabase("paperclip-router-db-"); db = createDb(database.connectionString); + if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl); + else { database = await startEmbeddedPostgresTestDatabase("paperclip-router-db-"); db = createDb(database.connectionString); } await db.insert(companies).values([{ id: companyId, name: "Router fixture", issuePrefix: "RTF" }, { id: otherCompanyId, name: "Other", issuePrefix: "RTO" }]); await db.insert(agents).values([agentId, secondAgentId].map(id => ({ id, companyId, name: "Router agent", adapterType: "paperclip_runner" }))); await db.insert(companyMemberships).values(["alice", "bob"].map(principalId => ({ companyId, principalId, principalType: "user", status: "active", membershipRole: "member" }))); @@ -59,7 +68,12 @@ beforeAll(async () => { { id: randomUUID(), binding: { mode: "delegated", provider: "anthropic", method: "subscription", ...claude }, profile: { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-5" } }, ]; }, 90000); -afterAll(async () => { await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); }); +afterAll(async () => { + await waitForPendingRunFailureReports(); + if (externalTestDatabaseUrl) await closeRegisteredClients(externalTestDatabaseUrl); + await db?.$client.end(); + await database?.cleanup(); vi.unstubAllEnvs(); if (home) await rm(home, { recursive: true, force: true }); +}); describe("durable, authorized connection routing", () => { it("defaults off and does not expose selectable pools before opt-in", async () => { @@ -321,16 +335,48 @@ describe("durable, authorized connection routing", () => { expect(run).not.toBeNull(); await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("cancelled"); const cancelled = await heartbeat.getRun(run!.id); expect(cancelled?.errorCode).toBe("ai_connection_pool_exhausted"); + expect(cancelled?.resultJson).toMatchObject({ + executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false }, + cancellation: { + source: "control_plane", expected: true, initiator: { type: "system" }, + reason: "Waiting for an AI connection pool account", + recordedAt: cancelled!.finishedAt!.toISOString(), + }, + }); expect(legacyExecutionNeedsReconciliation(cancelled!)).toBe(false); expect(legacyExecutionNeedsReconciliation({ ...cancelled!, resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: true } } })).toBe(true); await expect.poll(async () => (await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.agentId, agent!.id), eq(heartbeatRuns.scheduledRetryReason, "ai_connection_pool_wait")))).length, { timeout: 5000 }).toBe(1); const [retry] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.agentId, agent!.id), eq(heartbeatRuns.scheduledRetryReason, "ai_connection_pool_wait"))); + await waitForPendingRunFailureReports(); + expect(captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(0); + expect(retry?.resultJson?.cancellation).toBeUndefined(); expect(retry?.contextSnapshot?.aiRouterTaskKey).toBe(issue!.id); expect(retry?.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 }); expect(execute).not.toHaveBeenCalled(); const [pin] = await db.select().from(aiConnectionTaskPins).where(and(eq(aiConnectionTaskPins.poolId, pool.id), eq(aiConnectionTaskPins.agentId, agent!.id))); expect(pin?.selection).toEqual(pinned); const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(1); await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.agentId, agent!.id)); } finally { fetch.mockRestore(); proposal.mockReset(); proposal.mockImplementation(defaultProposal); unregisterServerAdapter("claude_local"); await heartbeat.drainActiveRunExecutions(); } }, 30_000); + it("keeps unexpected pool policy failures reportable before provider work", async () => { + await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true, enableWorktreeRunExecution: true }); + const pool = await makePool({ members: [members[1]!] }); + const [agent] = await db.insert(agents).values({ companyId, name: "Unexpected router failure", adapterType: "claude_local", adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false } } }).returning(); + const [issue] = await db.insert(issues).values({ companyId, title: "Router failure fixture", status: "todo", assigneeAgentId: agent!.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning(); + const execute = vi.fn(); registerServerAdapter({ ...getServerAdapter("claude_local"), execute }); + proposal.mockRejectedValueOnce(new Error("Synthetic unexpected router policy failure")); + const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {} }); + try { + const run = await heartbeat.wakeup(agent!.id, { source: "assignment", reason: "issue_assigned", payload: { issueId: issue!.id }, contextSnapshot: { issueId: issue!.id, wakeReason: "issue_assigned", responsibleUserId: "alice" }, triggerDetail: "system" }); + expect(run).not.toBeNull(); + await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("failed"); + const failed = await heartbeat.getRun(run!.id); + expect(failed?.errorCode).toBe("configuration_incomplete"); + expect(failed?.resultJson).toMatchObject({ configurationIncomplete: { reason: "ai_connection_unavailable" } }); + expect(failed?.resultJson?.cancellation).toBeUndefined(); + expect(execute).not.toHaveBeenCalled(); + await expect.poll(() => captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(1); + expect(captureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ runId: run!.id, runStatus: "failed", errorCode: "configuration_incomplete" })); + } finally { proposal.mockReset(); proposal.mockImplementation(defaultProposal); unregisterServerAdapter("claude_local"); await heartbeat.drainActiveRunExecutions(); } + }, 30_000); it("repairs the failed pool account without replacing the pool or adopting the configured provider", async () => { const pool = await makePool({ members: [members[1]!] }); const [agent] = await db.insert(agents).values({ companyId, name: "Pool repair", adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: "gpt-5.6-sol" }, runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id } } }).returning(); diff --git a/server/src/__tests__/error-handler.test.ts b/server/src/__tests__/error-handler.test.ts index 52bf838559f..934cf7aa7c0 100644 --- a/server/src/__tests__/error-handler.test.ts +++ b/server/src/__tests__/error-handler.test.ts @@ -2,6 +2,7 @@ import type { NextFunction, Request, Response } from "express"; import { beforeEach, describe, expect, it, vi } from "vitest"; import { HttpError } from "../errors.js"; import { errorHandler } from "../middleware/error-handler.js"; +import { mcpDiscoveryHttpFailure, retainMcpConnectionFailure } from "../services/mcp-connection-failure.js"; const recordResponsibleUserDenialOnActiveRunMock = vi.hoisted(() => vi.fn()); const captureExceptionMock = vi.hoisted(() => vi.fn()); @@ -121,6 +122,35 @@ describe("errorHandler", () => { expect(res.__errorContext?.error?.message).toBe("db exploded"); }); + it("keeps HTTP failures and existing telemetry for proven external MCP outages while skipping Sentry", () => { + const req = makeReq(); + const res = makeRes() as any; + const original = mcpDiscoveryHttpFailure(new Response(null, { status: 503 }), "Remote app returned HTTP 503"); + const err = retainMcpConnectionFailure(original, new HttpError(502, original.message)); + + errorHandler(err, req, res, vi.fn()); + + expect(res.status).toHaveBeenCalledWith(502); + expect(res.json).toHaveBeenCalledWith({ error: original.message }); + expect(res.err).toBe(err); + expect(res.__errorContext.error.message).toBe(original.message); + expect(telemetryMocks.trackErrorHandlerCrash).toHaveBeenCalledOnce(); + expect(captureExceptionMock).not.toHaveBeenCalled(); + }); + + it("reports copied connection markers and unrelated database failures", () => { + for (const err of [ + Object.assign(new HttpError(502, "Remote app returned HTTP 503", { status: 503 }), { + connectionFailure: { schemaVersion: 1, provider: "mcp_http", operation: "discover_tools", reason: "remote_unavailable" }, + }), + new HttpError(500, "database unavailable"), + ]) { + errorHandler(err, makeReq(), makeRes(), vi.fn()); + } + expect(captureExceptionMock).toHaveBeenCalledTimes(2); + expect(telemetryMocks.trackErrorHandlerCrash).toHaveBeenCalledTimes(2); + }); + it("sanitizes chat setup errors before logs and crash reporting", () => { const req = { ...makeReq(), diff --git a/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts b/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts index 279fb75cc4f..74c1b602e79 100644 --- a/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts +++ b/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts @@ -8,6 +8,7 @@ import { ensureManagedProjectWorkspace, prepareProjectRepositoryWorkspaces } fro import { buildGitAuthInvocation, GIT_CREDENTIAL_TOKEN_ENV_KEY } from "../services/git-credentials.ts"; import { sanitizeRuntimeServiceBaseEnv } from "../services/workspace-runtime.ts"; import { resolveManagedProjectWorkspaceDir } from "../home-paths.ts"; +import { readGitConnectionFailure } from "../services/git-connection-failure.js"; const execFile = promisify(execFileCallback); @@ -38,6 +39,79 @@ async function createLocalSourceRepo() { } describe("ensureManagedProjectWorkspace clone credentials", () => { + it.each([ + ["authentication_failed", "fatal: Authentication failed for 'https://example.test/team/repo.git/'"], + ["dns_failure", "ssh: Could not resolve hostname repository-alias.example.test: Name or service not known\nfatal: Could not read from remote repository."], + [null, "fatal: Authentication failed for 'https://example.test/team/repo.git/'\nfatal: bad config line 1 in file /tmp/config"], + [null, "fatal: corrupt loose object"], + ] as const)("preserves only proven outbound clone classification through the actionable wrapper: %s", async (reason, stderr) => { + const bin = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-fake-clone-")); + await fs.writeFile(path.join(bin, "git"), `#!/bin/sh +printf '%s\\n' "$FIXTURE_CLONE_STDERR" >&2 +exit 128 +`, { mode: 0o755 }); + try { + const failure = await ensureManagedProjectWorkspace({ + companyId: "remote-clone-fixture", projectId: path.basename(bin), + repoUrl: "https://fixture-user:fixture-password@example.test/team/repo.git", + resolveGitAuth: async () => ({ + configArgs: [], env: { PATH: bin, FIXTURE_CLONE_STDERR: stderr }, + source: "company_secret", secretName: "GITHUB_TOKEN", + }), + }).catch((error: unknown) => error); + expect(failure).toBeInstanceOf(Error); + expect((failure as Error).message).toContain("Failed to prepare managed checkout"); + expect((failure as Error).message).not.toContain("fixture-password"); + expect(readGitConnectionFailure(failure)).toEqual(reason ? { + schemaVersion: 1, provider: "git", operation: "clone", reason, + } : null); + const homeEntries = await fs.readdir(path.dirname(resolveManagedProjectWorkspaceDir({ + companyId: "remote-clone-fixture", projectId: path.basename(bin), repoName: "repo", + }))); + expect(homeEntries.some((name) => name.includes(".clone-"))).toBe(false); + } finally { + await fs.rm(bin, { recursive: true, force: true }); + } + }); + + it("does not classify a credential-provider exception that resembles a Git failure", async () => { + const failure = Object.assign(new Error("fatal: Authentication failed for 'https://example.test/repo.git'"), { + code: 128, stderr: "fatal: Authentication failed for 'https://example.test/repo.git'", + }); + const caught = await ensureManagedProjectWorkspace({ + companyId: "credential-provider-fixture", projectId: "project", repoUrl: "https://example.test/repo.git", + resolveGitAuth: async () => { throw failure; }, + }).catch((error: unknown) => error); + expect(caught).toBe(failure); + expect(readGitConnectionFailure(caught)).toBeNull(); + }); + + it("does not transfer clone classification to a later local checkout failure", async () => { + const bin = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-fake-checkout-")); + const previousPath = process.env.PATH; + await fs.writeFile(path.join(bin, "git"), `#!/bin/sh +if [ "$1" = clone ]; then + for argument in "$@"; do destination="$argument"; done + /bin/mkdir -p "$destination/.git" + exit 0 +fi +printf '%s\\n' "fatal: Authentication failed for 'https://example.test/repo.git'" >&2 +exit 128 +`, { mode: 0o755 }); + try { + process.env.PATH = bin; + const failure = await ensureManagedProjectWorkspace({ + companyId: "checkout-failure-fixture", projectId: "project", repoUrl: "https://example.test/repo.git", repoRef: "missing-branch", + }).catch((error: unknown) => error); + expect(failure).toBeInstanceOf(Error); + expect((failure as Error).message).toContain("Authentication failed"); + expect(readGitConnectionFailure(failure)).toBeNull(); + } finally { + if (previousPath === undefined) delete process.env.PATH; else process.env.PATH = previousPath; + await fs.rm(bin, { recursive: true, force: true }); + } + }); + it("materializes every repository-only project row inside the task workspace and reuses local edits", async () => { const first = await createLocalSourceRepo(); const second = await createLocalSourceRepo(); diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index 386f2a30ac3..87c14e17b66 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -12,7 +12,7 @@ import { recordLegacyWorkspaceRestoreFailure, terminalizeLegacyExecution } from import { issueService } from "../services/issues.js"; import { getExecutionBlocker } from "../services/execution-blocker.js"; import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; -import { spawn, type ChildProcess } from "node:child_process"; +import { execFileSync, spawn, type ChildProcess } from "node:child_process"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -5591,6 +5591,179 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { } }); + it.each([ + { name: "HTTPS authentication", failures: ["auth"], report: false }, + { name: "SSH alias DNS", failures: ["dns"], report: false }, + { name: "mixed recognized and unknown candidates", failures: ["auth", "unknown"], report: true }, + { name: "mixed local and authentication diagnostics", failures: ["mixed"], report: true }, + { name: "unknown clone failure", failures: ["unknown"], report: true }, + ])("preserves failed workspace recovery while classifying managed Git connections: $name", async ({ failures, report }) => { + const { companyId, agentId, runId, issueId } = await seedQueuedIssueRunFixture(); + const home = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-connection-lifecycle-")); + const bin = path.join(home, "bin"); + const callsPath = path.join(home, "git-calls.jsonl"); + const projectId = randomUUID(); + const workspaceIds = failures.map(() => randomUUID()); + const remotes = failures.map((failure, index) => failure === "dns" + ? `git@clone-alias.example.test:team/repository-${index}.git` + : `https://clone.example.test/team/repository-${index}.git`); + const diagnostics = failures.map((failure) => failure === "dns" + ? "ssh: Could not resolve hostname clone-alias.example.test: Name or service not known\nfatal: Could not read from remote repository." + : failure === "unknown" ? "fatal: unexpected internal clone failure" + : "fatal: Authentication failed for 'https://clone.example.test/'" + + (failure === "mixed" ? "\nfatal: No space left on device" : "")); + await fs.mkdir(bin); + // A real child process supplies execFile's numeric exit/stderr evidence. + // Every command is intercepted, so no fixture can reach a remote Git host. + await fs.writeFile(path.join(bin, "git"), `#!${process.execPath}\n` + + `const fs = require('node:fs');\n` + + `const args = process.argv.slice(2);\n` + + `fs.appendFileSync(${JSON.stringify(callsPath)}, JSON.stringify(args) + '\\n');\n` + + `const remotes = ${JSON.stringify(remotes)};\n` + + `const diagnostics = ${JSON.stringify(diagnostics)};\n` + + `const index = remotes.indexOf(args.at(-2));\n` + + `if (!args.includes('clone') || index < 0) { process.stderr.write('Unexpected fixture Git command\\n'); process.exit(2); }\n` + + `process.stderr.write(diagnostics[index] + '\\n'); process.exitCode = 128;\n`, { mode: 0o755 }); + const heartbeat = heartbeatService(db); + const previousSettings = await instanceSettingsService(db).getExperimental(); + const previousPath = process.env.PATH; + const previousHome = process.env.PAPERCLIP_HOME; + try { + process.env.PATH = `${bin}${path.delimiter}${previousPath ?? ""}`; + process.env.PAPERCLIP_HOME = home; + await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: true }); + await db.insert(projects).values({ id: projectId, companyId, name: "Git connection fixture", status: "in_progress" }); + await db.insert(projectWorkspaces).values(workspaceIds.map((id, index) => ({ + id, companyId, projectId, name: `Repository ${index}`, sourceType: "git_repo", + repoUrl: remotes[index], cwd: null, isPrimary: index === 0, + }))); + await db.update(issues).set({ + projectId, projectWorkspaceId: workspaceIds[0], executionWorkspaceSettings: { mode: "isolated_workspace" }, + }).where(eq(issues.id, issueId)); + await db.update(agents).set({ adapterConfig: { workspaceStrategy: { type: "git_worktree" } } }).where(eq(agents.id, agentId)); + + await heartbeat.resumeQueuedRuns(); + await waitForRunToSettle(heartbeat, runId, 5_000); + await heartbeat.waitForRunExecutionDrain(runId); + await waitForPendingRunFailureReports(); + + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(run).toMatchObject({ + status: "failed", errorCode: "workspace_validation_failed", processStartedAt: null, + resultJson: { + workspaceValidation: { reason: "git_worktree_base_materialization_failed" }, + executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, + }, + }); + expect(run.error).toContain("checkout could not be prepared"); + const validation = run.resultJson!.workspaceValidation as Record; + const materializations = validation.materializationFailures as Array>; + expect(materializations).toHaveLength(failures.length); + expect(materializations.map((failure) => failure.connectionFailure)).toEqual(failures.map((failure) => + failure === "auth" || failure === "dns" ? { + schemaVersion: 1, provider: "git", operation: "clone", + reason: failure === "auth" ? "authentication_failed" : "dns_failure", + } : undefined)); + expect(run.resultJson!.connectionFailure).toEqual(report ? undefined : materializations[0].connectionFailure); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + expect(issue).toMatchObject({ status: "blocked", executionRunId: null, projectId, projectWorkspaceId: workspaceIds[0] }); + const [action] = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + )); + expect(action).toMatchObject({ status: "active", kind: "workspace_validation", ownerType: "board" }); + expect(action.nextAction).toContain("workspace"); + const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, issueId)); + expect(comments.some((comment) => comment.body.includes("workspace failed validation"))).toBe(true); + expect(mockCaptureRunFailure.mock.calls.filter(([event]) => event.runId === runId)).toHaveLength(report ? 1 : 0); + const calls = (await fs.readFile(callsPath, "utf8")).trim().split("\n").map((line) => JSON.parse(line) as string[]); + expect(calls).toHaveLength(failures.length); + expect(calls.every((args) => args.includes("clone"))).toBe(true); + } finally { + await heartbeat.waitForRunExecutionDrain(runId); + if (previousPath === undefined) delete process.env.PATH; else process.env.PATH = previousPath; + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: previousSettings.enableIsolatedWorkspaces }); + await fs.rm(home, { recursive: true, force: true }); + } + }); + + it("keeps a direct additional-repository connection failure visible without reporting it to Sentry", async () => { + const { companyId, agentId, runId, issueId } = await seedQueuedIssueRunFixture(); + const home = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-additional-git-connection-")); + const cwd = path.join(home, "primary"); + const bin = path.join(home, "bin"); + const cloneCalls = path.join(home, "clone-calls.jsonl"); + const projectId = randomUUID(); + const workspaceId = randomUUID(); + const repoUrl = "git@clone-alias.example.test:team/additional.git"; + const previousPath = process.env.PATH; + const previousHome = process.env.PAPERCLIP_HOME; + const heartbeat = heartbeatService(db); + await fs.mkdir(cwd); + await fs.mkdir(bin); + execFileSync("git", ["init", "-b", "main"], { cwd, stdio: "ignore" }); + await fs.writeFile(path.join(cwd, "README.md"), "Keep this local repository intact.\n"); + execFileSync("git", ["add", "README.md"], { cwd, stdio: "ignore" }); + execFileSync("git", ["-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-m", "Fixture"], { cwd, stdio: "ignore" }); + await fs.writeFile(path.join(bin, "git"), `#!${process.execPath}\n` + + `const fs = require('node:fs'); const { spawnSync } = require('node:child_process');\n` + + `const args = process.argv.slice(2);\n` + + `if (args.includes('clone')) {\n` + + ` fs.appendFileSync(${JSON.stringify(cloneCalls)}, JSON.stringify(args) + '\\n');\n` + + ` if (args.at(-2) !== ${JSON.stringify(repoUrl)}) process.exit(2);\n` + + ` process.stderr.write('ssh: Could not resolve hostname clone-alias.example.test: Name or service not known\\nfatal: Could not read from remote repository.\\n'); process.exit(128);\n` + + `}\n` + + `if (args.some(arg => ['fetch', 'pull', 'push', 'ls-remote', 'submodule'].includes(arg))) process.exit(2);\n` + + `const result = spawnSync('git', args, { stdio: 'inherit', env: { ...process.env, PATH: ${JSON.stringify(previousPath)} } });\n` + + `process.exit(result.status ?? 2);\n`, { mode: 0o755 }); + try { + process.env.PATH = `${bin}${path.delimiter}${previousPath ?? ""}`; + process.env.PAPERCLIP_HOME = home; + await db.insert(projects).values({ id: projectId, companyId, name: "Additional repository fixture", status: "in_progress" }); + await db.insert(projectWorkspaces).values([ + { id: workspaceId, companyId, projectId, name: "Primary", sourceType: "local_path", cwd, isPrimary: true }, + { id: randomUUID(), companyId, projectId, name: "Additional", sourceType: "git_repo", repoUrl, cwd: null, isPrimary: false }, + ]); + await db.update(issues).set({ projectId, projectWorkspaceId: workspaceId, + executionWorkspaceSettings: { mode: "shared_workspace" }, + }).where(eq(issues.id, issueId)); + await db.update(agents).set({ adapterConfig: { cwd } }).where(eq(agents.id, agentId)); + + await heartbeat.resumeQueuedRuns(); + await waitForRunToSettle(heartbeat, runId, 5_000); + await heartbeat.waitForRunExecutionDrain(runId); + await waitForPendingRunFailureReports(); + + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(run).toMatchObject({ status: "failed", errorCode: "setup_failed", executionStage: "preparing", processStartedAt: null, + resultJson: { + connectionFailure: { schemaVersion: 1, provider: "git", operation: "clone", reason: "dns_failure" }, + executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, + }, + }); + expect(run.resultJson?.workspaceValidation).toBeUndefined(); + expect(run.error).toContain("Could not resolve hostname"); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + expect(issue).toMatchObject({ status: "blocked", executionRunId: null, projectId, projectWorkspaceId: workspaceId }); + const [action] = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + )); + expect(action).toMatchObject({ status: "active", ownerType: "board" }); + expect(mockCaptureRunFailure.mock.calls.filter(([event]) => event.runId === runId)).toHaveLength(0); + const calls = (await fs.readFile(cloneCalls, "utf8")).trim().split("\n").map((line) => JSON.parse(line) as string[]); + expect(calls).toHaveLength(1); + expect(calls[0].at(-2)).toBe(repoUrl); + expect(await fs.readFile(path.join(cwd, "README.md"), "utf8")).toBe("Keep this local repository intact.\n"); + } finally { + await heartbeat.waitForRunExecutionDrain(runId); + if (previousPath === undefined) delete process.env.PATH; else process.env.PATH = previousPath; + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + await fs.rm(home, { recursive: true, force: true }); + } + }); + it.each(["throw", "result"])("redacts opaque environment-bound credentials from Sentry diagnostics: %s", async (mode) => { const { companyId, agentId, runId } = await seedQueuedIssueRunFixture(); const svc = secretService(db); diff --git a/server/src/__tests__/heartbeat-workspace-busy.test.ts b/server/src/__tests__/heartbeat-workspace-busy.test.ts index 0d2fc3f1be1..af0d1f63c94 100644 --- a/server/src/__tests__/heartbeat-workspace-busy.test.ts +++ b/server/src/__tests__/heartbeat-workspace-busy.test.ts @@ -14,6 +14,7 @@ import { companies, companySkills, createDb, + closeRegisteredClients, environments, environmentLeases, executionWorkspaces, @@ -46,6 +47,13 @@ import { heartbeatService, } from "../services/heartbeat.ts"; import { instanceSettingsService } from "../services/instance-settings.ts"; +import { reportRunFailure, waitForPendingRunFailureReports } from "../services/run-failure-report.js"; + +const captureRunFailure = vi.hoisted(() => vi.fn()); +vi.mock("../sentry.js", async (importOriginal) => ({ + ...await importOriginal(), + captureRunFailure, +})); // Exercise the real SSH lease and heartbeat paths without connecting to a host. vi.mock("@paperclipai/adapter-utils/ssh", async (importOriginal) => ({ @@ -55,7 +63,8 @@ vi.mock("@paperclipai/adapter-utils/ssh", async (importOriginal) => ({ }), })); -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); +const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; const WORKSPACE_BUSY_TEST_ADAPTER = "workspace_busy_test"; @@ -91,10 +100,14 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { let workspaceCwd!: string; const executedRunIds: string[] = []; const executedInputs = new Map(); + const failExecutionRunIds = new Set(); beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-workspace-busy-"); - db = createDb(tempDb.connectionString); + if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl); + else { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-workspace-busy-"); + db = createDb(tempDb.connectionString); + } heartbeat = heartbeatService(db); workspaceCwd = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-busy-")); registerServerAdapter({ @@ -102,6 +115,15 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { execute: async (input) => { executedRunIds.push(input.runId); executedInputs.set(input.runId, input); + if (failExecutionRunIds.has(input.runId)) { + return { + exitCode: 1, + signal: null, + timedOut: false, + errorMessage: "Synthetic unexpected adapter failure after workspace wait", + resultJson: {}, + }; + } return { exitCode: 0, signal: null, @@ -127,15 +149,20 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { .set({ status: "cancelled", finishedAt: new Date() }) .where(eq(heartbeatRuns.status, "running")); await drainHeartbeatRunsToQuiescence(db, heartbeat); + await waitForPendingRunFailureReports(); await cleanupFixture(); executedRunIds.length = 0; executedInputs.clear(); + failExecutionRunIds.clear(); + captureRunFailure.mockClear(); await instanceSettingsService(db).updateGeneral({ executionMode: "any" }); }); afterAll(async () => { unregisterServerAdapter(WORKSPACE_BUSY_TEST_ADAPTER); if (workspaceCwd) await fs.rm(workspaceCwd, { recursive: true, force: true }); + if (externalTestDatabaseUrl) await closeRegisteredClients(externalTestDatabaseUrl); + await db.$client.end(); await tempDb?.cleanup(); }); @@ -547,6 +574,16 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { holderIssueId: fixture.holderIssueId, deferralAttempt: 0, }); + expect(finishedRun?.resultJson).toMatchObject({ + executionRecovery: { kind: "workspace_wait", providerWorkStarted: false }, + cancellation: { + source: "control_plane", + expected: true, + initiator: { type: "system" }, + reason: "Waiting for the shared project workspace", + recordedAt: finishedRun!.finishedAt!.toISOString(), + }, + }); // The deferred run's adapter never executed — the whole point of the gate. expect(executedRunIds).not.toContain(run!.id); @@ -557,6 +594,8 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { scheduledRetryAttempt: 1, scheduledRetryReason: WORKSPACE_BUSY_RETRY_REASON, }); + await waitForPendingRunFailureReports(); + expect(captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(0); expect((retryRun?.contextSnapshot as Record | null)?.wakeReason).toBe( WORKSPACE_BUSY_RETRY_WAKE_REASON, ); @@ -603,7 +642,7 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { } }); - it("executes the scheduled retry once the holder has finished", async () => { + it.each([false, true])("executes the scheduled retry without inheriting expected cancellation (adapter fails: %s)", async (adapterFails) => { const fixture = await seedWorkspaceFixture(); const run = await heartbeat.invoke( @@ -618,6 +657,8 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { const retryRun = await waitForRetryRun(run!.id); expect(retryRun?.status).toBe("scheduled_retry"); + expect(retryRun?.resultJson?.cancellation).toBeUndefined(); + if (adapterFails) failExecutionRunIds.add(retryRun!.id); // Holder finishes; the due retry promotes, queues, and executes. await db @@ -631,8 +672,50 @@ describeEmbeddedPostgres("shared-workspace run serialization", () => { await heartbeat.resumeQueuedRuns(); const finishedRetry = await waitForRunToLeaveActiveStates(retryRun!.id); - expect(finishedRetry?.status).toBe("succeeded"); + expect(finishedRetry?.status).toBe(adapterFails ? "failed" : "succeeded"); expect(executedRunIds).toContain(retryRun!.id); + expect(finishedRetry?.resultJson?.cancellation).toBeUndefined(); + if (adapterFails) { + await expect.poll(() => captureRunFailure.mock.calls.filter(([report]) => report.runId === retryRun!.id)).toHaveLength(1); + expect(captureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ + runId: retryRun!.id, + runStatus: "failed", + exitCode: 1, + })); + } + await waitForPendingRunFailureReports(); + expect(captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(0); + }); + + it("does not hide ambiguous, malformed, provider, or transport cancellations behind workspace_busy", async () => { + const fixture = await seedWorkspaceFixture(); + const run = await heartbeat.invoke(fixture.agentId, "assignment", { + issueId: fixture.issueId, wakeReason: "issue_assigned", + }, "system"); + expect(run).not.toBeNull(); + await waitForRunToLeaveActiveStates(run!.id); + await waitForRetryRun(run!.id); + const [deferred] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)); + expect(deferred?.status).toBe("cancelled"); + const receipt = deferred!.resultJson!.cancellation as Record; + for (const cancellation of [ + undefined, + { ...receipt, expected: "true" }, + { ...receipt, expected: false, source: "provider", initiator: { type: "provider" } }, + { ...receipt, expected: false, source: "transport" }, + ]) { + captureRunFailure.mockClear(); + const unexpected = { + ...deferred!, id: randomUUID(), + resultJson: { ...deferred!.resultJson, cancellation }, + }; + await reportRunFailure(db, unexpected); + expect(captureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ + runId: unexpected.id, + runStatus: "cancelled", + errorCode: WORKSPACE_BUSY_ERROR_CODE, + })); + } }); it("preserves a legacy accepted mention run through workspace retry", async () => { diff --git a/server/src/__tests__/heartbeat-workspace-session.test.ts b/server/src/__tests__/heartbeat-workspace-session.test.ts index e57c34ff195..906a7d23c6e 100644 --- a/server/src/__tests__/heartbeat-workspace-session.test.ts +++ b/server/src/__tests__/heartbeat-workspace-session.test.ts @@ -462,6 +462,29 @@ describe("assertGitSensitiveAdapterWorkspaceValid", () => { }); describe("assertGitWorktreeBaseWorkspaceReady", () => { + it.each(["all_external", "mixed", "malformed"])("keeps materialization aggregation fail-closed: %s", async (kind) => { + const first = { schemaVersion: 1, provider: "git", operation: "clone", reason: "authentication_failed" } as const; + const second = kind === "all_external" ? { ...first, reason: "dns_failure" as const } : + kind === "malformed" ? { ...first, reason: "unknown" } : undefined; + const failures = [first, second].map((connectionFailure, index) => ({ + projectWorkspaceId: `workspace-${index}`, repoUrl: "https://example.test/team/repo.git", + error: "Managed checkout failed", ...(connectionFailure ? { connectionFailure } : {}), + })); + const error = await assertGitWorktreeBaseWorkspaceReady({ + requestedExecutionWorkspaceMode: "isolated_workspace", + config: { workspaceStrategy: { type: "git_worktree" } }, + issue: { id: "issue-1", identifier: "TEST-1", projectId: "project-1", projectWorkspaceId: "workspace-0" }, + base: { baseCwd: "/tmp/unused-fallback", source: "project_primary", projectId: "project-1", workspaceId: "workspace-0", repoUrl: "https://example.test/team/repo.git", repoRef: null }, + anchor: { baseCwdFallback: true, materializationFailures: failures as never }, + }).catch((error: unknown) => error); + expect(error).toMatchObject({ + code: "workspace_validation_failed", + resultJson: { workspaceValidation: { reason: "git_worktree_base_materialization_failed", materializationFailures: failures } }, + }); + expect((error as { resultJson: Record }).resultJson.connectionFailure) + .toEqual(kind === "all_external" ? first : undefined); + }); + it("rejects projectless isolated git worktrees that resolved to agent_home", async () => { const fallbackCwd = resolveDefaultAgentWorkspaceDir("agent-1"); diff --git a/server/src/__tests__/issue-execution-policy-persistence.test.ts b/server/src/__tests__/issue-execution-policy-persistence.test.ts new file mode 100644 index 00000000000..743c248efc1 --- /dev/null +++ b/server/src/__tests__/issue-execution-policy-persistence.test.ts @@ -0,0 +1,45 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { companies, createDb, issues } from "@paperclipai/db"; +import { buildExecutionPolicy } from "../../../ui/src/lib/issue-execution-policy.ts"; +import { normalizeIssueExecutionPolicy } from "../services/issue-execution-policy.js"; +import { issueService } from "../services/issues.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("edited execution policy persistence", () => { + let temporary: Awaited>; + let db: ReturnType; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-policy-persistence-"); + db = createDb(temporary.connectionString); + }, 30_000); + afterAll(async () => { await temporary?.cleanup(); }); + + it.each(["reviewer", "monitor"] as const)("retains the independent review limit after removing the last %s", async (removed) => { + const companyId = randomUUID(); + const issueId = randomUUID(); + const original = normalizeIssueExecutionPolicy({ + maxReviewRounds: 4, + ...(removed === "monitor" + ? { monitor: { nextCheckAt: "2099-01-01T00:00:00.000Z" } } + : { stages: [{ type: "review", participants: [{ type: "user", userId: "board-reviewer" }] }] }), + }); + await db.insert(companies).values({ id: companyId, name: "Policy fixture", issuePrefix: `P${companyId.slice(0, 6)}` }); + await db.insert(issues).values({ id: issueId, companyId, title: "Review limit fixture", executionPolicy: original }); + const edited = buildExecutionPolicy({ existingPolicy: original, reviewerValues: [], approverValues: [] })!; + // The monitor editor removes this key after building the unchanged stages. + const { monitor: _removedMonitor, ...withoutMonitor } = edited; + const normalized = normalizeIssueExecutionPolicy(removed === "monitor" ? withoutMonitor : edited); + const service = issueService(db); + await service.update(issueId, { executionPolicy: normalized, companyGuard: companyId }); + const saved = await service.getById(issueId); + expect(saved?.executionPolicy).toEqual({ mode: "normal", commentRequired: true, stages: [], maxReviewRounds: 4 }); + + const restored = normalizeIssueExecutionPolicy(buildExecutionPolicy({ + existingPolicy: saved?.executionPolicy, + reviewerValues: ["user:board-reviewer"], approverValues: [], + })); + expect(restored).toMatchObject({ maxReviewRounds: 4, stages: [{ type: "review" }] }); + }); +}); diff --git a/server/src/__tests__/issue-execution-policy.test.ts b/server/src/__tests__/issue-execution-policy.test.ts index 8b9d560191c..c49890ba0b9 100644 --- a/server/src/__tests__/issue-execution-policy.test.ts +++ b/server/src/__tests__/issue-execution-policy.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { applyIssueExecutionPolicyTransition, normalizeIssueExecutionPolicy, parseIssueExecutionState } from "../services/issue-execution-policy.ts"; import type { IssueExecutionPolicy, IssueExecutionState } from "@paperclipai/shared"; +import { buildExecutionPolicy } from "../../../ui/src/lib/issue-execution-policy.ts"; const coderAgentId = "11111111-1111-4111-8111-111111111111"; const qaAgentId = "22222222-2222-4222-8222-222222222222"; @@ -43,6 +44,23 @@ describe("normalizeIssueExecutionPolicy", () => { expect(normalizeIssueExecutionPolicy({ stages: [] })).toBeNull(); }); + it("retains an independent review limit through removing and restoring the last reviewer", () => { + const original = normalizeIssueExecutionPolicy({ + maxReviewRounds: 4, + stages: [{ type: "review", participants: [{ type: "agent", agentId: qaAgentId }] }], + }); + const edited = buildExecutionPolicy({ existingPolicy: original, reviewerValues: [], approverValues: [] }); + const saved = normalizeIssueExecutionPolicy(edited); + expect(saved).toEqual({ mode: "normal", commentRequired: true, stages: [], maxReviewRounds: 4 }); + expect(applyIssueExecutionPolicyTransition({ + issue: { status: "in_progress", assigneeAgentId: coderAgentId }, + policy: saved, requestedStatus: "done", requestedAssigneePatch: {}, actor: { agentId: coderAgentId }, + })).toEqual({ patch: {} }); + expect(normalizeIssueExecutionPolicy(buildExecutionPolicy({ + existingPolicy: saved, reviewerValues: [`agent:${qaAgentId}`], approverValues: [], + }))).toMatchObject({ maxReviewRounds: 4, stages: [{ type: "review" }] }); + }); + it("throws when all participants are invalid (missing agentId)", () => { expect(() => normalizeIssueExecutionPolicy({ diff --git a/server/src/__tests__/remote-http-endpoint-guard.test.ts b/server/src/__tests__/remote-http-endpoint-guard.test.ts index 3602be470f4..2e23eaeec45 100644 --- a/server/src/__tests__/remote-http-endpoint-guard.test.ts +++ b/server/src/__tests__/remote-http-endpoint-guard.test.ts @@ -1,11 +1,33 @@ import { describe, expect, it } from "vitest"; import { assertPublicRemoteHttpEndpoint } from "../services/remote-http-endpoint-guard.js"; +import { readRemoteConnectionFailure } from "../services/remote-connection-failure.js"; function guardError(message: string, code: string) { return Object.assign(new Error(message), { code }); } describe("remote HTTP endpoint guard", () => { + it.each(["ENOTFOUND", "ENODATA", "EAI_AGAIN"])("retains proven DNS failure %s across the public wrapper", async (code) => { + const failure = await assertPublicRemoteHttpEndpoint(new URL("https://missing.example.test/mcp"), { + lookup: async () => { throw Object.assign(new Error("lookup failed"), { code }); }, + }, guardError).catch((error) => error); + expect(failure).toMatchObject({ code: "remote_http_dns_failed" }); + expect(readRemoteConnectionFailure(failure)).toBe("dns_failure"); + }); + + it("marks its exact DNS deadline without trusting unknown lookup exceptions", async () => { + const timedOut = await assertPublicRemoteHttpEndpoint(new URL("https://silent.example.test/mcp"), { + lookup: async () => new Promise(() => {}), dnsTimeoutMs: 1, + }, guardError).catch((error) => error); + expect(timedOut).toMatchObject({ code: "remote_http_dns_failed" }); + expect(readRemoteConnectionFailure(timedOut)).toBe("connection_timeout"); + const unknown = await assertPublicRemoteHttpEndpoint(new URL("https://broken.example.test/mcp"), { + lookup: async () => { throw new TypeError("internal lookup bug: ENOTFOUND"); }, + }, guardError).catch((error) => error); + expect(unknown).toMatchObject({ code: "remote_http_dns_failed" }); + expect(readRemoteConnectionFailure(unknown)).toBeNull(); + }); + it("blocks hostnames that resolve to private network addresses", async () => { await expect(assertPublicRemoteHttpEndpoint( new URL("https://metadata.example/mcp"), diff --git a/server/src/__tests__/remote-http-rebinding.test.ts b/server/src/__tests__/remote-http-rebinding.test.ts index 6d9b834977a..7730086e184 100644 --- a/server/src/__tests__/remote-http-rebinding.test.ts +++ b/server/src/__tests__/remote-http-rebinding.test.ts @@ -1,9 +1,13 @@ import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http"; -import { connect as netConnect, createServer as netCreateServer, type AddressInfo, type Socket } from "node:net"; +import { connect as netConnect, createServer as netCreateServer, Socket, type AddressInfo } from "node:net"; import { gzipSync } from "node:zlib"; import { afterEach, describe, expect, it } from "vitest"; import { guardedRemoteHttpFetch, type RemoteHttpSocketFactory } from "../services/remote-http-fetch.js"; +import { readRemoteConnectionFailure } from "../services/remote-connection-failure.js"; +import { isExpectedMcpConnectionFailure, retainMcpConnectionFailure, withMcpConnectionFailure } from "../services/mcp-connection-failure.js"; +import { HttpError } from "../errors.js"; +import { initializeMcpHttpSession } from "../services/mcp-http.js"; /** * PAP-17098 — DNS-rebinding regression coverage for outbound MCP/OAuth calls. @@ -147,14 +151,16 @@ describe("guarded remote HTTP fetch (PAP-17098 DNS rebinding)", () => { throw Object.assign(new TypeError("fetch failed"), { cause }); }) as typeof fetch; - await expect(guardedRemoteHttpFetch("https://8.8.8.8/mcp", {}, { + const failure = await guardedRemoteHttpFetch("https://8.8.8.8/mcp", {}, { allowPrivateNetwork: true, unpinnedFetch, error: guardError, - })).rejects.toMatchObject({ + }).catch((error) => error); + expect(failure).toMatchObject({ code: "remote_http_dns_failed", message: "Remote MCP connection hostname could not be resolved", }); + expect(readRemoteConnectionFailure(failure)).toBe("dns_failure"); }); it("pins the connection to the approved address so a rebind never reaches loopback", async () => { @@ -352,13 +358,15 @@ describe("guarded remote HTTP fetch (PAP-17098 DNS rebinding)", () => { }); const network = routingSocketFactory({ [PUBLIC_ADDRESS]: upstream.port }); - await expect(guardedRemoteHttpFetch(`http://${REBIND_HOST}/mcp`, {}, { + const failure = await guardedRemoteHttpFetch(`http://${REBIND_HOST}/mcp`, {}, { allowPrivateNetwork: false, lookup: async () => [{ address: PUBLIC_ADDRESS, family: 4 }], socketFactory: network.factory, responseTimeoutMs: 150, error: guardError, - })).rejects.toMatchObject({ code: "remote_http_response_timeout" }); + }).catch((error) => error); + expect(failure).toMatchObject({ code: "remote_http_response_timeout" }); + expect(readRemoteConnectionFailure(failure)).toBe("connection_timeout"); // The deadline has to hand back the socket, not just the request handler: // a bounded call that still leaks a descriptor per silent peer is the same @@ -416,11 +424,69 @@ describe("guarded remote HTTP fetch (PAP-17098 DNS rebinding)", () => { }); expect(response.status).toBe(200); - await expect(response.text()).rejects.toMatchObject({ code: "remote_http_response_timeout" }); + const failure = await response.text().catch((error) => error); + expect(failure).toMatchObject({ code: "remote_http_response_timeout" }); + expect(readRemoteConnectionFailure(failure)).toBe("connection_timeout"); await flush(); expect(network.sockets.map((socket) => socket.destroyed)).toEqual([true]); }); + it("preserves the owned body timeout through the MCP initialization wrapper", async () => { + const upstream = await startServer((_req, res) => { + res.writeHead(200, { "content-type": "application/json" }); + res.write('{"jsonrpc":"2.0",'); + }); + const network = routingSocketFactory({ [PUBLIC_ADDRESS]: upstream.port }); + const failure = await initializeMcpHttpSession({ + requestId: "owned-deadline", + send: (init) => withMcpConnectionFailure(() => guardedRemoteHttpFetch(`http://${REBIND_HOST}/mcp`, init, { + allowPrivateNetwork: false, + lookup: async () => [{ address: PUBLIC_ADDRESS, family: 4 }], + socketFactory: network.factory, + responseTimeoutMs: 150, + error: guardError, + })), + }).catch((error) => error); + expect(failure).toMatchObject({ name: "McpHttpInitializationError", stage: "initialize", status: null, + message: "Remote MCP initialization returned an invalid response" }); + expect(failure.response).toBeUndefined(); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(failure, new HttpError(502, failure.message)))).toBe(true); + await flush(); + expect(network.sockets.map((socket) => socket.destroyed)).toEqual([true]); + }); + + it.each([ + ["EMFILE", "ECONNREFUSED", false], + ["ERR_SSL_UNKNOWN_PROTOCOL", "deadline", false], + ["ECONNRESET", "ECONNREFUSED", true], + ["ETIMEDOUT", "deadline", true], + ] as const)("keeps every failed address relevant to reporting: %s then %s", async (firstCode, lastCode, expected) => { + const first = Object.assign(new Error("first address failed"), { code: firstCode }); + const last = Object.assign(new Error("last address failed"), { code: lastCode }); + const addresses = ["93.184.216.35", PUBLIC_ADDRESS]; + const dialled: string[] = []; + const failure = await withMcpConnectionFailure(() => guardedRemoteHttpFetch(`http://${REBIND_HOST}/mcp`, {}, { + allowPrivateNetwork: false, + lookup: async () => addresses.map((address) => ({ address, family: 4 })), + socketFactory: ({ address }) => { + dialled.push(address); + const socket = new Socket(); + openSockets.push(socket); + if (dialled.length === 1 || lastCode !== "deadline") { + queueMicrotask(() => socket.emit("error", dialled.length === 1 ? first : last)); + } + return socket; + }, + connectTimeoutMs: 5, + error: guardError, + })).catch((error) => error); + expect(dialled).toEqual(addresses); + if (lastCode === "deadline") expect(failure.message).toBe("Timed out trying to connect to the remote MCP endpoint"); + else expect(failure).toBe(last); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(failure, new HttpError(502, failure.message)))).toBe(expected); + expect(openSockets.every((socket) => socket.destroyed)).toBe(true); + }); + it("falls over to the next approved address when the first is unreachable", async () => { // Pinning must not cost the failover `fetch` gave a multi-homed host: one // dead A record should not take the connection down with it. @@ -447,6 +513,29 @@ describe("guarded remote HTTP fetch (PAP-17098 DNS rebinding)", () => { expect(network.dialled).toEqual(["93.184.216.35", PUBLIC_ADDRESS]); }); + it("keeps a successful fallback unchanged even after an unknown first-address failure", async () => { + const upstream = await startServer(); + const network = routingSocketFactory({ [PUBLIC_ADDRESS]: upstream.port }); + const dialled: string[] = []; + const response = await withMcpConnectionFailure(() => guardedRemoteHttpFetch(`http://${REBIND_HOST}/mcp`, {}, { + allowPrivateNetwork: false, + lookup: async () => [{ address: "93.184.216.35", family: 4 }, { address: PUBLIC_ADDRESS, family: 4 }], + socketFactory: (target) => { + dialled.push(target.address); + if (target.address === PUBLIC_ADDRESS) return network.factory(target); + const socket = new Socket(); + openSockets.push(socket); + queueMicrotask(() => socket.emit("error", Object.assign(new Error("local limit"), { code: "EMFILE" }))); + return socket; + }, + error: guardError, + })); + expect(dialled).toEqual(["93.184.216.35", PUBLIC_ADDRESS]); + await expect(response.json()).resolves.toEqual({ ok: true }); + expect(readRemoteConnectionFailure(response)).toBeNull(); + expect(upstream.requests).toHaveLength(1); + }); + it("fails closed instead of falling over when a peer fails the address check", async () => { // Failover covers unreachable addresses only. A peer that answers from an // unapproved address is the rebinding defence firing, and must not be diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 27b8c6d8b4f..ba944f45d45 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -87,6 +87,7 @@ import { import { toolAccessRoutes } from "../routes/tool-access.js"; import { errorHandler } from "../middleware/index.js"; import * as sentry from "../sentry.js"; +import { HttpError } from "../errors.js"; import type { VercelConnectClient } from "../services/vercel-connect.js"; import { invalidatePaperclipCloudConnectorCapabilities, type PaperclipCloudConnector } from "../services/paperclip-cloud-connector.js"; @@ -2965,6 +2966,35 @@ describeEmbeddedPostgres("tool access service", () => { ); }); + it("reports persistence failure after a recognized MCP outage instead of suppressing the database error", async () => { + const company = await createCompany(db); + const [application] = await db.insert(toolApplications).values({ + companyId: company.id, applicationKey: `audit-failure-${randomUUID()}`, + name: "MCP audit fixture", type: "mcp_http", status: "active", + }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: company.id, applicationId: application!.id, name: "MCP audit fixture", + uid: `test/${randomUUID()}`, transport: "mcp_remote", status: "draft", enabled: false, + config: { url: "https://audit-failure.example.test/mcp" }, + transportConfig: { url: "https://audit-failure.example.test/mcp" }, + }).returning(); + vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response(null, { status: 503 })); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + await db.execute(sql`CREATE FUNCTION test_mcp_audit_failure() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN RAISE EXCEPTION 'synthetic MCP audit write failure'; END $$`); + await db.execute(sql`CREATE TRIGGER test_mcp_audit_failure BEFORE INSERT ON tool_access_audit_events + FOR EACH ROW EXECUTE FUNCTION test_mcp_audit_failure()`); + try { + const response = await request(createRouteApp(db)).post(`/api/tool-connections/${connection!.id}/catalog/refresh`); + expect(response.status).toBe(500); + expect(capture).toHaveBeenCalledOnce(); + expect(capture.mock.calls[0]?.[0]).toMatchObject({ cause: expect.objectContaining({ message: "synthetic MCP audit write failure" }) }); + } finally { + await db.execute(sql`DROP TRIGGER test_mcp_audit_failure ON tool_access_audit_events`); + await db.execute(sql`DROP FUNCTION test_mcp_audit_failure()`); + } + }); + it.each([ [ "local_trusted", @@ -15297,15 +15327,81 @@ describeEmbeddedPostgres("tool access service", () => { }, ); + it.each( + (["catalog", "catalog/refresh", "health-check"] as const).flatMap((path) => + ([401, 403] as const).map((upstreamStatus) => ({ path, upstreamStatus })), + ), + )( + "preserves reconnect guidance and degraded health for explicit OAuth scope failure on $path (HTTP $upstreamStatus)", + async ({ path, upstreamStatus }) => { + const company = await createCompany(db); + const [application] = await db.insert(toolApplications).values({ + companyId: company.id, + applicationKey: `scope-status-${randomUUID()}`, + name: "OAuth scope fixture", + type: "mcp_http", + status: "active", + }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: company.id, + applicationId: application!.id, + name: "OAuth scope fixture", + uid: `test/${randomUUID()}`, + transport: "mcp_remote", + status: "draft", + enabled: false, + config: { url: "https://scope-status.example.test/mcp" }, + transportConfig: { url: "https://scope-status.example.test/mcp" }, + }).returning(); + vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response( + JSON.stringify({ error: "private provider details" }), + { status: upstreamStatus, headers: { "www-authenticate": 'Bearer error="insufficient_scope", scope="tools:read"' } }, + )); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const app = createRouteApp(db); + const url = `/api/tool-connections/${connection!.id}/${path}`; + const response = await (path === "catalog" ? request(app).get(url) : request(app).post(url)); + + expect(response.status).toBe(422); + expect(response.body).toMatchObject({ + code: "oauth_insufficient_scope", + error: expect.stringContaining("Reconnect this connection"), + details: { + code: "oauth_insufficient_scope", + setupUrl: expect.any(String), + reconnectUrl: expect.any(String), + }, + }); + expect(JSON.stringify(response.body)).not.toContain("private provider details"); + expect(capture).not.toHaveBeenCalled(); + const [updated] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id)); + expect(updated?.healthStatus).toBe("degraded"); + expect(updated?.healthMessage).toBe(response.body.error); + await expect(db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.connectionId, connection!.id))) + .resolves.toEqual(expect.arrayContaining([expect.objectContaining({ + action: path === "health-check" ? "tool_connection.health_check" : "tool_connection.catalog_refresh", + outcome: "failure", + reasonCode: "oauth_insufficient_scope", + })])); + await expect(db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.connectionId, connection!.id))) + .resolves.toHaveLength(0); + }, + ); + it.each([ ["catalog", 401, 'Bearer realm="app"', 422, false], ["catalog/refresh", 401, 'Bearer realm="app"', 422, false], ["catalog", 400, null, 502, true], ["catalog/refresh", 400, null, 502, true], - ["catalog", 503, null, 502, true], - ["catalog/refresh", 503, null, 502, true], + ["catalog", 503, null, 502, false], + ["catalog/refresh", 503, null, 502, false], + ["catalog/refresh", 403, null, 502, false], + ["health-check", 503, 'Bearer error="insufficient_scope"', 502, false], + ["catalog/refresh", 404, null, 502, false], + ["catalog/refresh", 429, null, 502, false], + ["health-check", 500, null, 502, false], ] as const)( - "classifies %s upstream HTTP %i without hiding provider failures", + "classifies %s upstream HTTP %i while retaining the failed response", async (path, upstreamStatus, challenge, expectedStatus, reportable) => { const company = await createCompany(db); const [application] = await db @@ -15349,7 +15445,7 @@ describeEmbeddedPostgres("tool access service", () => { : request(app).post(url)); expect(res.status).toBe(expectedStatus); - if (challenge) { + if (challenge && upstreamStatus === 401) { expect(res.body).toMatchObject({ error: "This app needs you to sign in.", code: "oauth_challenge", @@ -15361,6 +15457,9 @@ describeEmbeddedPostgres("tool access service", () => { }); } expect(capture).toHaveBeenCalledTimes(reportable ? 1 : 0); + const [updated] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id)); + expect(updated?.healthStatus).toBe("error"); + expect(updated?.healthMessage).toBe(res.body.error); await expect( db.select().from(toolCatalogEntries) .where(eq(toolCatalogEntries.connectionId, connection!.id)), @@ -15368,6 +15467,92 @@ describeEmbeddedPostgres("tool access service", () => { }, ); + it.each([ + ["initialize reset", false], + ["initialize parser", true], + ["initialize internal reader", true], + ["notification 503", false], + ["notification 400", true], + ] as const)("preserves session-required MCP failure reporting: %s", async (scenario, reportable) => { + const company = await createCompany(db); + const [application] = await db.insert(toolApplications).values({ + companyId: company.id, applicationKey: `session-status-${randomUUID()}`, + name: "MCP session fixture", type: "mcp_http", status: "active", + }).returning(); + const config = { url: "https://session-status.example.test/mcp", mcpSessionRequired: true }; + const [connection] = await db.insert(toolConnections).values({ + companyId: company.id, applicationId: application!.id, name: "MCP session fixture", + uid: `test/${randomUUID()}`, transport: "mcp_remote", status: "draft", enabled: false, + config, transportConfig: config, + }).returning(); + const methods: string[] = []; + vi.spyOn(globalThis, "fetch").mockImplementation(async (_url, init) => { + const request = JSON.parse(String(init?.body)); + methods.push(request.method); + if (request.method === "notifications/initialized") return new Response(null, { status: scenario === "notification 503" ? 503 : 400 }); + expect(request.method).toBe("initialize"); + if (scenario === "initialize parser") return new Response("not JSON"); + if (scenario === "initialize reset" || scenario === "initialize internal reader") { + const error = scenario === "initialize reset" + ? new TypeError("body terminated", { cause: { code: "ECONNRESET" } }) + : new TypeError("internal reader failed"); + return new Response(new ReadableStream({ start(controller) { controller.error(error); } })); + } + return new Response(JSON.stringify({ jsonrpc: "2.0", id: request.id, result: { protocolVersion: "2025-06-18" } })); + }); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const response = await request(createRouteApp(db)).post(`/api/tool-connections/${connection!.id}/catalog/refresh`); + expect(response.status).toBe(502); + expect(response.body.error).toBeTruthy(); + expect(capture).toHaveBeenCalledTimes(reportable ? 1 : 0); + expect(methods).toEqual(scenario.startsWith("notification") ? ["initialize", "notifications/initialized"] : ["initialize"]); + const [updated] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id)); + expect(updated?.healthStatus).toBe("error"); + expect(updated?.healthMessage).toBe(response.body.error); + await expect(db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.connectionId, connection!.id))) + .resolves.toEqual(expect.arrayContaining([expect.objectContaining({ outcome: "failure", action: "tool_connection.catalog_refresh" })])); + }); + + it.each([ + ["known transport", () => { throw new TypeError("fetch failed", { cause: { code: "ECONNREFUSED" } }); }, false], + ["direct transport", () => { throw Object.assign(new Error("connection reset"), { code: "ECONNRESET" }); }, false], + ["unknown transport", () => { throw new TypeError("ECONNREFUSED"); }, true], + ["database code", () => { throw Object.assign(new Error("query failed"), { code: "42P01" }); }, true], + ["mixed transport", () => { throw new TypeError("fetch failed", { cause: { errors: [{ code: "ECONNREFUSED" }, { code: "ERR_INTERNAL_ASSERTION" }] } }); }, true], + ["forged classification", () => { throw new HttpError(502, "Remote app returned HTTP 503", { status: 503, connectionFailure: { schemaVersion: 1, provider: "mcp_http", operation: "discover_tools", reason: "remote_unavailable" } }); }, true], + ["malformed response", () => new Response("not JSON"), true], + ["invalid catalog", () => new Response(JSON.stringify({ jsonrpc: "2.0", id: "paperclip-catalog-refresh", result: {} })), true], + ] as const)( + "preserves MCP diagnostics and reports only unclassified failure: %s", + async (_name, reply, reportable) => { + const company = await createCompany(db); + const [application] = await db.insert(toolApplications).values({ + companyId: company.id, applicationKey: `transport-status-${randomUUID()}`, + name: "MCP transport fixture", type: "mcp_http", status: "active", + }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: company.id, applicationId: application!.id, name: "MCP transport fixture", + uid: `test/${randomUUID()}`, transport: "mcp_remote", status: "draft", enabled: false, + config: { url: "https://transport-status.example.test/mcp" }, + transportConfig: { url: "https://transport-status.example.test/mcp" }, + }).returning(); + vi.spyOn(globalThis, "fetch").mockImplementation(async () => reply()); + const capture = vi.spyOn(sentry, "captureException").mockImplementation(() => {}); + const response = await request(createRouteApp(db)).post(`/api/tool-connections/${connection!.id}/catalog/refresh`); + + expect(response.status).toBe(502); + expect(response.body.error).toBeTruthy(); + expect(capture).toHaveBeenCalledTimes(reportable ? 1 : 0); + const [updated] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection!.id)); + expect(updated?.healthStatus).toBe("error"); + expect(updated?.healthMessage).toBe(response.body.error); + await expect(db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.connectionId, connection!.id))) + .resolves.toEqual(expect.arrayContaining([expect.objectContaining({ outcome: "failure", action: "tool_connection.catalog_refresh" })])); + await expect(db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.connectionId, connection!.id))) + .resolves.toHaveLength(0); + }, + ); + it.each([ [ "local_trusted", diff --git a/server/src/middleware/error-handler.ts b/server/src/middleware/error-handler.ts index 31dd8535e62..23103185c79 100644 --- a/server/src/middleware/error-handler.ts +++ b/server/src/middleware/error-handler.ts @@ -13,6 +13,7 @@ import { redactSensitiveValueOccurrences, } from "./redact-sensitive.js"; import { recordResponsibleUserDenialOnActiveRun } from "../services/responsible-user-denial-run-outcomes.js"; +import { isExpectedMcpConnectionFailure } from "../services/mcp-connection-failure.js"; export interface ErrorContext { error: { @@ -85,10 +86,10 @@ function sanitizeSecretSensitiveResponse( } /** Report a server-side crash to every error sink. */ -function reportCrash(error: Error): void { +function reportCrash(error: Error, reportToSentry = true): void { const tc = getTelemetryClient(); if (tc) trackErrorHandlerCrash(tc, { errorCode: error.name }); - captureException(error); + if (reportToSentry) captureException(error); } function getPaperclipDb(req: Request): Db | null { @@ -176,7 +177,7 @@ export function errorHandler( }, reportableError, ); - reportCrash(reportableError); + reportCrash(reportableError, !isExpectedMcpConnectionFailure(err)); } const secretSensitiveServerError = err.status >= 500 && diff --git a/server/src/services/__tests__/git-connection-failure.test.ts b/server/src/services/__tests__/git-connection-failure.test.ts new file mode 100644 index 00000000000..17692dfe407 --- /dev/null +++ b/server/src/services/__tests__/git-connection-failure.test.ts @@ -0,0 +1,202 @@ +import { describe, expect, it } from "vitest"; +import type { GitConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; +import { + classifyGitCloneFailure, + GitConnectionFailureError, + readGitConnectionFailure, +} from "../git-connection-failure.js"; + +const httpsRemote = "https://example.test/team/repository.git"; +const sshRemote = "git@example.test:team/repository.git"; +const authenticationStderr = `fatal: Authentication failed for '${httpsRemote}/'`; + +function childFailure(stderr: string, overrides: Record = {}) { + return Object.assign(new Error("Git clone subprocess failed"), { + code: 128, + stderr, + signal: null, + killed: false, + ...overrides, + }); +} + +function classification(reason: GitConnectionFailure["reason"]): GitConnectionFailure { + return { schemaVersion: 1, provider: "git", operation: "clone", reason }; +} + +describe("Git clone connection failure classification", () => { + it.each([ + ["HTTPS authentication", httpsRemote, authenticationStderr, "authentication_failed"], + ["disabled username prompt", httpsRemote, + "fatal: could not read Username for 'https://example.test': terminal prompts disabled", "authentication_failed"], + ["disabled password prompt", httpsRemote, + "fatal: could not read Password for 'https://git@example.test': terminal prompts disabled", "authentication_failed"], + ["SSH public key", sshRemote, + "git@example.test: Permission denied (publickey).\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.", "authentication_failed"], + ["private repository", httpsRemote, + `remote: Repository not found.\nfatal: repository '${httpsRemote}/' not found`, "repository_unavailable"], + ["HTTP unauthorized", httpsRemote, + `fatal: unable to access '${httpsRemote}/': The requested URL returned error: 401`, "authentication_failed"], + ["HTTP forbidden", httpsRemote, + `fatal: unable to access '${httpsRemote}/': The requested URL returned error: 403`, "authentication_failed"], + ["HTTP missing repository", httpsRemote, + `fatal: unable to access '${httpsRemote}/': The requested URL returned error: 404`, "repository_unavailable"], + ["invalid HTTP port", "https://example.test:invalid/team/repository.git", + "fatal: unable to access 'https://example.test:invalid/team/repository.git/': Port number was not a decimal number between 0 and 65535", "invalid_remote"], + ["invalid HTTP URL", "https://example.test:invalid/team/repository.git", + "fatal: unable to access 'https://example.test:invalid/team/repository.git/': URL rejected: Port number was not a decimal number between 0 and 65535", "invalid_remote"], + ["SSH alias DNS", "git@repo-alias.example.test:team/repository.git", + "ssh: Could not resolve hostname repo-alias.example.test: Name or service not known\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.", "dns_failure"], + ["HTTPS DNS", httpsRemote, + `fatal: unable to access '${httpsRemote}/': Could not resolve host: example.test`, "dns_failure"], + ["SSH refusal", sshRemote, + "ssh: connect to host example.test port 22: Connection refused\nfatal: Could not read from remote repository.", "connection_refused"], + ["HTTPS refusal", httpsRemote, + `fatal: unable to access '${httpsRemote}/': Failed to connect to example.test: Connection refused`, "connection_refused"], + ["SSH unreachable network", "ssh://git@example.test/team/repository.git", + "ssh: connect to host example.test port 22: Network is unreachable\nfatal: Could not read from remote repository.", "network_unreachable"], + ["SSH missing route", sshRemote, + "ssh: connect to host example.test port 22: No route to host\nfatal: Could not read from remote repository.", "network_unreachable"], + ["HTTPS reset", httpsRemote, + `fatal: unable to access '${httpsRemote}/': Recv failure: Connection was reset`, "connection_reset"], + ["SSH reset", sshRemote, + "Connection reset by example.test port 22\nfatal: Could not read from remote repository.", "connection_reset"], + ["SSH timeout", sshRemote, + "ssh: connect to host example.test port 22: Connection timed out\nfatal: Could not read from remote repository.", "connection_timeout"], + ["HTTPS timeout", httpsRemote, + `fatal: unable to access '${httpsRemote}/': Operation timed out after 30000 milliseconds with 0 bytes received`, "connection_timeout"], + ["HTTPS certificate", httpsRemote, + `fatal: unable to access '${httpsRemote}/': SSL certificate problem: unable to get local issuer certificate`, "tls_failure"], + ["HTTPS rate limit", httpsRemote, + `fatal: unable to access '${httpsRemote}/': The requested URL returned error: 429`, "rate_limited"], + ["HTTPS service unavailable", httpsRemote, + `fatal: unable to access '${httpsRemote}/': The requested URL returned error: 503`, "remote_unavailable"], + ["HTTPS bad gateway", httpsRemote, + `fatal: unable to access '${httpsRemote}/': The requested URL returned error: 502`, "remote_unavailable"], + ] as const)("classifies the actual remote clone diagnostic: %s", (_name, remote, stderr, reason) => { + expect(classifyGitCloneFailure(remote, childFailure(`Cloning into 'repository'...\n${stderr}\n`))) + .toEqual(classification(reason)); + }); + + it.each([ + "/tmp/repository", "./repository", "../repository", "repository", "~/repository", + "C:\\repository", "C:/repository", "file:///tmp/repository", "file://example.test/repository", + "ftp://example.test/repository", "gopher://example.test/repository", "ext::example.test/repository", + "https::example.test/repository", "", "example.test/repository", + ])("keeps local paths and unsupported transports reportable: %s", (remote) => { + expect(classifyGitCloneFailure(remote, childFailure(authenticationStderr))).toBeNull(); + }); + + it.each([ + ["missing Git executable", { code: "ENOENT" }], + ["string exit code", { code: "128" }], + ["successful exit", { code: 0 }], + ["non-fatal exit", { code: 1 }], + ["command error exit", { code: 2 }], + ["usage exit", { code: 129 }], + ["negative exit", { code: -1 }], + ["fractional exit", { code: 128.5 }], + ["unknown exit", { code: undefined }], + ["nonfinite exit", { code: Infinity }], + ["killed child", { killed: true }], + ["terminated child", { signal: "SIGTERM" }], + ["missing stderr", { stderr: undefined }], + ["buffer stderr", { stderr: Buffer.from(authenticationStderr) }], + ["object stderr", { stderr: { message: authenticationStderr } }], + ] as const)("requires unambiguous child-process failure evidence: %s", (_name, overrides) => { + expect(classifyGitCloneFailure(httpsRemote, childFailure(authenticationStderr, overrides))).toBeNull(); + }); + + it.each([ + "fatal: No space left on device", + "fatal: Read-only file system", + "fatal: Input/output error", + "fatal: Too many open files", + "fatal: Cannot allocate memory", + "fatal: could not create work tree dir 'repository': Permission denied", + "fatal: cannot open file 'repository/.git/config': Permission denied", + "fatal: bad object HEAD", + "fatal: corrupt loose object", + "fatal: invalid index", + "fatal: not a git repository", + "fatal: detected dubious ownership in repository", + "fatal: unexpected internal clone failure", + "error: unrelated local helper failed", + "warning: unrelated local configuration failure", + ])("does not hide a mixed local or unknown diagnostic: %s", (otherDiagnostic) => { + for (const stderr of [ + `${authenticationStderr}\n${otherDiagnostic}`, + `${otherDiagnostic}\n${authenticationStderr}`, + ]) { + expect(classifyGitCloneFailure(httpsRemote, childFailure(stderr))).toBeNull(); + } + }); + + it.each([ + "", "fatal: Could not read from remote repository.", + "fatal: destination path 'repository' already exists and is not an empty directory.", + "fatal: unable to access 'https://example.test/team/repository.git/': The requested URL returned error: 418", + "fatal: invalid object name 'The requested URL returned error: 503'", + "fatal: cannot open file 'Permission denied (publickey)': Permission denied", + "error: local helper reported fatal: Authentication failed before crashing", + ])("does not infer a remote cause from ambiguous stderr: %s", (stderr) => { + const error = childFailure(stderr); + error.message = authenticationStderr; + expect(classifyGitCloneFailure(httpsRemote, error)).toBeNull(); + }); + + it("does not classify messages or serialized child-error lookalikes", () => { + for (const error of [ + authenticationStderr, + new Error(authenticationStderr), + { message: authenticationStderr, code: 128, stderr: authenticationStderr }, + null, + ]) { + expect(classifyGitCloneFailure(httpsRemote, error)).toBeNull(); + } + }); +}); + +describe("producer-owned Git connection failure errors", () => { + it("preserves the existing error message and returns only the bounded classification", () => { + const diagnostic = classifyGitCloneFailure(httpsRemote, childFailure(authenticationStderr)); + expect(diagnostic).toEqual(classification("authentication_failed")); + const error = new GitConnectionFailureError("Unable to prepare project repository", diagnostic!); + + expect(error).toBeInstanceOf(Error); + expect(error.name).toBe("GitConnectionFailureError"); + expect(error.message).toBe("Unable to prepare project repository"); + expect(readGitConnectionFailure(error)).toEqual(classification("authentication_failed")); + expect(JSON.stringify(readGitConnectionFailure(error))).not.toContain("example.test"); + expect(error).not.toHaveProperty("connectionFailure"); + expect(error).not.toHaveProperty("stderr"); + }); + + it("does not let caller mutations change retained failure evidence", () => { + const input = classification("authentication_failed"); + const error = new GitConnectionFailureError("Clone failed", input); + input.reason = "remote_unavailable"; + const firstRead = readGitConnectionFailure(error)!; + expect(firstRead).toEqual(classification("authentication_failed")); + firstRead.reason = "dns_failure"; + expect(readGitConnectionFailure(error)).toEqual(classification("authentication_failed")); + expect(readGitConnectionFailure(error)).not.toBe(firstRead); + }); + + it("rejects spoofed markers, prototype lookalikes, and serialized errors", () => { + const diagnostic = classification("authentication_failed"); + const real = new GitConnectionFailureError("Clone failed", diagnostic); + const spoofed = Object.assign(new Error("Clone failed"), { + name: "GitConnectionFailureError", connectionFailure: diagnostic, + }); + for (const value of [ + null, "Clone failed", diagnostic, { connectionFailure: diagnostic }, spoofed, + Object.assign(Object.create(GitConnectionFailureError.prototype), spoofed), + JSON.parse(JSON.stringify(real)), + new Error("Wrapper", { cause: real }), + ]) { + expect(readGitConnectionFailure(value)).toBeNull(); + } + expect(readGitConnectionFailure(real)).toEqual(diagnostic); + }); +}); diff --git a/server/src/services/__tests__/mcp-connection-failure.test.ts b/server/src/services/__tests__/mcp-connection-failure.test.ts new file mode 100644 index 00000000000..6a422a1af31 --- /dev/null +++ b/server/src/services/__tests__/mcp-connection-failure.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from "vitest"; +import { HttpError } from "../../errors.js"; +import { isExpectedMcpConnectionFailure, retainMcpConnectionFailure, withMcpConnectionFailure } from "../mcp-connection-failure.js"; +import { assertPublicRemoteHttpEndpoint } from "../remote-http-endpoint-guard.js"; +import { classifyRemoteConnectionError, markRemoteConnectionFailure, readRemoteConnectionFailure, recordRemoteConnectionAttempts } from "../remote-connection-failure.js"; + +describe("MCP outbound failure provenance", () => { + it("lets an unknown address override existing MCP and owned-timeout receipts on the final error", async () => { + const last = markRemoteConnectionFailure(Object.assign(new HttpError(502, "last address timed out"), { code: "ETIMEDOUT" }), "connection_timeout"); + await expect(withMcpConnectionFailure(async () => { throw last; })).rejects.toBe(last); + expect(isExpectedMcpConnectionFailure(last)).toBe(true); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(last, new HttpError(502, last.message)))).toBe(true); + const unknown = Object.assign(new Error("first address exhausted file descriptors"), { code: "EMFILE" }); + expect(recordRemoteConnectionAttempts(last, [unknown, last])).toBe(last); + expect(readRemoteConnectionFailure(last)).toBeNull(); + expect(classifyRemoteConnectionError(last)).toBeNull(); + expect(classifyRemoteConnectionError(new TypeError("wrapped", { cause: last }))).toBeNull(); + expect(isExpectedMcpConnectionFailure(last)).toBe(false); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(last, new HttpError(502, last.message)))).toBe(false); + await expect(withMcpConnectionFailure(async () => { throw last; })).rejects.toBe(last); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(last, new HttpError(502, last.message)))).toBe(false); + }); + + it.each([ + "ECONNREFUSED", "ENOTFOUND", "ENODATA", "EAI_AGAIN", "EHOSTUNREACH", "ENETUNREACH", + "ECONNRESET", "EPIPE", "UND_ERR_SOCKET", "ETIMEDOUT", "UND_ERR_CONNECT_TIMEOUT", + "UND_ERR_HEADERS_TIMEOUT", "UND_ERR_BODY_TIMEOUT", "CERT_HAS_EXPIRED", "CERT_NOT_YET_VALID", + "DEPTH_ZERO_SELF_SIGNED_CERT", "SELF_SIGNED_CERT_IN_CHAIN", "UNABLE_TO_VERIFY_LEAF_SIGNATURE", + "UNABLE_TO_GET_ISSUER_CERT_LOCALLY", "ERR_TLS_CERT_ALTNAME_INVALID", + ])("recognizes the structured transport failure %s without changing the thrown error", async (code) => { + const original = new TypeError("request failed", { cause: { code } }); + await expect(withMcpConnectionFailure(async () => { throw original; })).rejects.toBe(original); + const wrapped = retainMcpConnectionFailure(original, new HttpError(502, original.message)); + expect(isExpectedMcpConnectionFailure(wrapped)).toBe(true); + expect(Object.keys(original)).not.toContain("connectionFailure"); + }); + + it.each([ + new TypeError("ECONNREFUSED"), + new DOMException("request expired", "TimeoutError"), + Object.assign(new Error("cancelled", { cause: { code: "ECONNRESET" } }), { name: "AbortError" }), + Object.assign(new Error("too many files"), { code: "EMFILE" }), + new TypeError("request failed", { cause: { code: "toString" } }), + new TypeError("request failed", { cause: { code: "ECONNRESET", errors: [] } }), + new TypeError("request failed", { cause: { errors: [{ code: "ECONNRESET" }, { code: "ERR_INTERNAL_ASSERTION" }] } }), + ])("keeps unknown, local, cancellation, and mixed failures visible: %s", async (original) => { + await expect(withMcpConnectionFailure(async () => { throw original; })).rejects.toBe(original); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(original, new HttpError(502, original.message)))) + .toBe(false); + }); + + it("does not carry provenance onto a subsequent independent error", async () => { + const transport = Object.assign(new Error("connection reset"), { code: "ECONNRESET" }); + await expect(withMcpConnectionFailure(async () => { throw transport; })).rejects.toBe(transport); + const persistence = new Error("database write failed", { cause: transport }); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(persistence, new HttpError(502, persistence.message)))) + .toBe(false); + }); + + it.each(["known DNS", "DNS deadline", "unknown lookup"])("retains only producer-proven %s across discovery wrapping", async (kind) => { + const original = await assertPublicRemoteHttpEndpoint(new URL("https://missing.example.test/mcp"), { + dnsTimeoutMs: 1, + lookup: async () => { + if (kind === "known DNS") throw Object.assign(new Error("lookup failed"), { code: "ENOTFOUND" }); + if (kind === "unknown lookup") throw new TypeError("lookup implementation failed"); + return new Promise(() => {}); + }, + }, (message, code) => new HttpError(400, message, { code })).catch((error) => error); + expect(original).toBeInstanceOf(HttpError); + const wrapped = retainMcpConnectionFailure(original, new HttpError(502, original.message)); + expect(isExpectedMcpConnectionFailure(wrapped)).toBe(kind !== "unknown lookup"); + const unrelated = new Error("database failed", { cause: original }); + expect(isExpectedMcpConnectionFailure(retainMcpConnectionFailure(unrelated, new HttpError(502, unrelated.message)))) + .toBe(false); + }); +}); diff --git a/server/src/services/__tests__/run-failure-report.test.ts b/server/src/services/__tests__/run-failure-report.test.ts index 6271b6f520c..be38e96b93b 100644 --- a/server/src/services/__tests__/run-failure-report.test.ts +++ b/server/src/services/__tests__/run-failure-report.test.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import os from "node:os"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { agents, companies, createDb, heartbeatRuns, type Db } from "@paperclipai/db"; +import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, heartbeatRuns, type Db } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -30,7 +30,8 @@ vi.mock("../../log-redaction.js", async (importOriginal) => { import { reportRunFailure, waitForPendingRunFailureReports } from "../run-failure-report.js"; import { REDACTED_EVENT_VALUE } from "../../redaction.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const testDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL; +const embeddedPostgresSupport = testDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; describeEmbeddedPostgres("reportRunFailure", () => { @@ -41,8 +42,13 @@ describeEmbeddedPostgres("reportRunFailure", () => { let inheritedEnv: NodeJS.ProcessEnv; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-failure-report-"); - db = createDb(tempDb.connectionString); + if (testDatabaseUrl) { + await applyPendingMigrations(testDatabaseUrl); + db = createDb(testDatabaseUrl); + } else { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-failure-report-"); + db = createDb(tempDb.connectionString); + } }, 20_000); beforeEach(() => { @@ -66,6 +72,7 @@ describeEmbeddedPostgres("reportRunFailure", () => { }); afterAll(async () => { + if (testDatabaseUrl) await closeRegisteredClients(testDatabaseUrl); await tempDb?.cleanup(); }); @@ -110,6 +117,172 @@ describeEmbeddedPostgres("reportRunFailure", () => { } as unknown as typeof heartbeatRuns.$inferSelect; } + const gitConnection = { schemaVersion: 1, provider: "git", operation: "clone", reason: "authentication_failed" }; + const hermesConnection = { schemaVersion: 1, provider: "hermes_gateway", operation: "create_run", reason: "connection_refused" }; + + function gitConnectionRun(overrides: Partial = {}) { + return buildRun({ + errorCode: "setup_failed", executionStage: "preparing", exitCode: null, signal: null, + resultJson: { connectionFailure: gitConnection, + executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, + ...overrides, + }); + } + + function hermesConnectionRun(overrides: Partial = {}) { + return buildRun({ + errorCode: "hermes_gateway_connect_failed", executionStage: "executing", exitCode: 1, signal: null, + resultJson: { connectionFailure: hermesConnection }, ...overrides, + }); + } + + it("keeps a proven Git clone failure and its persisted task error unchanged", async () => { + await seedCompanyAndAgent(); + const input = gitConnectionRun(); + const [run] = await db.insert(heartbeatRuns).values(input).returning(); + const before = structuredClone(run); + await reportRunFailure(db, run!, { phase: "setup" }); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + expect(run).toEqual(before); + expect(await db.select().from(heartbeatRuns)).toEqual([before]); + }); + + it("keeps an all-external Git materialization aggregate local even across different external causes", async () => { + await seedCompanyAndAgent(); + const run = gitConnectionRun({ errorCode: "workspace_validation_failed" }); + run.resultJson!.workspaceValidation = { reason: "git_worktree_base_materialization_failed", materializationFailures: [ + { connectionFailure: gitConnection }, + { connectionFailure: { ...gitConnection, reason: "repository_unavailable" } }, + ] }; + const before = structuredClone(run); + await reportRunFailure(db, run, { phase: "setup" }); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + expect(run).toEqual(before); + }); + + it.each([ + { status: "timed_out" }, { executionStage: "executing" }, { executionStage: null }, + { errorCode: "adapter_failed" }, { exitCode: 1 }, { exitCode: 0 }, { signal: "SIGTERM" }, + { resultJson: {} }, + { resultJson: { connectionFailure: { ...gitConnection, schemaVersion: 2 }, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }, + { resultJson: { connectionFailure: hermesConnection, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }, + ])("reports a Git marker without its matching setup outcome: %j", async (overrides) => { + await seedCompanyAndAgent(); + await reportRunFailure(db, gitConnectionRun(overrides), { phase: "setup" }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it.each([undefined, "execute"] as const)("reports Git failures outside setup: %s", async (phase) => { + await seedCompanyAndAgent(); + await reportRunFailure(db, gitConnectionRun(), { phase }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it.each([undefined, { kind: "bootstrap" }, { kind: "bootstrap", providerWorkStarted: true }, + { kind: "runtime", providerWorkStarted: false }])("reports Git failures without unstarted bootstrap proof: %j", async (recovery) => { + await seedCompanyAndAgent(); + const run = gitConnectionRun(); + run.resultJson!.executionRecovery = recovery; + await reportRunFailure(db, run, { phase: "setup" }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it.each([ + [], [null], [{ connectionFailure: gitConnection }, {}], + [{ connectionFailure: gitConnection }, { connectionFailure: hermesConnection }], + [{ connectionFailure: gitConnection }, { connectionFailure: { ...gitConnection, reason: "unknown" } }], + [{ connectionFailure: { ...gitConnection, reason: "repository_unavailable" } }], + ].map((failures) => ({ failures })))("reports mixed, malformed, or inconsistent Git materialization evidence: %j", async ({ failures }) => { + await seedCompanyAndAgent(); + const run = gitConnectionRun({ errorCode: "workspace_validation_failed" }); + run.resultJson!.workspaceValidation = { reason: "git_worktree_base_materialization_failed", materializationFailures: failures }; + await reportRunFailure(db, run, { phase: "setup" }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it.each([ + { reason: "git_worktree_base_materialization_failed", materializationFailures: [{ connectionFailure: gitConnection }] }, + { reason: "git_worktree_not_reusable" }, + ])("does not use setup_failed to bypass separate workspace validation evidence: %j", async (validation) => { + await seedCompanyAndAgent(); + const run = gitConnectionRun(); + run.resultJson!.workspaceValidation = validation; + await reportRunFailure(db, run, { phase: "setup" }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it.each([ + ["configuration", "endpoint_missing", "hermes_gateway_api_base_url_missing"], + ["configuration", "endpoint_invalid", "hermes_gateway_api_base_url_invalid"], + ["configuration", "insecure_transport", "hermes_gateway_plain_http_remote_denied"], + ["configuration", "credentials_missing", "hermes_gateway_api_key_missing"], + ["create_run", "authentication_failed", "hermes_gateway_auth_failed"], + ["create_run", "endpoint_not_found", "hermes_gateway_runs_unsupported"], + ["create_run", "rate_limited", "hermes_gateway_rate_limited"], + ["create_run", "remote_unavailable", "hermes_gateway_upstream_error"], + ...["connection_refused", "dns_failure", "network_unreachable", "connection_reset", "connection_timeout", "tls_failure"] + .flatMap((reason) => ["hermes_gateway_connect_failed", "hermes_gateway_protocol_error"].map((code) => ["create_run", reason, code])), + ])("keeps the matching Hermes %s/%s task error local", async (operation, reason, errorCode) => { + await seedCompanyAndAgent({ adapterType: "hermes_gateway" }); + const run = hermesConnectionRun({ errorCode, resultJson: { connectionFailure: { ...hermesConnection, operation, reason } } }); + const before = structuredClone(run); + await reportRunFailure(db, run, { phase: "execute" }); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + expect(run).toEqual(before); + }); + + it.each([undefined, { kind: "bootstrap", providerWorkStarted: true }, { kind: "runtime", providerWorkStarted: true }])( + "does not infer remote nonacceptance from a classified Hermes connection failure: %j", async (recovery) => { + await seedCompanyAndAgent({ adapterType: "hermes_gateway" }); + const input = hermesConnectionRun(); + input.resultJson!.executionRecovery = recovery; + const [run] = await db.insert(heartbeatRuns).values(input).returning(); + const before = structuredClone(run); + await reportRunFailure(db, run!); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + expect(await db.select().from(heartbeatRuns)).toEqual([before]); + }, + ); + + it.each([ + { status: "timed_out" }, { status: "cancelled", startedAt: new Date(0) }, + { exitCode: 0 }, { exitCode: null }, { signal: "SIGTERM" }, + { errorCode: "hermes_gateway_timeout" }, { errorCode: "hermes_gateway_run_failed" }, + { errorCode: "hermes_gateway_auth_failed" }, + { resultJson: {} }, + { resultJson: { connectionFailure: { ...hermesConnection, reason: "unknown" } } }, + { resultJson: { connectionFailure: { ...hermesConnection, privateUrl: "https://private.example.test" } } }, + { resultJson: { connectionFailure: gitConnection } }, + { resultJson: { connectionFailure: { ...hermesConnection, operation: "configuration", reason: "credentials_missing" } } }, + ])("reports an unproven or mismatched Hermes outcome: %j", async (overrides) => { + await seedCompanyAndAgent({ adapterType: "hermes_gateway" }); + const run = hermesConnectionRun(overrides); + const before = structuredClone(run); + await reportRunFailure(db, run, { phase: "execute" }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + expect(run).toEqual(before); + }); + + it("keeps setup errors reportable even when they carry a matching Hermes marker", async () => { + await seedCompanyAndAgent({ adapterType: "hermes_gateway" }); + await reportRunFailure(db, hermesConnectionRun(), { phase: "setup" }); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it.each(["codex_local", "hermes_local"])("requires the actual database adapter, not a Hermes marker on %s", async (adapterType) => { + await seedCompanyAndAgent({ adapterType }); + const run = hermesConnectionRun({ contextSnapshot: { adapterType: "hermes_gateway" } }); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ agentAdapter: adapterType })); + }); + + it.each(["missing", "other_company"])("does not suppress Hermes when its database agent is %s", async (kind) => { + await seedCompanyAndAgent({ adapterType: "hermes_gateway" }); + const run = hermesConnectionRun(kind === "missing" ? { agentId: randomUUID() } : { companyId: randomUUID() }); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ agentAdapter: "unknown" })); + }); + function missingSecretRun(overrides: Partial = {}) { return buildRun({ errorCode: "configuration_incomplete", diff --git a/server/src/services/git-connection-failure.ts b/server/src/services/git-connection-failure.ts new file mode 100644 index 00000000000..f8b7cac02de --- /dev/null +++ b/server/src/services/git-connection-failure.ts @@ -0,0 +1,90 @@ +import type { GitConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; + +const classifiedErrors = new WeakMap(); + +/** Only the outbound Git call may create this reporting classification. */ +export class GitConnectionFailureError extends Error { + constructor(message: string, failure: GitConnectionFailure) { + super(message); + this.name = "GitConnectionFailureError"; + classifiedErrors.set(this, { ...failure }); + } +} + +export function readGitConnectionFailure(error: unknown): GitConnectionFailure | null { + const failure = error instanceof Error ? classifiedErrors.get(error) : undefined; + return failure ? { ...failure } : null; +} + +function isNetworkRemote(remote: string): boolean { + // Local paths/file remotes and arbitrary remote helpers are not connections. + // Keep malformed HTTP/SSH URLs eligible only for Git's explicit URL error. + if (/^(?:https?|ssh|git):\/\//i.test(remote)) return true; + if (/^[a-z][a-z0-9+.-]*:\/\//i.test(remote) || remote.includes("::")) return false; + return /^(?:[^\s/@:]+@)?[a-z0-9][a-z0-9.-]+:[^:\s][^\s]*$/i.test(remote) && !/^[a-z]:[\\/]/i.test(remote); +} + +/** + * Read the actual failed clone subprocess, not a wrapped error message. These + * diagnostics belong to the remote transport; filesystem/repository damage + * and mixed failures remain application errors even if auth text is present. + */ +export function classifyGitCloneFailure(remote: string, error: unknown): GitConnectionFailure | null { + if (!isNetworkRemote(remote) || !(error instanceof Error)) return null; + const failure = error as Error & { code?: unknown; stderr?: unknown; signal?: unknown; killed?: unknown }; + if (failure.code !== 128 || failure.signal != null || failure.killed === true || typeof failure.stderr !== "string") return null; + const lines = failure.stderr.split(/\r?\n/).map((line) => line.trim()).filter(Boolean); + let reason: GitConnectionFailure["reason"] | undefined; + for (const line of lines) { + const classified = classifyTransportLine(line); + if (classified) { + reason ??= classified; + continue; + } + // Git and SSH append these fixed progress/help lines to recognized + // transport errors. Unknown diagnostics, including a second local failure, + // keep the whole operation reportable. The boilerplate alone proves nothing. + if (/^Cloning into ['"].+['"]\.\.\.$/.test(line) || + /^(?:remote: )?(?:Enumerating|Counting|Compressing|Receiving|Resolving|Updating) (?:objects|deltas|files):/.test(line) || + /^remote: Total \d+ /.test(line) || + /^fatal: Could not read from remote repository\.$/.test(line) || + /^Please make sure you have the correct access rights$/.test(line) || + /^and the repository exists\.$/.test(line) || + /^remote: Please see https:\/\/[^\s]+ for information/.test(line)) continue; + return null; + } + return reason ? { schemaVersion: 1, provider: "git", operation: "clone", reason } : null; +} + +function classifyTransportLine(line: string): GitConnectionFailure["reason"] | undefined { + // Parse the transport's own diagnostic prefix before examining its detail. + // A local error may quote exactly the same words in a path or object name. + const http = /^fatal: unable to access ['"][^'"\r\n]+['"]: (.+)$/i.exec(line)?.[1]; + if (http) { + const status = /^The requested URL returned error: (\d{3})$/i.exec(http)?.[1]; + if (status === "401" || status === "403") return "authentication_failed"; + if (status === "404") return "repository_unavailable"; + if (status === "429") return "rate_limited"; + if (status && /^5\d\d$/.test(status)) return "remote_unavailable"; + if (/^(?:URL (?:using bad\/illegal format|rejected)|Port number was not a decimal number)/i.test(http)) return "invalid_remote"; + if (/^Could not resolve host: [^\s]+$/i.test(http)) return "dns_failure"; + if (/^Failed to connect to .+: Connection refused$/i.test(http)) return "connection_refused"; + if (/^Failed to connect to .+: (?:Network is unreachable|No route to host)$/i.test(http)) return "network_unreachable"; + if (/^Recv failure: Connection was reset$/i.test(http)) return "connection_reset"; + if (/^Failed to connect to .+: Connection timed out$|^Operation timed out(?: after \d+ milliseconds with \d+ bytes received)?$/i.test(http)) return "connection_timeout"; + if (/^(?:SSL certificate problem|server certificate verification failed|SSL peer certificate|gnutls_handshake\(\) failed)/i.test(http)) return "tls_failure"; + return undefined; + } + if (/^fatal: Authentication failed for ['"][^'"\r\n]+['"]$/i.test(line) || + /^fatal: could not read (?:Username|Password) for ['"][^'"\r\n]+['"]: terminal prompts disabled$/i.test(line) || + /^remote: (?:Invalid username or (?:password|token)|Permission [^\r\n]+ denied|Support for password authentication was removed|Password authentication is not supported)/i.test(line) || + /^(?:[^\s:@]+@)?[a-z0-9][a-z0-9.-]*: Permission denied \((?:publickey|password|keyboard-interactive)(?:,[a-z-]+)*\)\.?$/i.test(line)) return "authentication_failed"; + if (/^remote: Repository not found\.?$|^fatal: repository ['"][^'"\r\n]+['"] not found$/i.test(line)) return "repository_unavailable"; + if (/^ssh: Could not resolve hostname [^\s]+: (?:Name or service not known|nodename nor servname provided, or not known|Temporary failure in name resolution|No such host is known\.?)$/i.test(line)) return "dns_failure"; + const ssh = /^ssh: connect to host [^\s]+ port \d+: (.+)$/i.exec(line)?.[1]; + if (ssh === "Connection refused") return "connection_refused"; + if (ssh === "Network is unreachable" || ssh === "No route to host") return "network_unreachable"; + if (ssh === "Connection timed out") return "connection_timeout"; + if (/^Connection reset by (?:peer|[^\s]+ port \d+)$/i.test(line)) return "connection_reset"; + return undefined; +} diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index f4199bba621..7c62ed73347 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,46 @@ +import { readGitConnectionFailure } from "./git-connection-failure.js"; +import { + CONFIGURATION_INCOMPLETE_FAILURE_CODE, + ConfigurationIncompleteFailure, + deriveTaskKey, + type WakeupOptions, + mergeCoalescedContextSnapshot, + PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY, + PAPERCLIP_HARNESS_CHECKOUT_KEY, + attestReviewedExternalChatRun, + resolveAcceptedPlanWakeRoutingDecision, + clearInteractionContinuationWakeContext, + buildPaperclipWakePayload, + PAPERCLIP_WAKE_PAYLOAD_KEY, + resolveRunScopedMentionedSkillKeys, + resolveExecutionRunAdapterConfig, + applyRunScopedMentionedSkillKeys, + MANAGED_GITHUB_TOKEN_KEYS, + configuredPaperclipApiBaseUrl, + buildPaperclipRuntimeMcpServers, + createAdapterRuntimeToolAccess, + paperclipApiBaseUrl, + createAdapterRuntimeMcpAccess, + createManagedMcpRunConfig, + revokeHeartbeatRunGatewayTokens, + enrichWakeContextSnapshot, + hasInteractionContinuationWakeContext, + isInteractionResolutionWakePayload, + createHeartbeatRunPreparation, +} from "./heartbeat/run-preparation.js"; +export { + ConfigurationIncompleteFailure, + resolveExecutionRunAdapterConfig, + extractMentionedSkillIdsFromSources, + applyRunScopedMentionedSkillKeys, + revokeHeartbeatRunGatewayTokens, + buildPaperclipRuntimeMcpServers, + createManagedMcpRunConfig, + mergeCoalescedContextSnapshot, + resolveExternalChatWakeProvider, + attestReviewedExternalChatRun, + buildPaperclipWakePayload, +} from "./heartbeat/run-preparation.js"; import { WORKSPACE_VALIDATION_FAILURE_CODE, isWorkspaceValidationFailedRun, @@ -107,7 +150,9 @@ export { buildPaperclipTaskMarkdown } from "./heartbeat/task-markdown.js"; import { preserveWorkspaceRestoreRecoveryMetadataSql } from "./legacy-workspace-restore-recovery.js"; import { preserveWorkspaceRestoreRecoveryMetadata } from "./workspace-restore-recovery-state.js"; import { recordLegacyWorkspaceRestoreFailure } from "./legacy-execution-recovery.js"; -import { CONFIGURED_ENVIRONMENT_KEYS, configuredEnvironmentProjection } from "../vendor/paperclip-runner/index.js"; +import { + configuredEnvironmentProjection, +} from "../vendor/paperclip-runner/index.js"; import { decisionModelService } from "./decision-models.js"; import { activeIssueInteractionCondition } from "./issue-question-context.js"; import { createAgentIdentityRedactor } from "./agent-identity-redaction.js"; @@ -126,7 +171,7 @@ import { AgentDirectoryReuseInvalidatedError, isAgentDirectoryCopy } from "./age import type { PaperclipTurnContext } from "@paperclipai/adapter-utils/server-utils"; import { restoreNativeWorkspaceBestEffort } from "./native-runtime/native-workspace-best-effort.js"; -import { ASSIGNED_MCP_SERVER_NAME } from "./mcp-tool-names.js"; + import { cancellationResultJson, canContinueCancelledRun, readRunCancellation, requestedRunCancellation } from "./run-cancellation.js"; import { withNativeWorkspaceFinalizationOwnership, @@ -140,11 +185,13 @@ import { accountRunCost, createCostAccountingReconciler } from "./run-cost-accou import { createRunUsageRecorder } from "./usage-receipts.js"; import { applyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-result"; import { compareCents, hasWorkspaceRestoreFailure } from "@paperclipai/shared"; -import { externalConversationStateSql, nonIdleSlackIssueCondition } from "./slack-conversation-state.js"; +import { + nonIdleSlackIssueCondition, +} from "./slack-conversation-state.js"; import { settleSlackConversation } from "./slack-conversation-lifecycle.js"; import { toolActionDeliveryService } from "./tool-action-delivery.js"; -import { githubBotConnectionIdsForRun } from "./chat-github-tools.js"; -import { isBrowserUseConnection } from "./browser-use-client.js"; + + import { readQueuedInteractionResponse } from "./queued-interaction-response.js"; import { isConversation, isConversationExecutionWake, isWaitingConversation, prepareConversationTurn, settleConversationTurn } from "./agent-conversations.js"; import { withAdapterExecutionPhase } from "@paperclipai/adapter-utils/execution-phase"; @@ -197,14 +244,15 @@ import { WORKSPACE_GIT_SCAN_ERROR_CODES, } from "./workspace-git-operation-scheduler.js"; -import { initializeRunIdentity, explicitOperatorRunIdentity } from "./run-identity.js"; +import { + initializeRunIdentity, +} from "./run-identity.js"; import { assertDurableChatWakeupReceipt, assertDurableChatWakeupRequest, authorizeFailedChatRunRetryWake, FailedChatRunRetryAuthorizationError, unadmittedChatWakeupCondition, - type DurableChatWakeupRequest, } from "./durable-chat-wakeup.js"; import { prepareHeartbeatGitHubLaunchers } from "./heartbeat-github-launchers.js"; import { @@ -243,16 +291,11 @@ import type { Db } from "@paperclipai/db"; import { AGENT_DEFAULT_MAX_CONCURRENT_RUNS, CHAT_PROVIDERS, - CONNECTION_INTENT_AGENT_GUIDANCE, - CONNECTION_RUNTIME_TOOL_NAMES, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY, ISSUE_DISPOSITION_REPAIR_RETRY_REASON, PROVIDER_QUOTA_MONITOR_SERVICE_NAME, - envBindingSchema, isEnvironmentDriverSupportedForAdapter, - isToolConnectionAttentionHealth, type BillingType, - type ChatProvider, type CostStatus, type EnvironmentLeaseStatus, type ExecutionWorkspace, @@ -261,9 +304,7 @@ import { type IssueExecutionMonitorPolicy, type IssueExecutionMonitorRecoveryPolicy, type RequestConfirmationResult, - type RoutineRevisionSnapshotV1, type RunLivenessState, - type SourceTrustMetadata, } from "@paperclipai/shared"; import { agents, @@ -274,21 +315,13 @@ import { agentWakeupRequests, activityLog, approvals, - assets, chatActions, chatConversations, - chatDeliveries, chatEndpoints, - chatMessageLinks, - companyMemberships, companySkillTestRuns, - companySkillVersions, - companySkills as companySkillsTable, companies, completionContracts, costEvents, - documentAnnotationComments, - documentAnnotationThreads, documentRevisions, environmentLeases, issueDocuments, @@ -296,9 +329,7 @@ import { heartbeatRunEvents, heartbeatRuns, issueApprovals, - issueAttachments, issueComments, - issuePlanDecompositions, issueRecoveryActions, issueRelations, issueThreadInteractions, @@ -307,16 +338,7 @@ import { nativeRunFinalizations, projects, projectWorkspaces, - routineRevisions, - routineRuns, routines, - toolMcpGateways, - toolMcpGatewayTokens, - toolCatalogEntries, - toolConnectionInstalls, - toolConnections, - toolProfileEntries, - toolProfiles, workspaceOperations, } from "@paperclipai/db"; import { conflict, HttpError, notFound } from "../errors.js"; @@ -330,7 +352,6 @@ import { logger } from "../middleware/logger.js"; import { createGitRemoteAuthProvider, resolveManagedGitHubIdentitySelection, - filterResolvedGitHubConnectionsForRun, scrubGitCredentialText, } from "./git-credentials.js"; // Re-exported because heartbeat's workspace surface exposed the scrubber before the @@ -425,9 +446,6 @@ import type { AdapterExecutionResult, AdapterInvocationMeta, AdapterRuntimeEvent, - AdapterRuntimeMcpAccess, - AdapterRuntimeMcpServer, - AdapterRuntimeToolAccess, AdapterSessionCodec, UsageSummary, } from "../adapters/index.js"; @@ -439,12 +457,10 @@ import { asNumber, } from "../adapters/utils.js"; import { - authorizeChatConversationForBoundRun, isExternalChatWaitAuthorizationContention, } from "./native-runtime/chat-attachment-reuse.js"; import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY, - resolveExternalChatQuestionResponse, } from "./native-runtime/external-chat-question-response.js"; import { materializeExternalChatQuestionResponseInput } from "./native-runtime/external-chat-question-response-input.js"; import { @@ -472,7 +488,9 @@ import { buildProcessLossDiagnostic } from "./process-loss-diagnostics.js"; import { collectRunFailureSecretValues, type RunFailureReportOptions } from "./run-failure-diagnostics.js"; import { companySkillService } from "./company-skills.js"; import { budgetService, withCurrentBudgetEnforcement, type BudgetEnforcementScope } from "./budgets.js"; -import { secretService, type MissingRuntimeBinding } from "./secrets.js"; +import { + secretService, +} from "./secrets.js"; import { resolveDefaultAgentWorkspaceDir, } from "../home-paths.js"; @@ -557,8 +575,8 @@ import { } from "./runner-goals.js"; import { resolveChatRunPresentationAuthorizationReason } from "./chat-run-publications.js"; -import { createToolGatewayService } from "./tool-gateway.js"; -import { toolAccessService } from "./tool-access.js"; + + import { visibleIssueCondition } from "./issue-visibility.js"; import { ISSUE_BLOCKERS_RESOLVED_WAKE_REASON } from "./issue-dependency-wakeups.js"; import { @@ -578,7 +596,6 @@ import { refreshIssueContinuationSummary, } from "./issue-continuation-summary.js"; import { - buildDocumentReviewContext, buildPlanReviewContext, } from "./plan-review-context.js"; import { @@ -661,7 +678,6 @@ import { AI_CONNECTION_POOL_WAIT_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_WAKE_REASON, - WAKE_COMMENT_IDS_KEY, isNonAssigneeWorkspaceBusyRetry, extractWakeCommentIds, deriveCommentId, @@ -696,7 +712,6 @@ import { } from "./agent-invokability.js"; import { isHeartbeatWakeOnDemandEnabled } from "./heartbeat-policy.js"; import { - isLowTrustQuarantined, redactQuarantinedBodyForHigherTrust, sanitizeQuarantinedCommentForHigherTrust, } from "./source-trust.js"; @@ -719,10 +734,11 @@ import { selectPaperclipTaskMarkdown, UNMANAGED_BACKGROUND_TASK_LIVENESS_REASON, UNMANAGED_BACKGROUND_TASK_STOP_REASON, - writePaperclipSkillSyncPreference, } from "@paperclipai/adapter-utils/server-utils"; -import { extractSkillMentionIds, isUuidLike } from "@paperclipai/shared"; -import { evaluateCodexCredentialReadiness } from "@paperclipai/adapter-codex-local/server"; +import { + isUuidLike, +} from "@paperclipai/shared"; + import { environmentService } from "./environments.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; import { retryChatControlAdmission } from "./chat-control-admission-retry.js"; @@ -756,7 +772,7 @@ import { import { assertLowTrustRuntimeServicesAllowed, } from "./low-trust-runtime-containment.js"; -import type { TrustPresetResolution } from "./trust-preset-resolver.js"; + import { resolveAndRetainRunTrustPreset } from "./run-trust-preset.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; @@ -795,14 +811,13 @@ const LIVENESS_BOOKKEEPING_ACTIVITY_ACTIONS = [ "cost.reported", ]; const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext"; -const EXTERNAL_ATTACHMENT_OMISSIONS_KEY = "externalAttachmentOmissions"; -const PAPERCLIP_WAKE_PAYLOAD_KEY = "paperclipWake"; + + const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; -const PAPERCLIP_AGENT_MESSAGE_KEY = "paperclipAgentMessage"; -const PAPERCLIP_HARNESS_CHECKOUT_KEY = "paperclipHarnessCheckedOut"; -const PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY = - "paperclipExternalChatExecutionBound"; + + + const DETACHED_PROCESS_ERROR_CODE = "process_detached"; const NATIVE_OWNERSHIP_UNVERIFIED_MESSAGE = "Native execution ownership could not be verified; automatic recovery is blocked"; @@ -889,12 +904,12 @@ function pendingCleanupCapWarnedSql() { )`; } -const MAX_INLINE_WAKE_COMMENTS = 8; -const MAX_INLINE_WAKE_ATTACHMENTS = 20; -const MAX_INLINE_WAKE_COMMENT_BODY_CHARS = 4_000; -const MAX_INLINE_WAKE_COMMENT_BODY_TOTAL_CHARS = 12_000; -const MAX_INLINE_WAKE_ISSUE_DESCRIPTION_CHARS = 12_000; -const MAX_AGENT_SESSION_MESSAGE_CHARS = 12_000; + + + + + + const execFile = promisify(execFileCallback); const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = [ "queued", @@ -945,7 +960,7 @@ export { const INTERACTION_CONTINUATION_INFRA_MAX_ATTEMPTS = 2; const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed"; -const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete"; + const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete"; const EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON = "execution_review_participant_recovery"; @@ -1038,20 +1053,6 @@ interface MaxTurnContinuationPolicy { delayMs: number; } -// Pre-dispatch gate outcome: required secret/env bindings are missing, so the -// run must not be dispatched. Surfaced as a configuration-incomplete blocker -// routed to a human owner instead of N opaque dispatched-then-failed runs. -export class ConfigurationIncompleteFailure extends Error { - code = CONFIGURATION_INCOMPLETE_FAILURE_CODE; - resultJson: Record; - - constructor(message: string, resultJson: Record) { - super(message); - this.name = "ConfigurationIncompleteFailure"; - this.resultJson = resultJson; - } -} - // Build the configuration-incomplete result payload for a workspace base ref // that never resolved to a commit. The setup catch maps this to errorCode // `configuration_incomplete`, so the recovery path routes it to a human owner @@ -1342,22 +1343,7 @@ const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([ ISSUE_BLOCKERS_RESOLVED_WAKE_REASON, "issue_recovery_action_restored", ]); -const ISSUE_RESPONSIBLE_USER_WAKE_REASONS = new Set([ - "issue_assigned", - "issue_checked_out", - "issue_commented", - "issue_comment_mentioned", - "issue_reopened_via_comment", - "issue_blockers_resolved", - "issue_children_completed", - "issue_status_changed", - "issue_tree_restored", - "issue_recovery_action_restored", - "execution_review_requested", - "execution_approval_requested", - "execution_changes_requested", - "approval_approved", -]); + const SESSIONED_LOCAL_ADAPTERS = new Set([ "claude_local", "codex_local", @@ -1443,555 +1429,6 @@ export function getTaskDrainStatus(): { }; } -type RuntimeConfigSecretResolver = Pick< - ReturnType, - | "resolveAdapterConfigForRuntime" - | "resolveEnvBindings" - | "collectMissingRuntimeBindings" - | "collectMissingAdapterConfigRuntimeBindings" ->; - -function formatMissingBindingForOperator( - missing: MissingRuntimeBinding, -): string { - if (missing.bindingType === "user_secret_ref") { - const definitionLabel = missing.userSecretDefinitionName - ? `"${missing.userSecretDefinitionName}"` - : missing.userSecretDefinitionKey - ? `"${missing.userSecretDefinitionKey}"` - : "declared user secret"; - const ownerLabel = missing.responsibleUserId - ? ` for responsible user ${missing.responsibleUserId}` - : ""; - return `user secret ${definitionLabel}${ownerLabel} not available at ${missing.consumerType} ${missing.configPath}`; - } - const secretLabel = missing.secretName - ? `"${missing.secretName}"` - : (missing.secretId ?? "unknown"); - return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`; -} - -function isConfiguredEnvBindingValue(binding: unknown) { - const parsed = envBindingSchema.safeParse(binding); - if (!parsed.success) return false; - const value = parsed.data; - if (typeof value === "string") return value.trim().length > 0; - if (value.type === "plain") return value.value.trim().length > 0; - return true; -} - -const LOW_TRUST_SENSITIVE_ENV_KEY_RE = - /(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i; - -// PAPERCLIP_* env binding policy: -// 1. PAPERCLIP_API_KEY is never accepted from user/adapter/project/routine -// config — the harness-minted run token is the only source. -// 2. A PAPERCLIP_* runtime var the harness assigns for the run (RUN_ID, -// AGENT_ID, wake/workspace vars, ...) always wins over a same-named -// binding; adapters enforce this at env-merge time. -// 3. Any other PAPERCLIP_*-named binding is user data and flows through to -// the run env like any non-prefixed binding. -const FORBIDDEN_ENV_BINDING_KEYS = new Set([ - CONFIGURED_ENVIRONMENT_KEYS, - "PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY", - "PAPERCLIP_RUNNER_NETWORK_ACCESS", - "PAPERCLIP_RUNNER_NETWORK_ROOTS", - "PAPERCLIP_API_KEY", - "PAPERCLIP_GITHUB_AUTH_MODE", - "PAPERCLIP_GITHUB_HOST_HOME", - "PAPERCLIP_GIT_METADATA_ROOTS", - "PAPERCLIP_GITHUB_BROKER_TOKEN", - "PAPERCLIP_GITHUB_BROKER_URL", - "PAPERCLIP_GITHUB_BRIDGE_TOKEN", - "PAPERCLIP_GITHUB_LAUNCHER_DIR", -]); -const MANAGED_GITHUB_TOKEN_KEYS = new Set([ - "GH_TOKEN", - "GITHUB_TOKEN", - "GH_ENTERPRISE_TOKEN", - "GITHUB_ENTERPRISE_TOKEN", - "PAPERCLIP_GIT_TOKEN", -]); - -function stripForbiddenEnvBindings( - envValue: unknown, - managedGitHubCredentials = false, -): Record | null { - const record = parseObject(envValue); - const filtered = Object.fromEntries( - Object.entries(record).filter( - ([key]) => - !FORBIDDEN_ENV_BINDING_KEYS.has(key.toUpperCase()) && - !(managedGitHubCredentials && MANAGED_GITHUB_TOKEN_KEYS.has(key)), - ), - ); - return Object.keys(filtered).length > 0 ? filtered : null; -} - -function stripForbiddenEnvFromAdapterConfig( - config: Record, - managedGitHubCredentials = false, -): Record { - if (!Object.prototype.hasOwnProperty.call(config, "env")) return config; - return { - ...config, - env: stripForbiddenEnvBindings(config.env, managedGitHubCredentials) ?? {}, - }; -} - -function assertLowTrustEnvConfigAllowed(envValue: unknown, source: string) { - const record = stripForbiddenEnvBindings(envValue); - if (!record) return; - for (const [key, rawBinding] of Object.entries(record)) { - const parsed = envBindingSchema.safeParse(rawBinding); - if (!parsed.success) continue; - const binding = parsed.data; - const isPlainBinding = - typeof binding === "string" || - (typeof binding === "object" && - binding !== null && - binding.type === "plain"); - if (isPlainBinding && LOW_TRUST_SENSITIVE_ENV_KEY_RE.test(key)) { - throw new HttpError( - 422, - `Low-trust execution cannot use inline sensitive env value ${source}.${key}`, - { - code: "low_trust_inline_sensitive_env_denied", - }, - ); - } - } -} - -export async function resolveExecutionRunAdapterConfig(input: { - managedAiCredentials?: boolean; - companyId: string; - agentId?: string | null; - adapterType?: string | null; - issueId?: string | null; - heartbeatRunId?: string | null; - responsibleUserId?: string | null; - environmentId?: string | null; - environmentEnv?: unknown; - environmentDriver?: string | null; - projectId?: string | null; - routineId?: string | null; - executionRunConfig: Record; - projectEnv: unknown; - routineEnv?: unknown; - secretsSvc: RuntimeConfigSecretResolver; - trustPreset?: TrustPresetResolution; - requiredScopedEnvBinding?: { - keys: string[]; - consumerScopes: Array<"agent" | "project">; - reason: string; - remediation: string; - }; - /** Managed GitHub tokens are resolved only when operations start. */ - managedGitHubCredentials?: boolean; - /** Audited class-3 values resolved by an internal credential broker. */ - trustedEnvProjection?: Record; - trustedEnvSecretKeys?: string[]; -}) { - const executionRunConfig = stripForbiddenEnvFromAdapterConfig( - input.executionRunConfig, - input.managedGitHubCredentials, - ); - const environmentEnv = stripForbiddenEnvBindings( - input.environmentEnv, - input.managedGitHubCredentials, - ); - const projectEnv = stripForbiddenEnvBindings( - input.projectEnv, - input.managedGitHubCredentials, - ); - const routineEnv = stripForbiddenEnvBindings( - input.routineEnv, - input.managedGitHubCredentials, - ); - const agentEnv = parseObject(executionRunConfig.env); - const lowTrustAllowedBindingIds = - input.trustPreset?.kind === "low_trust_review" - ? (input.trustPreset.boundary.allowedSecretBindingIds ?? []) - : undefined; - const allowTrustedEnvProjection = - input.trustPreset?.kind !== "low_trust_review"; - if (input.trustPreset?.kind === "low_trust_review") { - assertLowTrustEnvConfigAllowed(environmentEnv, "environment.env"); - assertLowTrustEnvConfigAllowed(executionRunConfig.env, "agent.env"); - assertLowTrustEnvConfigAllowed(projectEnv, "project.env"); - assertLowTrustEnvConfigAllowed(routineEnv, "routine.env"); - } - const requiredScopedEnvBinding = input.requiredScopedEnvBinding ?? null; - const requiredScopedBindingsConfigured = requiredScopedEnvBinding - ? requiredScopedEnvBinding.keys.some( - (key) => - (allowTrustedEnvProjection && - typeof input.trustedEnvProjection?.[key] === "string") || - (requiredScopedEnvBinding.consumerScopes.includes("agent") && - isConfiguredEnvBindingValue(agentEnv[key])) || - (requiredScopedEnvBinding.consumerScopes.includes("project") && - isConfiguredEnvBindingValue(projectEnv?.[key])), - ) - : false; - if (requiredScopedEnvBinding && !requiredScopedBindingsConfigured) { - throw new ConfigurationIncompleteFailure( - `configuration incomplete: ${requiredScopedEnvBinding.remediation}`, - { - configurationIncomplete: { - reason: requiredScopedEnvBinding.reason, - companyId: input.companyId, - agentId: input.agentId ?? null, - issueId: input.issueId ?? null, - projectId: input.projectId ?? null, - routineId: input.routineId ?? null, - requiredEnvKeys: requiredScopedEnvBinding.keys, - requiredScopes: requiredScopedEnvBinding.consumerScopes, - missingBindings: [], - }, - }, - ); - } - // Pre-dispatch binding-validation gate: detect declared secret refs that have - // no binding before resolving any secret value. Missing bindings short-circuit - // to a configuration-incomplete blocker routed to a human owner instead of a - // dispatched-then-failed run (which previously surfaced as opaque setup_failed). - if (typeof input.secretsSvc.collectMissingRuntimeBindings === "function") { - const missingBindings: MissingRuntimeBinding[] = []; - if (environmentEnv && input.environmentId) { - missingBindings.push( - ...(await input.secretsSvc.collectMissingRuntimeBindings( - input.companyId, - environmentEnv, - { - consumerType: "environment", - consumerId: input.environmentId, - responsibleUserId: input.responsibleUserId ?? null, - }, - )), - ); - } - if (input.agentId) { - missingBindings.push( - ...(await input.secretsSvc.collectMissingRuntimeBindings( - input.companyId, - parseObject(executionRunConfig.env), - { - consumerType: "agent", - consumerId: input.agentId, - responsibleUserId: input.responsibleUserId ?? null, - }, - )), - ); - if ( - typeof input.secretsSvc.collectMissingAdapterConfigRuntimeBindings === - "function" - ) { - missingBindings.push( - ...(await input.secretsSvc.collectMissingAdapterConfigRuntimeBindings( - input.companyId, - executionRunConfig, - input.adapterType ?? null, - { - consumerType: "agent", - consumerId: input.agentId, - responsibleUserId: input.responsibleUserId ?? null, - }, - )), - ); - } - } - if (projectEnv && input.projectId) { - missingBindings.push( - ...(await input.secretsSvc.collectMissingRuntimeBindings( - input.companyId, - projectEnv, - { - consumerType: "project", - consumerId: input.projectId, - responsibleUserId: input.responsibleUserId ?? null, - }, - )), - ); - } - if (routineEnv && input.routineId) { - missingBindings.push( - ...(await input.secretsSvc.collectMissingRuntimeBindings( - input.companyId, - routineEnv, - { - consumerType: "routine", - consumerId: input.routineId, - responsibleUserId: input.responsibleUserId ?? null, - }, - )), - ); - } - if (requiredScopedEnvBinding) { - const requiredEnvKeys = new Set(requiredScopedEnvBinding.keys); - const requiredScopes = new Set(requiredScopedEnvBinding.consumerScopes); - const requiredMissingBindings = missingBindings.filter( - (binding) => - requiredScopes.has(binding.consumerType as "agent" | "project") && - requiredEnvKeys.has(binding.envKey), - ); - if (requiredMissingBindings.length > 0) { - const detail = requiredMissingBindings - .map(formatMissingBindingForOperator) - .join("; "); - throw new ConfigurationIncompleteFailure( - `configuration incomplete: ${requiredScopedEnvBinding.remediation}; ${detail}`, - { - configurationIncomplete: { - reason: requiredScopedEnvBinding.reason, - companyId: input.companyId, - agentId: input.agentId ?? null, - issueId: input.issueId ?? null, - projectId: input.projectId ?? null, - routineId: input.routineId ?? null, - requiredEnvKeys: requiredScopedEnvBinding.keys, - requiredScopes: requiredScopedEnvBinding.consumerScopes, - missingBindings: requiredMissingBindings, - }, - }, - ); - } - } - if (missingBindings.length > 0) { - const detail = missingBindings - .map(formatMissingBindingForOperator) - .join("; "); - throw new ConfigurationIncompleteFailure( - `configuration incomplete: ${detail}`, - { - configurationIncomplete: { - reason: "secret_binding_missing", - companyId: input.companyId, - agentId: input.agentId ?? null, - issueId: input.issueId ?? null, - projectId: input.projectId ?? null, - routineId: input.routineId ?? null, - missingBindings, - }, - }, - ); - } - } - const environmentEnvResolution = environmentEnv - ? await input.secretsSvc.resolveEnvBindings( - input.companyId, - environmentEnv, - input.environmentId - ? { - consumerType: "environment", - consumerId: input.environmentId, - actorType: "agent", - actorId: input.agentId ?? null, - responsibleUserId: input.responsibleUserId ?? null, - issueId: input.issueId ?? null, - heartbeatRunId: input.heartbeatRunId ?? null, - ...(lowTrustAllowedBindingIds !== undefined - ? { allowedBindingIds: lowTrustAllowedBindingIds } - : {}), - } - : undefined, - ) - : { env: {}, secretKeys: new Set(), manifest: [] }; - const { - config: resolvedConfig, - secretKeys, - manifest, - } = await input.secretsSvc.resolveAdapterConfigForRuntime( - input.companyId, - executionRunConfig, - input.agentId - ? { - consumerType: "agent", - consumerId: input.agentId, - actorType: "agent", - actorId: input.agentId, - responsibleUserId: input.responsibleUserId ?? null, - issueId: input.issueId ?? null, - heartbeatRunId: input.heartbeatRunId ?? null, - ...(lowTrustAllowedBindingIds !== undefined - ? { allowedBindingIds: lowTrustAllowedBindingIds } - : {}), - } - : undefined, - { adapterType: input.adapterType ?? null }, - ); - if (Object.keys(environmentEnvResolution.env).length > 0) { - resolvedConfig.env = { - ...environmentEnvResolution.env, - ...parseObject(resolvedConfig.env), - }; - for (const key of environmentEnvResolution.secretKeys) { - secretKeys.add(key); - } - } - const projectEnvResolution = projectEnv - ? await input.secretsSvc.resolveEnvBindings( - input.companyId, - projectEnv, - input.projectId - ? { - consumerType: "project", - consumerId: input.projectId, - actorType: "agent", - actorId: input.agentId ?? null, - responsibleUserId: input.responsibleUserId ?? null, - issueId: input.issueId ?? null, - heartbeatRunId: input.heartbeatRunId ?? null, - ...(lowTrustAllowedBindingIds !== undefined - ? { allowedBindingIds: lowTrustAllowedBindingIds } - : {}), - } - : undefined, - ) - : { env: {}, secretKeys: new Set(), manifest: [] }; - if (Object.keys(projectEnvResolution.env).length > 0) { - resolvedConfig.env = { - ...parseObject(resolvedConfig.env), - ...projectEnvResolution.env, - }; - for (const key of projectEnvResolution.secretKeys) { - secretKeys.add(key); - } - } - const routineEnvResolution = routineEnv - ? await input.secretsSvc.resolveEnvBindings( - input.companyId, - routineEnv, - input.routineId - ? { - consumerType: "routine", - consumerId: input.routineId, - actorType: "agent", - actorId: input.agentId ?? null, - responsibleUserId: input.responsibleUserId ?? null, - issueId: input.issueId ?? null, - heartbeatRunId: input.heartbeatRunId ?? null, - ...(lowTrustAllowedBindingIds !== undefined - ? { allowedBindingIds: lowTrustAllowedBindingIds } - : {}), - } - : undefined, - ) - : { env: {}, secretKeys: new Set(), manifest: [] }; - if (Object.keys(routineEnvResolution.env).length > 0) { - resolvedConfig.env = { - ...parseObject(resolvedConfig.env), - ...routineEnvResolution.env, - }; - for (const key of routineEnvResolution.secretKeys) { - secretKeys.add(key); - } - } - if ( - allowTrustedEnvProjection && - input.trustedEnvProjection && - Object.keys(input.trustedEnvProjection).length > 0 - ) { - resolvedConfig.env = { - ...parseObject(resolvedConfig.env), - ...input.trustedEnvProjection, - }; - for (const key of input.trustedEnvSecretKeys ?? []) secretKeys.add(key); - } - // Pre-dispatch credential gate for codex_local: a managed Codex home with no - // usable auth.json and an empty OPENAI_API_KEY would dispatch a run that - // immediately fails with "no Codex credentials provisioned" (adapter_failed), - // making a configuration problem look like a runtime failure. Surface it as a - // configuration-incomplete blocker instead, naming the missing credential - // action and owner without leaking any secret value. This runs after secret - // resolution so a per-agent OPENAI_API_KEY (plain or resolved secret) counts - // as satisfying the credential. It shares the exact readiness predicate the - // adapter uses at execute time, so the two cannot drift. - // - // Sandbox-destined runs are exempt: the sandbox image may carry its own - // Codex login (`~/.codex/auth.json` baked in at image setup), which only the - // adapter can probe once the sandbox is up — and on managed cloud hosts a - // host-side login never exists at all. The adapter's execute-time gate - // remains the authority there; it probes the sandbox before failing. - if ( - !input.managedAiCredentials && (input.adapterType ?? null) === "codex_local" && - (input.environmentDriver ?? null) !== "sandbox" - ) { - const resolvedEnv = parseObject(resolvedConfig.env); - const readiness = await evaluateCodexCredentialReadiness({ - env: process.env, - companyId: input.companyId, - configuredCodexHome: readNonEmptyString(resolvedEnv.CODEX_HOME), - configuredApiKey: readNonEmptyString(resolvedEnv.OPENAI_API_KEY), - }); - if (readiness.managed && !readiness.ready) { - throw new ConfigurationIncompleteFailure( - `configuration incomplete: no Codex credentials available for managed home "${readiness.effectiveHome}". ` + - `Sign in to Codex on the host with a ChatGPT subscription, or bind a per-agent OPENAI_API_KEY secret for this agent.`, - { - configurationIncomplete: { - reason: "codex_credentials_missing", - companyId: input.companyId, - agentId: input.agentId ?? null, - issueId: input.issueId ?? null, - projectId: input.projectId ?? null, - routineId: input.routineId ?? null, - responsibleUserId: input.responsibleUserId ?? null, - adapterType: "codex_local", - requiredEnvKeys: ["OPENAI_API_KEY"], - effectiveCodexHome: readiness.effectiveHome, - missingBindings: [], - }, - }, - ); - } - } - return { - resolvedConfig, - // Capture resolved task values before provider credential injection or host - // inheritance. Native-only launch limits are checked at native dispatch. - configuredTaskEnvironment: Object.fromEntries( - Object.entries(parseObject(resolvedConfig.env)).filter((entry): entry is [string, string] => typeof entry[1] === "string"), - ), - secretKeys, - secretManifest: [ - ...(environmentEnvResolution.manifest ?? []), - ...(manifest ?? []), - ...(projectEnvResolution.manifest ?? []), - ...(routineEnvResolution.manifest ?? []), - ], - }; -} - -export function extractMentionedSkillIdsFromSources( - sources: Array, -): string[] { - const mentionedIds = new Set(); - for (const source of sources) { - if (typeof source !== "string" || source.length === 0) continue; - for (const skillId of extractSkillMentionIds(source)) { - if (!isUuidLike(skillId)) continue; - mentionedIds.add(skillId); - } - } - return [...mentionedIds]; -} - -export function applyRunScopedMentionedSkillKeys( - config: Record, - skillKeys: string[], -): Record { - const normalizedSkillKeys = Array.from( - new Set(skillKeys.map((value) => value.trim()).filter(Boolean)), - ); - if (normalizedSkillKeys.length === 0) return config; - - const existingPreference = readPaperclipSkillSyncPreference(config); - return writePaperclipSkillSyncPreference(config, [ - ...existingPreference.desiredSkillEntries, - ...normalizedSkillKeys, - ]); -} export function computeBoundedTransientHeartbeatRetrySchedule( attempt: number, @@ -2014,60 +1451,6 @@ export function computeBoundedTransientHeartbeatRetrySchedule( }; } -async function resolveRunScopedMentionedSkillKeys(input: { - db: Db; - companyId: string; - issueId: string | null; -}): Promise { - if (!input.issueId) return []; - - const issue = await input.db - .select({ - title: issues.title, - description: issues.description, - }) - .from(issues) - .where( - and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)), - ) - .then((rows) => rows[0] ?? null); - if (!issue) return []; - - const comments = await input.db - .select({ body: issueComments.body }) - .from(issueComments) - .where( - and( - eq(issueComments.issueId, input.issueId), - eq(issueComments.companyId, input.companyId), - isNull(issueComments.deletedAt), - ), - ); - const mentionedSkillIds = extractMentionedSkillIdsFromSources([ - issue.title, - issue.description ?? "", - ...comments.map((comment) => comment.body), - ]); - if (mentionedSkillIds.length === 0) return []; - - const skillRows = await input.db - .select({ - id: companySkillsTable.id, - key: companySkillsTable.key, - }) - .from(companySkillsTable) - .where( - and( - eq(companySkillsTable.companyId, input.companyId), - inArray(companySkillsTable.id, mentionedSkillIds), - ), - ); - const skillKeyById = new Map(skillRows.map((row) => [row.id, row.key])); - return mentionedSkillIds - .map((skillId) => skillKeyById.get(skillId) ?? null) - .filter((skillKey): skillKey is string => Boolean(skillKey)); -} - export function leaseReleaseStatusForRunStatus( status: string | null | undefined, ): Extract { @@ -2485,45 +1868,6 @@ function normalizeMaxConcurrentRuns(value: unknown) { ); } -interface WakeupOptions { - /** Set only by authenticated board wake routes; never copied from caller payloads. */ - manualUserWake?: boolean; - /** Internal resume of a queue with persisted board interruption intent. */ - queuedCommentInterruptId?: string; - /** Internal delivery of an existing undelivered user comment. */ - queuedCommentRequestId?: string; - /** Exact failed run selected by an authenticated board Retry request. */ - failedRunId?: string | null; - durableChatRequest?: DurableChatWakeupRequest; - /** Server-owned Dot admission receipt; never accepted from API payloads. */ - durableDotRequest?: { - id: string; - companyId: string; - agentId: string; - issueId: string; - requestId: string; - idempotencyKey: string; - requestedAt: Date; - }; - source?: "timer" | "assignment" | "on_demand" | "automation"; - triggerDetail?: "manual" | "ping" | "callback" | "system"; - reason?: string | null; - payload?: Record | null; - idempotencyKey?: string | null; - requestedByActorType?: "user" | "agent" | "system"; - requestedByActorId?: string | null; - contextSnapshot?: Record; - issueStateGuard?: { - statuses: string[]; - assigneeAgentId: string; - statusVersion?: number; - monitorNextCheckAt?: string; - monitorWakeRequestedAt?: string; - }; - /** Keep causally distinct external chat continuations out of an existing run. */ - allowRunCoalescing?: boolean; -} - type UsageTotals = { inputTokens: number; cachedInputTokens: number; @@ -2575,703 +1919,6 @@ function parseNativeSessionGoalControl( }; } -function sanitizeAgentSessionMessageText(value: unknown): string | null { - const text = readNonEmptyString(value); - if (!text) return null; - const redacted = redactSensitiveText(text).slice( - 0, - MAX_AGENT_SESSION_MESSAGE_CHARS, - ); - return redacted.trim().length > 0 ? redacted : null; -} - -type ManagedMcpGatewayRunConfig = { - version: 1; - managedMcpOnly: boolean; - gateways: Array<{ - id: string; - name: string; - endpointPath: string; - bearerToken: string; - tokenPrefix: string; - }>; -}; - -function configuredPaperclipApiBaseUrl(): string | null { - const configured = readNonEmptyString(process.env.PAPERCLIP_API_URL); - return configured - ? configured.replace(/\/+$/, "").replace(/\/api$/, "") - : null; -} - -function paperclipApiBaseUrl(): string { - const configured = configuredPaperclipApiBaseUrl(); - if (!configured) { - throw new Error( - "PAPERCLIP_API_URL is required to deliver managed runtime MCP servers", - ); - } - return configured; -} - -export async function revokeHeartbeatRunGatewayTokens(input: { - db: Db; - companyId: string; - runId: string; -}): Promise { - const now = new Date(); - await input.db - .update(toolMcpGatewayTokens) - .set({ revokedAt: now, updatedAt: now }) - .where( - and( - eq(toolMcpGatewayTokens.companyId, input.companyId), - eq(toolMcpGatewayTokens.subjectType, "heartbeat_run"), - eq(toolMcpGatewayTokens.subjectId, input.runId), - isNull(toolMcpGatewayTokens.revokedAt), - ), - ); -} - -export async function buildPaperclipRuntimeMcpServers(input: { - db: Db; - agent: Pick; - runId: string; - expectedAssignmentDigest?: string | null; -}): Promise { - const access = toolAccessService(input.db); - const effective = await access.getEffectiveProfilesForAgent( - input.agent.companyId, - input.agent.id, - ); - const [runIdentity] = await input.db - .select({ - responsibleUserId: heartbeatRuns.responsibleUserId, - activeIdentityContextId: heartbeatRuns.activeIdentityContextId, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.runId), - eq(heartbeatRuns.companyId, input.agent.companyId), - eq(heartbeatRuns.agentId, input.agent.id), - ), - ) - .limit(1); - const resolvedInstalledConnections = runIdentity?.activeIdentityContextId - ? effective.installedConnections - : await filterResolvedGitHubConnectionsForRun({ - db: input.db, - companyId: input.agent.companyId, - agentId: input.agent.id, - responsibleUserId: runIdentity?.responsibleUserId ?? null, - connections: effective.installedConnections, - }); - const permittedConnectionIds = new Set([ - ...effective.entries - .filter((entry) => entry.effect === "include" && entry.connectionId) - .map((entry) => entry.connectionId!), - ...effective.allowedTools.map((tool) => tool.connectionId), - ]); - const allInstalledConnectionIds = new Set( - effective.installedConnections.map((connection) => connection.id), - ); - const permittedConnections = - permittedConnectionIds.size > 0 - ? await input.db - .select({ - id: toolConnections.id, - name: toolConnections.name, - transport: toolConnections.transport, - config: toolConnections.config, - }) - .from(toolConnections) - .where( - and( - eq(toolConnections.companyId, input.agent.companyId), - inArray(toolConnections.id, [...permittedConnectionIds]), - ), - ) - : []; - const permittedNotInstalledConnections = permittedConnections - .filter( - (connection) => - (connection.transport === "mcp_remote" || - connection.transport === "local_stdio" || isBrowserUseConnection(connection)) && - !allInstalledConnectionIds.has(connection.id), - ) - .map(({ id, name }) => ({ id, name })) - .sort((a, b) => a.name.localeCompare(b.name)); - const githubBotConnectionIds = await githubBotConnectionIdsForRun(input.db, input.agent.companyId, input.agent.id, input.runId); - const assignedConnections = resolvedInstalledConnections.filter( - (connection) => - permittedConnectionIds.has(connection.id) && - connection.status === "active" && - connection.enabled && - ((Boolean(runIdentity?.activeIdentityContextId) && - (connection.config?.sourceTemplateKey === "github" || - connection.transportConfig?.sourceTemplateKey === "github")) || - connection.credentialPolicy === "per_user" || - !isToolConnectionAttentionHealth(connection.healthStatus)) && - (connection.transport === "mcp_remote" || - connection.transport === "local_stdio" || isBrowserUseConnection(connection) || githubBotConnectionIds.has(connection.id)), - ); - const assignedConnectionIds = new Set( - assignedConnections.map((connection) => connection.id), - ); - const assignedTools = effective.allowedTools.filter((tool) => - assignedConnectionIds.has(tool.connectionId), - ); - const service = createToolGatewayService(input.db); - if (assignedConnections.length === 0) { - await service.recordRuntimeMcpDeliveryDiagnostic({ - companyId: input.agent.companyId, - agentId: input.agent.id, - runId: input.runId, - permittedNotInstalledConnections, - }); - return []; - } - const assignment = { - version: 1, - agentId: input.agent.id, - connections: assignedConnections.map((connection) => connection.id).sort(), - tools: assignedTools.map((tool) => tool.id).sort(), - }; - const assignmentDigest = createHash("sha256") - .update(JSON.stringify(assignment)) - .digest("hex"); - // Native runs may lose access after their immutable context is captured, but - // they must never gain a new or changed assignment during dispatch. - if ( - input.expectedAssignmentDigest !== undefined && - input.expectedAssignmentDigest !== assignmentDigest - ) { - return []; - } - const profileKey = `native:${input.agent.id}:${assignmentDigest}`; - let [profile] = await input.db - .select() - .from(toolProfiles) - .where( - and( - eq(toolProfiles.companyId, input.agent.companyId), - eq(toolProfiles.profileKey, profileKey), - ), - ) - .limit(1); - - if (!profile) { - const fullConnectionIds = new Set( - effective.entries - .filter( - (entry) => - entry.effect === "include" && - entry.selectorType === "connection" && - entry.connectionId, - ) - .map((entry) => entry.connectionId!), - ); - const entries = [ - ...assignedConnections - .filter((connection) => fullConnectionIds.has(connection.id)) - .map((connection) => ({ - selectorType: "connection" as const, - effect: "include" as const, - applicationId: connection.applicationId, - connectionId: connection.id, - })), - ...assignedTools - .filter((tool) => !fullConnectionIds.has(tool.connectionId)) - .map((tool) => ({ - selectorType: "catalog_entry" as const, - effect: "include" as const, - applicationId: tool.applicationId, - connectionId: tool.connectionId, - catalogEntryId: tool.id, - })), - ]; - // The 250-entry limit bounds a public profile-edit request, not the - // effective assignment assembled from existing profiles. Keep every exact - // selector here: truncating or replacing them with connection-wide grants - // would either lose assigned tools or authorize tools outside this snapshot. - try { - const created = await access.createProfile(input.agent.companyId, { - profileKey, - name: `Native ${input.agent.id.slice(0, 8)} ${assignmentDigest.slice(0, 12)}`, - description: "Immutable Paperclip Runner MCP assignment profile.", - status: "active", - defaultAction: "deny", - metadata: { - source: "paperclip_runner", - agentId: input.agent.id, - assignmentDigest, - }, - entries, - }); - [profile] = await input.db - .select() - .from(toolProfiles) - .where(eq(toolProfiles.id, created.id)) - .limit(1); - } catch (error) { - [profile] = await input.db - .select() - .from(toolProfiles) - .where( - and( - eq(toolProfiles.companyId, input.agent.companyId), - eq(toolProfiles.profileKey, profileKey), - ), - ) - .limit(1); - if (!profile) throw error; - } - } - - if (profile!.metadata?.source !== "paperclip_runner" || profile!.metadata?.agentId !== input.agent.id || - profile!.metadata?.assignmentDigest !== assignmentDigest) { - throw new Error("Invalid native runtime profile provenance"); - } - - let [gateway] = ( - await input.db - .select() - .from(toolMcpGateways) - .where( - and( - eq(toolMcpGateways.companyId, input.agent.companyId), - eq(toolMcpGateways.status, "active"), - isNull(toolMcpGateways.archivedAt), - ), - ) - ).filter( - (candidate) => - candidate.metadata?.nativeRuntimeAssignmentDigest === assignmentDigest && - candidate.metadata?.agentId === input.agent.id && - candidate.profileId === profile!.id && - (!candidate.agentId || candidate.agentId === input.agent.id), - ); - // Gateways created before the agent binding existed have a null agentId, and - // named-gateway auth only rejects another agent's run token when it is set. - // The assignment digest includes the agent id, so a reused gateway is this - // agent's own: bind it here instead of waiting for the assignment to change. - if (gateway && !gateway.agentId) { - await input.db - .update(toolMcpGateways) - .set({ agentId: input.agent.id, contextScopeType: "agent", contextScopeId: input.agent.id, updatedAt: new Date() }) - .where( - and( - eq(toolMcpGateways.id, gateway.id), - eq(toolMcpGateways.companyId, input.agent.companyId), - isNull(toolMcpGateways.agentId), - ), - ); - [gateway] = await input.db - .select() - .from(toolMcpGateways) - .where(eq(toolMcpGateways.id, gateway.id)) - .limit(1); - } - if (!gateway) { - const slug = `native-${input.agent.id.replaceAll("-", "").slice(0, 12)}-${assignmentDigest.slice(0, 16)}`; - try { - const created = await service.createNamedGateway({ - companyId: input.agent.companyId, - body: { - name: `Native ${input.agent.name} ${assignmentDigest.slice(0, 8)}`, - slug, - description: "Run-scoped Paperclip Runner MCP gateway.", - profileId: profile!.id, - defaultProfileMode: "gateway_only", - agentId: input.agent.id, - contextScopeType: "agent", - contextScopeId: input.agent.id, - metadata: { - nativeRuntimeAssignmentDigest: assignmentDigest, - agentId: input.agent.id, - }, - }, - actor: { agentId: input.agent.id }, - }); - [gateway] = await input.db - .select() - .from(toolMcpGateways) - .where(eq(toolMcpGateways.id, created.id)) - .limit(1); - } catch (error) { - [gateway] = await input.db - .select() - .from(toolMcpGateways) - .where( - and( - eq(toolMcpGateways.companyId, input.agent.companyId), - eq(toolMcpGateways.slug, slug), - ), - ) - .limit(1); - if (!gateway) throw error; - } - } - - if (gateway!.agentId !== input.agent.id || gateway!.profileId !== profile!.id || - gateway!.metadata?.agentId !== input.agent.id || - gateway!.metadata?.nativeRuntimeAssignmentDigest !== assignmentDigest) { - throw new Error("Invalid native runtime gateway provenance"); - } - - const token = await service.createNamedGatewayToken({ - companyId: input.agent.companyId, - gatewayId: gateway!.id, - body: { - name: `Run ${input.runId.slice(0, 8)}`, - subjectType: "heartbeat_run", - subjectId: input.runId, - clientLabel: `${input.agent.name} heartbeat run`, - ownerNote: `Short-lived runtime MCP token for heartbeat run ${input.runId}.`, - allowedActions: ["tools/list", "tools/call"], - expiresAt: new Date(Date.now() + 60 * 60 * 1_000), - }, - actor: { agentId: input.agent.id }, - }); - - return [ - { - name: ASSIGNED_MCP_SERVER_NAME, - url: `${paperclipApiBaseUrl()}/mcp/gateways/${gateway!.gatewayPublicId}`, - token: token.token, - connectionId: `assignment:${assignmentDigest}`, - }, - ]; -} -function createAdapterRuntimeMcpAccess( - servers: AdapterRuntimeMcpServer[], -): AdapterRuntimeMcpAccess | undefined { - if (servers.length === 0) return undefined; - const snapshot = servers.map((server) => Object.freeze({ ...server })); - return Object.freeze({ - getServers: () => snapshot.map((server) => ({ ...server })), - }); -} - -function createAdapterRuntimeToolAccess(input: { - agentId: string; - companyId: string; - runId: string; - responsibleUserId: string | null; -}): AdapterRuntimeToolAccess | undefined { - if (!input.responsibleUserId) return undefined; - const minted = createRuntimeToolsToken({ - agentId: input.agentId, - companyId: input.companyId, - runId: input.runId, - responsibleUserId: input.responsibleUserId, - }); - if (!minted) return undefined; - // The normal server bootstrap always exports PAPERCLIP_API_URL. Some service - // tests invoke heartbeat execution without booting an HTTP server, however; - // in that context there is no reachable endpoint to advertise and runtime - // tools should simply remain unavailable instead of failing the run. - const baseUrl = configuredPaperclipApiBaseUrl(); - if (!baseUrl) return undefined; - return Object.freeze({ - version: 1, - guidance: CONNECTION_INTENT_AGENT_GUIDANCE, - mcpEndpoint: `${baseUrl}/mcp/runtime-tools`, - rest: { - connectionsSearch: `${baseUrl}/runtime-tools/connections/search`, - connectionRequest: `${baseUrl}/runtime-tools/connections/request`, - }, - bearerToken: minted.token, - expiresAt: minted.expiresAt, - tools: CONNECTION_RUNTIME_TOOL_NAMES, - }); -} - -const MANAGED_MCP_LOCAL_ADAPTERS = new Set(["codex_local"]); - -function adapterSupportsManagedMcpConfig(adapterType: string): boolean { - return MANAGED_MCP_LOCAL_ADAPTERS.has(adapterType); -} - -function gatewayAppliesToRun(input: { - gateway: typeof toolMcpGateways.$inferSelect; - agentId: string; - projectId: string | null; - issueId: string | null; -}): boolean { - const { gateway, agentId, projectId, issueId } = input; - if (gateway.agentId && gateway.agentId !== agentId) return false; - if (gateway.projectId && gateway.projectId !== projectId) return false; - if (gateway.issueId && gateway.issueId !== issueId) return false; - if ( - gateway.contextScopeType === "agent" && - gateway.contextScopeId && - gateway.contextScopeId !== agentId - ) - return false; - if ( - gateway.contextScopeType === "project" && - gateway.contextScopeId && - gateway.contextScopeId !== projectId - ) - return false; - if ( - gateway.contextScopeType === "issue" && - gateway.contextScopeId && - gateway.contextScopeId !== issueId - ) - return false; - return true; -} - -async function gatewayConnectionIds(input: { - db: Db; - companyId: string; - gateway: typeof toolMcpGateways.$inferSelect; -}): Promise> { - const managedRuntimeConnectionId = readNonEmptyString( - input.gateway.metadata?.managedRuntimeConnectionId, - ); - if (managedRuntimeConnectionId) return new Set([managedRuntimeConnectionId]); - - const [profile, entries, catalog, connections] = await Promise.all([ - input.db - .select({ defaultAction: toolProfiles.defaultAction }) - .from(toolProfiles) - .where( - and( - eq(toolProfiles.companyId, input.companyId), - eq(toolProfiles.id, input.gateway.profileId), - ), - ) - .then((rows) => rows[0] ?? null), - input.db - .select() - .from(toolProfileEntries) - .where( - and( - eq(toolProfileEntries.companyId, input.companyId), - eq(toolProfileEntries.profileId, input.gateway.profileId), - ), - ), - input.db - .select({ - id: toolCatalogEntries.id, - connectionId: toolCatalogEntries.connectionId, - applicationId: toolCatalogEntries.applicationId, - toolName: toolCatalogEntries.toolName, - riskLevel: toolCatalogEntries.riskLevel, - }) - .from(toolCatalogEntries) - .where( - and( - eq(toolCatalogEntries.companyId, input.companyId), - eq(toolCatalogEntries.status, "active"), - ), - ), - input.db - .select({ - id: toolConnections.id, - applicationId: toolConnections.applicationId, - }) - .from(toolConnections) - .where(eq(toolConnections.companyId, input.companyId)), - ]); - if (!profile) return new Set(); - if (profile.defaultAction === "allow") - return new Set(catalog.map((entry) => entry.connectionId)); - - const connectionIds = new Set(); - for (const entry of entries) { - if (entry.effect !== "include") continue; - if (entry.connectionId) connectionIds.add(entry.connectionId); - if (entry.applicationId) { - for (const connection of connections) { - if (connection.applicationId === entry.applicationId) - connectionIds.add(connection.id); - } - } - for (const catalogEntry of catalog) { - if ( - (entry.catalogEntryId && entry.catalogEntryId === catalogEntry.id) || - (entry.toolName && entry.toolName === catalogEntry.toolName) || - (entry.riskLevel && entry.riskLevel === catalogEntry.riskLevel) - ) { - connectionIds.add(catalogEntry.connectionId); - } - } - } - return connectionIds; -} - -export async function createManagedMcpRunConfig(input: { - db: Db; - agent: Pick< - typeof agents.$inferSelect, - "id" | "companyId" | "name" | "adapterType" - >; - runId: string; - config: Record; - projectId: string | null; - issueId: string | null; -}): Promise { - if (!adapterSupportsManagedMcpConfig(input.agent.adapterType)) return null; - if (input.config.managedMcpOnly === false) return null; - - const rows = await input.db - .select() - .from(toolMcpGateways) - .where( - and( - eq(toolMcpGateways.companyId, input.agent.companyId), - eq(toolMcpGateways.status, "active"), - isNull(toolMcpGateways.archivedAt), - // A native profile remains an immutable assignment even if gateway - // metadata is cleared. Explicit shared gateways use ordinary profiles. - sql`not exists ( - select 1 from ${toolProfiles} - where ${toolProfiles.id} = ${toolMcpGateways.profileId} - and ${toolProfiles.companyId} = ${toolMcpGateways.companyId} - and (${toolProfiles.profileKey} like 'native:%' - or ${toolProfiles.metadata}->>'source' = 'paperclip_runner') - )`, - ), - ) - .orderBy(asc(toolMcpGateways.name)); - - const installRows = await input.db - .select({ - connectionId: toolConnectionInstalls.connectionId, - enabled: toolConnections.enabled, - status: toolConnections.status, - healthStatus: toolConnections.healthStatus, - config: toolConnections.config, - transportConfig: toolConnections.transportConfig, - }) - .from(toolConnectionInstalls) - .innerJoin( - toolConnections, - and( - eq(toolConnections.id, toolConnectionInstalls.connectionId), - eq(toolConnections.companyId, toolConnectionInstalls.companyId), - ), - ) - .where( - and( - eq(toolConnectionInstalls.companyId, input.agent.companyId), - sql`((${toolConnectionInstalls.targetType} = 'company' and ${toolConnectionInstalls.targetId} = ${input.agent.companyId}) or (${toolConnectionInstalls.targetType} = 'agent' and ${toolConnectionInstalls.targetId} = ${input.agent.id}))`, - ), - ); - const [runIdentity] = await input.db - .select({ - responsibleUserId: heartbeatRuns.responsibleUserId, - activeIdentityContextId: heartbeatRuns.activeIdentityContextId, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.runId), - eq(heartbeatRuns.companyId, input.agent.companyId), - eq(heartbeatRuns.agentId, input.agent.id), - ), - ) - .limit(1); - const resolvedAvailableInstalls = runIdentity?.activeIdentityContextId - ? installRows - .filter((install) => install.enabled && install.status === "active") - .map((install) => ({ id: install.connectionId })) - : await filterResolvedGitHubConnectionsForRun({ - db: input.db, - companyId: input.agent.companyId, - agentId: input.agent.id, - responsibleUserId: runIdentity?.responsibleUserId ?? null, - connections: installRows - .filter( - (install) => - install.enabled && - install.status === "active" && - !["degraded", "failed", "error", "missing_secret"].includes( - install.healthStatus, - ), - ) - .map((install) => ({ - id: install.connectionId, - config: install.config, - transportConfig: install.transportConfig, - })), - }); - const availableInstalledConnectionIds = new Set( - resolvedAvailableInstalls.map((install) => install.id), - ); - - const applicableGateways = rows.filter((gateway) => - // The immutable native assignment is delivered separately. Including any - // historical native gateway here duplicates and broadens that assignment. - !Object.hasOwn(gateway.metadata ?? {}, "nativeRuntimeAssignmentDigest") && gatewayAppliesToRun({ - gateway, - agentId: input.agent.id, - projectId: input.projectId, - issueId: input.issueId, - }), - ); - const gateways = ( - await Promise.all( - applicableGateways.map(async (gateway) => ({ - gateway, - connectionIds: await gatewayConnectionIds({ - db: input.db, - companyId: input.agent.companyId, - gateway, - }), - })), - ) - ) - .filter( - ({ connectionIds }) => - connectionIds.size > 0 && - [...connectionIds].every((connectionId) => - availableInstalledConnectionIds.has(connectionId), - ), - ) - .map(({ gateway }) => gateway); - if (gateways.length === 0) return null; - - const service = createToolGatewayService(input.db); - const expiresAt = new Date(Date.now() + 60 * 60 * 1000); - const managedGateways: ManagedMcpGatewayRunConfig["gateways"] = []; - for (const gateway of gateways) { - const token = await service.createNamedGatewayToken({ - companyId: input.agent.companyId, - gatewayId: gateway.id, - body: { - name: `Managed ${input.agent.name} ${input.runId.slice(0, 8)}`, - subjectType: "heartbeat_run", - subjectId: input.runId, - clientLabel: `${input.agent.name} managed local adapter`, - ownerNote: `Short-lived Paperclip-managed MCP token for heartbeat run ${input.runId}.`, - allowedActions: ["tools/list", "tools/call"], - expiresAt, - }, - actor: { agentId: input.agent.id }, - }); - managedGateways.push({ - id: gateway.id, - name: gateway.name, - // This path must bypass the normal /api agent-JWT middleware. The MCP - // gateway performs its own bearer validation for the run-scoped token. - endpointPath: `/mcp/gateways/${gateway.gatewayPublicId}`, - bearerToken: token.token, - tokenPrefix: token.tokenPrefix, - }); - } - - return { - version: 1, - managedMcpOnly: true, - gateways: managedGateways, - }; -} export function summarizeHeartbeatRunContextSnapshot( contextSnapshot: Record | null | undefined, @@ -3646,21 +2293,6 @@ function parseIssueAssigneeAdapterOverrides( */ const HEARTBEAT_TASK_KEY = "__heartbeat__"; -function deriveTaskKey( - contextSnapshot: Record | null | undefined, - payload: Record | null | undefined, -) { - return ( - readNonEmptyString(contextSnapshot?.taskKey) ?? - readNonEmptyString(contextSnapshot?.taskId) ?? - readNonEmptyString(contextSnapshot?.issueId) ?? - readNonEmptyString(payload?.taskKey) ?? - readNonEmptyString(payload?.taskId) ?? - readNonEmptyString(payload?.issueId) ?? - null - ); -} - /** * Extended task key derivation that falls back to a stable synthetic key * for timer/heartbeat wakes. The synthetic key keeps the @@ -3923,1246 +2555,6 @@ function isCheckoutConflictError(error: unknown): boolean { export { extractWakeCommentIds }; -function mergeWakeCommentIds(...values: Array): string[] { - const merged: string[] = []; - const append = (value: unknown) => { - const normalized = readNonEmptyString(value); - if (!normalized || merged.includes(normalized)) return; - merged.push(normalized); - }; - - for (const value of values) { - if (Array.isArray(value)) { - for (const entry of value) append(entry); - continue; - } - if (typeof value === "object" && value !== null) { - const candidate = value as Record; - const batched = extractWakeCommentIds(candidate); - if (batched.length > 0) { - for (const entry of batched) append(entry); - continue; - } - append(candidate.wakeCommentId); - append(candidate.commentId); - continue; - } - append(value); - } - - return merged; -} - -const EXTERNAL_ATTACHMENT_OMISSION_REASONS = new Set([ - "attachment_limit", - "storage_unavailable", - "declared_too_large", - "download_unavailable", - "unsupported_type", - "empty_download", - "downloaded_too_large", - "processing_failed", -]); - -type ExternalAttachmentOmission = { - commentId: string; - reasons: Record; -}; - -function readExternalAttachmentOmissions( - value: unknown, -): ExternalAttachmentOmission[] { - if (!Array.isArray(value)) return []; - const byCommentId = new Map(); - for (const candidate of value) { - const record = parseObject(candidate); - const commentId = readNonEmptyString(record.commentId); - if (!commentId) continue; - const reasons = Object.fromEntries( - Object.entries(parseObject(record.reasons)).flatMap(([reason, count]) => - EXTERNAL_ATTACHMENT_OMISSION_REASONS.has(reason) && - typeof count === "number" && - Number.isSafeInteger(count) && - count > 0 - ? [[reason, count]] - : [], - ), - ); - if (Object.keys(reasons).length === 0) continue; - byCommentId.set(commentId, { commentId, reasons }); - } - return [...byCommentId.values()].slice(-50); -} - -function mergeExternalAttachmentOmissions( - existing: Record, - incoming: Record, -) { - return readExternalAttachmentOmissions([ - ...readExternalAttachmentOmissions( - existing[EXTERNAL_ATTACHMENT_OMISSIONS_KEY], - ), - ...readExternalAttachmentOmissions( - incoming[EXTERNAL_ATTACHMENT_OMISSIONS_KEY], - ), - ]); -} - -function externalAttachmentOmissionNotice( - omission: ExternalAttachmentOmission, -) { - const entries = Object.entries(omission.reasons); - const omitted = entries.reduce((total, [, count]) => total + count, 0); - const reasons = entries - .map(([reason, count]) => `${reason.replaceAll("_", " ")}: ${count}`) - .join(", "); - return `Paperclip could not import every attachment from this exact external message: ${omitted} attachment${omitted === 1 ? " was" : "s were"} omitted (${reasons}). Treat omitted attachments as unavailable; do not infer their contents or substitute an older workspace file.`; -} - -function enrichWakeContextSnapshot(input: { - contextSnapshot: Record; - reason: string | null; - source: WakeupOptions["source"]; - triggerDetail: WakeupOptions["triggerDetail"] | null; - payload: Record | null; -}) { - const { contextSnapshot, reason, source, triggerDetail, payload } = input; - const issueIdFromPayload = - readNonEmptyString(payload?.["issueId"]) ?? - readNonEmptyString(payload?.["taskId"]); - const commentIdFromPayload = readNonEmptyString(payload?.["commentId"]); - const taskKey = deriveTaskKey(contextSnapshot, payload); - const wakeCommentId = deriveCommentId(contextSnapshot, payload); - const wakeCommentIds = mergeWakeCommentIds( - contextSnapshot, - commentIdFromPayload, - ); - - if (!readNonEmptyString(contextSnapshot["wakeReason"]) && reason) { - contextSnapshot.wakeReason = reason; - } - if (!readNonEmptyString(contextSnapshot["issueId"]) && issueIdFromPayload) { - contextSnapshot.issueId = issueIdFromPayload; - } - if (!readNonEmptyString(contextSnapshot["taskId"]) && issueIdFromPayload) { - contextSnapshot.taskId = issueIdFromPayload; - } - if (!readNonEmptyString(contextSnapshot["taskKey"]) && taskKey) { - contextSnapshot.taskKey = taskKey; - } - if ( - !readNonEmptyString(contextSnapshot["commentId"]) && - commentIdFromPayload - ) { - contextSnapshot.commentId = commentIdFromPayload; - } - if (wakeCommentIds.length > 0) { - const latestCommentId = wakeCommentIds[wakeCommentIds.length - 1]; - contextSnapshot[WAKE_COMMENT_IDS_KEY] = wakeCommentIds; - contextSnapshot.commentId = latestCommentId; - contextSnapshot.wakeCommentId = latestCommentId; - // Once comment ids are normalized into the snapshot, rebuild the structured - // wake payload from those ids later instead of carrying forward stale data. - delete contextSnapshot[PAPERCLIP_WAKE_PAYLOAD_KEY]; - } else if ( - !readNonEmptyString(contextSnapshot["wakeCommentId"]) && - wakeCommentId - ) { - contextSnapshot.wakeCommentId = wakeCommentId; - } - if (!readNonEmptyString(contextSnapshot["wakeSource"]) && source) { - contextSnapshot.wakeSource = source; - } - if ( - !readNonEmptyString(contextSnapshot["wakeTriggerDetail"]) && - triggerDetail - ) { - contextSnapshot.wakeTriggerDetail = triggerDetail; - } - normalizeInteractionContinuationWakeContext(contextSnapshot, payload); - - return { - contextSnapshot, - issueIdFromPayload, - commentIdFromPayload, - taskKey, - wakeCommentId, - }; -} - -const INTERACTION_CONTINUATION_CONTEXT_KEYS = [ - "interactionId", - "interactionKind", - "interactionStatus", - "continuationPolicy", - "checkboxSelection", - "itemVerdicts", - "newlyResolvedItemIds", -] as const; - -function isInteractionResolutionWakePayload( - payload: Record | null | undefined, -) { - return readNonEmptyString(payload?.mutation) === "interaction"; -} - -function clearInteractionContinuationWakeContext( - contextSnapshot: Record, -) { - for (const key of INTERACTION_CONTINUATION_CONTEXT_KEYS) { - delete contextSnapshot[key]; - } -} - -function hasInteractionContinuationWakeContext( - contextSnapshot: Record, -) { - return INTERACTION_CONTINUATION_CONTEXT_KEYS.some((key) => - readNonEmptyString(contextSnapshot[key]), - ); -} - -function normalizeInteractionContinuationWakeContext( - contextSnapshot: Record, - payload: Record | null | undefined, -) { - if (isInteractionResolutionWakePayload(payload)) return; - clearInteractionContinuationWakeContext(contextSnapshot); -} - -type AcceptedPlanWakeRoutingDecision = { - otherActiveClaimIssueId: string; - otherActiveClaimIdentifier: string | null; - otherActiveClaimTitle: string; - forceFreshSession: boolean; - suppressAcceptedContinuation: boolean; -}; - -async function resolveAcceptedPlanWakeRoutingDecision(args: { - db: Db; - companyId: string; - agentId: string; - issueId: string | null; - acceptedPlanContinuationWake: boolean; - contextSnapshot: Record; -}): Promise { - if (args.issueId === null) return null; - if (!args.acceptedPlanContinuationWake) return null; - - const activeClaims = await args.db - .select({ - sourceIssueId: issuePlanDecompositions.sourceIssueId, - identifier: issues.identifier, - title: issues.title, - }) - .from(issuePlanDecompositions) - .innerJoin(issues, eq(issues.id, issuePlanDecompositions.sourceIssueId)) - .where( - and( - eq(issuePlanDecompositions.companyId, args.companyId), - eq(issuePlanDecompositions.ownerAgentId, args.agentId), - eq(issuePlanDecompositions.status, "in_flight"), - ), - ) - .orderBy( - desc(issuePlanDecompositions.updatedAt), - asc(issuePlanDecompositions.createdAt), - ); - - if (activeClaims.length === 0) return null; - if (activeClaims.some((claim) => claim.sourceIssueId === args.issueId)) - return null; - - const otherActiveClaim = activeClaims[0]; - if (!otherActiveClaim) return null; - - const hasAcceptedContinuationWake = - readNonEmptyString(args.contextSnapshot.interactionKind) === - "request_confirmation" && - readNonEmptyString(args.contextSnapshot.interactionStatus) === "accepted"; - - return { - otherActiveClaimIssueId: otherActiveClaim.sourceIssueId, - otherActiveClaimIdentifier: otherActiveClaim.identifier ?? null, - otherActiveClaimTitle: otherActiveClaim.title, - forceFreshSession: true, - suppressAcceptedContinuation: hasAcceptedContinuationWake, - }; -} - -export function mergeCoalescedContextSnapshot( - existingRaw: unknown, - incoming: Record, - options: { preserveExistingInteractionContinuation?: boolean } = {}, -) { - const existing = parseObject(existingRaw); - const existingSource = readNonEmptyString(existing.source); - const incomingSource = readNonEmptyString(incoming.source); - const preservesExternalChatOrigin = - existingSource?.startsWith("chat:") === true && - readNonEmptyString(existing.issueId) !== null && - existing.issueId === incoming.issueId && - incomingSource === "native_status_decision" && - readNonEmptyString(incoming.statusDecisionSource) === - "native_status_decision"; - const merged: Record = { - ...existing, - ...incoming, - }; - // Only executeRun can mint this proof. Coalescence may retain an unchanged - // admitted proof, but must never accept a new marker from an incoming wake. - delete merged[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY]; - delete merged[EXTERNAL_CHAT_QUESTION_RESPONSE_KEY]; - const mergedAttachmentOmissions = mergeExternalAttachmentOmissions( - existing, - incoming, - ); - if (mergedAttachmentOmissions.length > 0) { - merged[EXTERNAL_ATTACHMENT_OMISSIONS_KEY] = mergedAttachmentOmissions; - } else { - delete merged[EXTERNAL_ATTACHMENT_OMISSIONS_KEY]; - } - // A native status wake is control-flow metadata, not a new user-input - // provenance. When it coalesces into the live run, retain the verified chat - // source so the eventual terminal presentation can still prove its route. - // Fresh status-decision runs keep their native_status_decision source. - if (preservesExternalChatOrigin) { - merged.source = existingSource; - } - if ( - existing.forceFreshSession === true || - incoming.forceFreshSession === true - ) { - merged.forceFreshSession = true; - } - if (existing.refreshTools === true || incoming.refreshTools === true) { - merged.refreshTools = true; - } - const mergedCommentIds = mergeWakeCommentIds(existing, incoming); - if (mergedCommentIds.length > 0) { - const latestCommentId = mergedCommentIds[mergedCommentIds.length - 1]; - merged[WAKE_COMMENT_IDS_KEY] = mergedCommentIds; - merged.commentId = latestCommentId; - merged.wakeCommentId = latestCommentId; - // The merged context should carry canonical comment ids; the next wake will - // regenerate any structured payload from those ids. - delete merged[PAPERCLIP_WAKE_PAYLOAD_KEY]; - } - const existingWake = parseObject(existing[PAPERCLIP_WAKE_PAYLOAD_KEY]); - const existingCommentIds = extractWakeCommentIds(existing); - const payloadCommentIds = Array.isArray(existingWake.commentIds) - ? existingWake.commentIds - : []; - const preservesAdmittedWake = - preservesExternalChatOrigin && - parseObject(existingWake.issue).id === existing.issueId && - CHAT_PROVIDERS.some( - (provider) => - existingWake.externalChatProvider === provider && - (existingSource === `chat:${provider}` || - existingSource === `chat:${provider}:recovery`), - ) && - existingCommentIds.length > 0 && - mergedCommentIds.length === existingCommentIds.length && - mergedCommentIds.every((id, index) => id === existingCommentIds[index]) && - payloadCommentIds.length === existingCommentIds.length && - payloadCommentIds.every((id, index) => id === existingCommentIds[index]) && - ((existing[PAPERCLIP_HARNESS_CHECKOUT_KEY] === true && - existingWake.checkedOutByHarness === true) || - (existing[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] === true && - existingWake.externalChatExecutionBound === true)); - if (preservesAdmittedWake) { - merged[PAPERCLIP_WAKE_PAYLOAD_KEY] = existingWake; - merged.wakeReason = existing.wakeReason; - if (existing[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] === true) { - merged[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] = true; - } - } - if ( - !hasInteractionContinuationWakeContext(incoming) && - !( - options.preserveExistingInteractionContinuation === true && - hasInteractionContinuationWakeContext(existing) - ) - ) { - clearInteractionContinuationWakeContext(merged); - } - return merged; -} - -// This is a prompt optimization, not an authorization grant. Verify the durable -// chat binding rather than trusting an arbitrary caller's source/context marker. -export async function resolveExternalChatWakeProvider(input: { - db: Db; - companyId: string; - agentId?: string | null; - runId?: string | null; - issueId: string | null; - contextSnapshot: Record; -}): Promise { - if (input.contextSnapshot.source === "issue.interaction.respond") { - if (!input.runId || !input.agentId || !input.issueId) return null; - const answer = await resolveExternalChatQuestionResponse( - input.db, - { - companyId: input.companyId, - agentId: input.agentId, - issueId: input.issueId, - runId: input.runId, - }, - input.contextSnapshot, - "read", - ); - return answer?.provider ?? null; - } - const source = readNonEmptyString(input.contextSnapshot.source); - const provider = CHAT_PROVIDERS.find( - (candidate) => - source === `chat:${candidate}` || source === `chat:${candidate}:recovery`, - ); - const commentIds = extractWakeCommentIds(input.contextSnapshot); - if ( - !provider || - !input.agentId || - !input.issueId || - commentIds.length === 0 || - (input.contextSnapshot[PAPERCLIP_HARNESS_CHECKOUT_KEY] !== true && - input.contextSnapshot[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] !== - true) - ) { - return null; - } - - const links = await input.db - .select({ - commentId: chatMessageLinks.commentId, - conversationId: chatConversations.id, - }) - .from(chatMessageLinks) - .innerJoin( - chatConversations, - and( - eq(chatConversations.companyId, chatMessageLinks.companyId), - eq(chatConversations.id, chatMessageLinks.conversationId), - eq(chatConversations.endpointId, chatMessageLinks.endpointId), - ), - ) - .innerJoin( - chatEndpoints, - and( - eq(chatEndpoints.companyId, chatConversations.companyId), - eq(chatEndpoints.id, chatConversations.endpointId), - ), - ) - .where( - and( - eq(chatMessageLinks.companyId, input.companyId), - eq(chatMessageLinks.direction, "inbound"), - inArray(chatMessageLinks.commentId, commentIds), - eq(chatConversations.issueId, input.issueId), - inArray(chatConversations.state, ["active", "waiting"]), - eq(chatEndpoints.provider, provider), - eq(chatEndpoints.assignedAgentId, input.agentId), - inArray(chatEndpoints.status, ["active", "verifying"]), - ), - ); - const linkedCommentIds = new Set(links.map((link) => link.commentId)); - return new Set(links.map((link) => link.conversationId)).size === 1 && - commentIds.every((id) => linkedCommentIds.has(id)) - ? provider - : null; -} - -/** Bind an already-claimed chat/reply turn without checking out or approving it. */ -export async function attestReviewedExternalChatRun(input: { - db: Db; - companyId: string; - agentId: string; - issueId: string; - runId: string; - contextSnapshot: Record; - /** Diagnostic-only value from the committed, authorized answer; never a wake field. */ - onQuestionResponseAttested?: (answeredAtMs: number) => void; -}): Promise { - const attempt = () => - input.db.transaction( - async ( - transaction, - ): Promise => { - const tx = transaction as unknown as Db; - const [issue] = await tx - .select() - .from(issues) - .where( - and( - eq(issues.id, input.issueId), - eq(issues.companyId, input.companyId), - ), - ) - .for("update", { noWait: true }) - .limit(1); - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.runId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("update", { noWait: true }) - .limit(1); - const [actor] = await tx - .select({ status: agents.status }) - .from(agents) - .where( - and( - eq(agents.id, input.agentId), - eq(agents.companyId, input.companyId), - ), - ) - .for("update", { noWait: true }) - .limit(1); - if ( - !issue || - !run || - !actor || - !( - issue.status === "in_review" || - (input.contextSnapshot.source === "issue.interaction.respond" && - issue.status === "in_progress") - ) || - issue.assigneeAgentId !== input.agentId || - issue.executionRunId !== input.runId || - run.status !== "running" || - (run.nativeIssueId !== null && run.nativeIssueId !== input.issueId) || - // This attestation precedes execution-start's transition to running. - // A prior run's error projection is invokable, unlike an operator - // pause, termination, or pending approval; all binding checks remain. - DIRECT_NON_INVOKABLE_STATUSES.has(actor.status) - ) - return false; - const admittedContext = parseObject(run.contextSnapshot); - const admittedIds = extractWakeCommentIds(admittedContext); - const suppliedIds = extractWakeCommentIds(input.contextSnapshot); - if ( - admittedContext.issueId !== input.issueId || - input.contextSnapshot.issueId !== input.issueId || - admittedContext.source !== input.contextSnapshot.source || - admittedIds.length === 0 || - admittedIds.length !== suppliedIds.length || - admittedIds.some((id, index) => id !== suppliedIds[index]) - ) - return false; - try { - const answer = - admittedContext.source === "issue.interaction.respond" - ? await resolveExternalChatQuestionResponse( - tx, - input, - admittedContext, - "nonblocking", - true, - ) - : null; - if (admittedContext.source === "issue.interaction.respond" && !answer) - return false; - // The marker is built here only after proving the real execution owner. - // The shared boundary then verifies current provider/resource/principal - // access for every admitted message; no lifecycle state is mutated. - await authorizeChatConversationForBoundRun( - tx, - input, - { - ...(answer?.authorizationContext ?? admittedContext), - [PAPERCLIP_HARNESS_CHECKOUT_KEY]: false, - [PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY]: true, - }, - "nonblocking", - ); - if (answer) - input.contextSnapshot[EXTERNAL_CHAT_QUESTION_RESPONSE_KEY] = - answer.marker; - return answer ? { answeredAtMs: answer.answeredAtMs } : true; - } catch (error) { - if ( - error instanceof Error && - error.message === "paperclip_runner_chat_attachment_binding_denied" - ) { - // Inbound processing commits the message/link before dispatching its - // wake, but completes subscription and marks delivery processed after - // dispatch. Wait for that exact committed batch, never admit it early. - const links = await tx - .select({ - commentId: chatMessageLinks.commentId, - state: chatDeliveries.state, - }) - .from(chatMessageLinks) - .innerJoin( - chatDeliveries, - and( - eq(chatDeliveries.id, chatMessageLinks.deliveryId), - eq(chatDeliveries.companyId, chatMessageLinks.companyId), - eq(chatDeliveries.endpointId, chatMessageLinks.endpointId), - eq( - chatDeliveries.conversationId, - chatMessageLinks.conversationId, - ), - ), - ) - .where( - and( - eq(chatMessageLinks.companyId, input.companyId), - eq(chatMessageLinks.direction, "inbound"), - inArray(chatMessageLinks.commentId, admittedIds), - ), - ); - if ( - admittedIds.every((id) => - links.some((link) => link.commentId === id), - ) && - links.some((link) => link.state === "processing") && - links.every( - (link) => - link.state === "processing" || link.state === "processed", - ) - ) { - return "pending_delivery"; - } - } - if ( - error instanceof Error && - [ - "paperclip_runner_chat_attachment_binding_denied", - "paperclip_runner_chat_attachment_destination_denied", - "paperclip_runner_chat_attachment_principal_denied", - ].includes(error.message) - ) - return false; - throw error; - } - }, - ); - // Release every lock before retrying and re-prove the current execution - // owner, admitted batch, and policy. Routine control-plane contention is not - // evidence that the user lost access; neither is it permission to bypass it. - for (let attemptNumber = 0; attemptNumber < 51; attemptNumber += 1) { - try { - const result = await attempt(); - if (typeof result === "object") { - // The transaction (including COMMIT) must succeed before any attempt- - // local timing is accepted. A retry/rollback cannot publish this value. - input.onQuestionResponseAttested?.(result.answeredAtMs); - return true; - } - if (result !== "pending_delivery") return result; - } catch (error) { - if (!isExternalChatWaitAuthorizationContention(error)) throw error; - } - if (attemptNumber < 50) - await new Promise((resolve) => setTimeout(resolve, 100)); - } - throw new Error("reviewed_chat_execution_binding_not_ready"); -} - -export async function buildPaperclipWakePayload(input: { - db: Db; - companyId: string; - agentId?: string | null; - runId?: string | null; - contextSnapshot: Record; - continuationSummary?: { - key: string; - title: string | null; - body: string; - sourceTrust?: SourceTrustMetadata | null; - updatedAt: Date; - } | null; - issueSummary?: { - id: string; - identifier: string | null; - title: string; - description: string | null; - status: string; - priority: string; - workMode: string; - projectId?: string | null; - executionPolicy?: unknown; - } | null; - exposeLowTrustRaw?: boolean; - // Experimental: agents write user-interaction content in ASD-STE100 - // Simplified Technical English (rendered as a prompt directive downstream). - simplifiedEnglishInteractions?: boolean; -}) { - const executionStage = parseObject(input.contextSnapshot.executionStage); - const commentIds = extractWakeCommentIds(input.contextSnapshot); - const annotationCommentId = readNonEmptyString( - input.contextSnapshot.annotationCommentId, - ); - const issueId = readNonEmptyString(input.contextSnapshot.issueId); - const conversationMode = input.contextSnapshot.conversationMode === true; - const continuationSummary = conversationMode ? null : input.continuationSummary ?? null; - const agentMessage = parseObject( - input.contextSnapshot[PAPERCLIP_AGENT_MESSAGE_KEY], - ); - const agentMessageText = sanitizeAgentSessionMessageText(agentMessage.text); - const issueSummary = - input.issueSummary ?? - (issueId - ? await input.db - .select({ - id: issues.id, - identifier: issues.identifier, - title: issues.title, - description: issues.description, - status: issues.status, - priority: issues.priority, - workMode: issues.workMode, - }) - .from(issues) - .where( - and(eq(issues.id, issueId), eq(issues.companyId, input.companyId)), - ) - .then((rows) => rows[0] ?? null) - : null); - if ( - commentIds.length === 0 && - Object.keys(executionStage).length === 0 && - !issueSummary && - !agentMessageText - ) - return null; - - const commentRows = - commentIds.length === 0 - ? [] - : await input.db - .select({ - id: issueComments.id, - issueId: issueComments.issueId, - body: issueComments.body, - authorType: issueComments.authorType, - authorAgentId: issueComments.authorAgentId, - authorUserId: issueComments.authorUserId, - presentation: issueComments.presentation, - metadata: issueComments.metadata, - deletedAt: issueComments.deletedAt, - deletedByType: issueComments.deletedByType, - deletedByAgentId: issueComments.deletedByAgentId, - deletedByUserId: issueComments.deletedByUserId, - deletedByRunId: issueComments.deletedByRunId, - sourceTrust: issueComments.sourceTrust, - createdAt: issueComments.createdAt, - }) - .from(issueComments) - .where( - and( - eq(issueComments.companyId, input.companyId), - issueId ? eq(issueComments.issueId, issueId) : undefined, - inArray(issueComments.id, commentIds), - ), - ); - - const commentsById = new Map( - commentRows.map((comment) => [comment.id, comment]), - ); - const issueDescription = conversationMode ? null : issueSummary?.description ?? null; - const issueDescriptionTruncated = - issueDescription !== null && - issueDescription.length > MAX_INLINE_WAKE_ISSUE_DESCRIPTION_CHARS; - const inlineIssueDescription = issueDescriptionTruncated - ? issueDescription.slice(0, MAX_INLINE_WAKE_ISSUE_DESCRIPTION_CHARS) - : issueDescription; - const comments: Array> = []; - let remainingBodyChars = MAX_INLINE_WAKE_COMMENT_BODY_TOTAL_CHARS; - let truncated = false; - let missingCommentCount = 0; - const safeContinuationSummary = - continuationSummary && !input.exposeLowTrustRaw - ? redactQuarantinedBodyForHigherTrust(continuationSummary) - : continuationSummary; - - for (const commentId of commentIds) { - const row = commentsById.get(commentId); - if (!row) { - truncated = true; - missingCommentCount += 1; - continue; - } - if (comments.length >= MAX_INLINE_WAKE_COMMENTS) { - truncated = true; - break; - } - - const deletedAt = row.deletedAt ?? null; - const safeRow = - deletedAt || input.exposeLowTrustRaw - ? row - : sanitizeQuarantinedCommentForHigherTrust(row); - const fullBody = deletedAt ? "" : safeRow.body; - const allowedBodyChars = Math.min( - MAX_INLINE_WAKE_COMMENT_BODY_CHARS, - remainingBodyChars, - ); - if (allowedBodyChars <= 0) { - truncated = true; - break; - } - - const body = - fullBody.length > allowedBodyChars - ? fullBody.slice(0, allowedBodyChars) - : fullBody; - const bodyTruncated = body.length < fullBody.length; - if (bodyTruncated) truncated = true; - remainingBodyChars -= body.length; - - comments.push({ - id: row.id, - issueId: row.issueId, - authorType: - row.authorType ?? - (row.authorAgentId ? "agent" : row.authorUserId ? "user" : "system"), - body, - bodyTruncated, - presentation: deletedAt ? null : (safeRow.presentation ?? null), - metadata: deletedAt ? null : (safeRow.metadata ?? null), - deletedAt: deletedAt ? deletedAt.toISOString() : null, - deletedByType: deletedAt ? (row.deletedByType ?? null) : null, - deletedByAgentId: deletedAt ? (row.deletedByAgentId ?? null) : null, - deletedByUserId: deletedAt ? (row.deletedByUserId ?? null) : null, - deletedByRunId: deletedAt ? (row.deletedByRunId ?? null) : null, - sourceTrust: row.sourceTrust ?? null, - createdAt: row.createdAt.toISOString(), - author: row.authorAgentId - ? { type: "agent", id: row.authorAgentId } - : row.authorUserId - ? { type: "user", id: row.authorUserId } - : { type: "system", id: null }, - }); - } - - const attachmentCommentIds = comments.flatMap((comment) => - typeof comment.id === "string" && - comment.deletedAt === null && - (input.exposeLowTrustRaw || - !isLowTrustQuarantined(comment.sourceTrust as SourceTrustMetadata | null)) - ? [comment.id] - : [], - ); - const attachmentRows = - !issueId || attachmentCommentIds.length === 0 - ? [] - : await input.db - .select({ - id: issueAttachments.id, - issueCommentId: issueAttachments.issueCommentId, - filename: assets.originalFilename, - contentType: assets.contentType, - byteSize: assets.byteSize, - }) - .from(issueAttachments) - .innerJoin( - assets, - and( - eq(issueAttachments.assetId, assets.id), - eq(assets.companyId, input.companyId), - ), - ) - .where( - and( - eq(issueAttachments.companyId, input.companyId), - eq(issueAttachments.issueId, issueId), - inArray(issueAttachments.issueCommentId, attachmentCommentIds), - ), - ) - .orderBy(asc(issueAttachments.createdAt), asc(issueAttachments.id)) - .limit(MAX_INLINE_WAKE_ATTACHMENTS + 1); - if (attachmentRows.length > MAX_INLINE_WAKE_ATTACHMENTS) truncated = true; - const attachmentsByCommentId = new Map< - string, - Array<{ - id: string; - filename: string; - contentType: string; - byteSize: number; - contentPath: string; - }> - >(); - for (const attachment of attachmentRows.slice( - 0, - MAX_INLINE_WAKE_ATTACHMENTS, - )) { - if (!attachment.issueCommentId) continue; - const descriptors = - attachmentsByCommentId.get(attachment.issueCommentId) ?? []; - descriptors.push({ - id: attachment.id, - filename: attachment.filename?.trim() || "attachment", - contentType: attachment.contentType, - byteSize: attachment.byteSize, - contentPath: `/api/attachments/${attachment.id}/content`, - }); - attachmentsByCommentId.set(attachment.issueCommentId, descriptors); - } - for (const comment of comments) { - if (typeof comment.id !== "string") continue; - const attachments = attachmentsByCommentId.get(comment.id); - if (attachments?.length) comment.attachments = attachments; - } - - const annotationDeltas = - annotationCommentId && issueId - ? await input.db - .select({ - id: documentAnnotationComments.id, - issueId: documentAnnotationComments.issueId, - threadId: documentAnnotationComments.threadId, - body: documentAnnotationComments.body, - authorType: documentAnnotationComments.authorType, - authorAgentId: documentAnnotationComments.authorAgentId, - authorUserId: documentAnnotationComments.authorUserId, - createdAt: documentAnnotationComments.createdAt, - documentKey: documentAnnotationThreads.documentKey, - status: documentAnnotationThreads.status, - anchorState: documentAnnotationThreads.anchorState, - anchorConfidence: documentAnnotationThreads.anchorConfidence, - currentRevisionNumber: - documentAnnotationThreads.currentRevisionNumber, - selectedText: documentAnnotationThreads.selectedText, - prefixText: documentAnnotationThreads.prefixText, - suffixText: documentAnnotationThreads.suffixText, - }) - .from(documentAnnotationComments) - .innerJoin( - documentAnnotationThreads, - eq( - documentAnnotationComments.threadId, - documentAnnotationThreads.id, - ), - ) - .where( - and( - eq(documentAnnotationComments.companyId, input.companyId), - eq(documentAnnotationComments.issueId, issueId), - eq(documentAnnotationComments.id, annotationCommentId), - eq(documentAnnotationThreads.companyId, input.companyId), - eq(documentAnnotationThreads.issueId, issueId), - ), - ) - .then((rows) => - rows.map((row) => ({ - id: row.id, - issueId: row.issueId, - threadId: row.threadId, - documentKey: row.documentKey, - revisionNumber: row.currentRevisionNumber, - quote: row.selectedText, - prefix: row.prefixText, - suffix: row.suffixText, - threadStatus: row.status, - anchorState: row.anchorState, - anchorConfidence: row.anchorConfidence, - body: - row.body.length > MAX_INLINE_WAKE_COMMENT_BODY_CHARS - ? row.body.slice(0, MAX_INLINE_WAKE_COMMENT_BODY_CHARS) - : row.body, - bodyTruncated: - row.body.length > MAX_INLINE_WAKE_COMMENT_BODY_CHARS, - createdAt: row.createdAt.toISOString(), - author: row.authorAgentId - ? { type: "agent", id: row.authorAgentId } - : row.authorUserId - ? { type: "user", id: row.authorUserId } - : { type: row.authorType, id: null }, - })), - ) - : []; - const interactionId = readNonEmptyString(input.contextSnapshot.interactionId); - const interactionKind = readNonEmptyString( - input.contextSnapshot.interactionKind, - ); - const interactionStatus = readNonEmptyString( - input.contextSnapshot.interactionStatus, - ); - const interactionContinuationSource = readNonEmptyString( - input.contextSnapshot.source, - ); - const externalInteractionContinuation = - (interactionStatus === "answered" || interactionStatus === "accepted") && - (input.contextSnapshot.externalChatContinuation === true || - interactionContinuationSource === "external_chat.interaction.resolve"); - const checkboxSelection = parseObject( - input.contextSnapshot.checkboxSelection, - ); - // A resolved plan review is new user input, including in chat. Ordinary chat - // wakes must still exclude historical plan context across /new boundaries. - const resolvedPlanInteraction = interactionId && interactionKind === "request_confirmation" && - (interactionStatus === "accepted" || interactionStatus === "rejected"); - const planReviewContext = issueId && (!conversationMode || resolvedPlanInteraction) - ? await buildPlanReviewContext({ - db: input.db, - companyId: input.companyId, - issueId, - issueWorkMode: conversationMode ? null : issueSummary?.workMode ?? null, - includeForIssueComment: !conversationMode && commentIds.length > 0, - includeForAnnotationDelta: !conversationMode && annotationDeltas.length > 0, - interactionId, - }) - : null; - const documentReviewContext = issueId && !conversationMode - ? await buildDocumentReviewContext({ - db: input.db, - companyId: input.companyId, - issueId, - includeForIssueComment: commentIds.length > 0, - includeForAnnotationDelta: annotationDeltas.length > 0, - }) - : null; - const payloadTruncated = - truncated || - issueDescriptionTruncated || - planReviewContext?.truncated === true || - documentReviewContext?.truncated === true; - const recoveryActionId = readNonEmptyString( - input.contextSnapshot.recoveryActionId, - ); - const recoveryCause = readNonEmptyString(input.contextSnapshot.recoveryCause); - const recoveryAction = recoveryActionId - ? await input.db - .select() - .from(issueRecoveryActions) - .where( - and( - eq(issueRecoveryActions.id, recoveryActionId), - eq(issueRecoveryActions.companyId, input.companyId), - ), - ) - .then((rows) => rows[0] ?? null) - : null; - const recoveryEvidence = parseObject(recoveryAction?.evidence); - // A restored task resumes its deliverable, not the completed repair. A - // referenced settled/missing action must not fall back to a stale cause. - const recoveryStillActive = recoveryAction - ? ["active", "escalated"].includes(recoveryAction.status) - : !recoveryActionId && Boolean(recoveryCause); - const originalAssigneeId = - recoveryAction?.returnOwnerAgentId ?? - recoveryAction?.previousOwnerAgentId ?? - null; - const originalAssignee = originalAssigneeId - ? await input.db - .select({ id: agents.id, name: agents.name }) - .from(agents) - .where( - and( - eq(agents.id, originalAssigneeId), - eq(agents.companyId, input.companyId), - ), - ) - .then((rows) => rows[0] ?? null) - : null; - - const externalChatProvider = await resolveExternalChatWakeProvider({ - db: input.db, - companyId: input.companyId, - agentId: input.agentId, - runId: input.runId, - issueId: issueSummary?.id === issueId ? issueId : null, - contextSnapshot: input.contextSnapshot, - }); - const attachmentOmissions = externalChatProvider - ? readExternalAttachmentOmissions( - input.contextSnapshot[EXTERNAL_ATTACHMENT_OMISSIONS_KEY], - ) - .filter((omission) => commentIds.includes(omission.commentId)) - .map((omission) => ({ - commentId: omission.commentId, - reasons: omission.reasons, - notice: externalAttachmentOmissionNotice(omission), - })) - : []; - const payload = { - reason: readNonEmptyString(input.contextSnapshot.wakeReason), - executionContinuation: input.contextSnapshot.executionContinuation ?? null, - chatCompletionUpdates: input.contextSnapshot.chatCompletionUpdates ?? null, - attachmentOmissions, - externalChatProvider, - recovery: - recoveryStillActive - ? { - cause: recoveryAction?.cause ?? recoveryCause, - failureSummary: readNonEmptyString(recoveryEvidence.failureSummary), - originalAssignee: originalAssignee - ? { id: originalAssignee.id, name: originalAssignee.name } - : originalAssigneeId - ? { id: originalAssigneeId, name: null } - : null, - attemptCount: recoveryAction?.attemptCount ?? null, - maxAttempts: recoveryAction?.maxAttempts ?? null, - nextAction: recoveryAction?.nextAction ?? null, - routingFallbackReason: readNonEmptyString( - recoveryEvidence.routingFallbackReason, - ), - } - : null, - issue: issueSummary - ? { - id: issueSummary.id, - identifier: issueSummary.identifier, - title: issueSummary.title, - description: inlineIssueDescription, - descriptionTruncated: issueDescriptionTruncated, - status: issueSummary.status, - priority: issueSummary.priority, - workMode: issueSummary.workMode, - } - : null, - agentMessage: agentMessageText - ? { - text: agentMessageText, - source: readNonEmptyString(agentMessage.source), - pluginKey: readNonEmptyString(agentMessage.pluginKey), - sessionId: readNonEmptyString(agentMessage.sessionId), - ...(Array.isArray(agentMessage.untrustedToolResults) - ? { - untrustedToolResults: agentMessage.untrustedToolResults - .slice(0, 8) - .map((value) => { - const result = parseObject(value); - return { - actionRequestId: - sanitizeAgentSessionMessageText( - result.actionRequestId, - ) ?? "", - toolName: - sanitizeAgentSessionMessageText(result.toolName) ?? "", - resultSummary: - sanitizeAgentSessionMessageText(result.resultSummary) ?? - "", - error: sanitizeAgentSessionMessageText(result.error), - declineReason: sanitizeAgentSessionMessageText( - result.declineReason, - ), - }; - }), - } - : {}), - } - : null, - childIssueSummaries: Array.isArray( - input.contextSnapshot.childIssueSummaries, - ) - ? input.contextSnapshot.childIssueSummaries - : [], - childIssueSummaryTruncated: - input.contextSnapshot.childIssueSummaryTruncated === true, - dispositionRepair: input.contextSnapshot.legacyDispositionEpisode ? { - attempt: input.contextSnapshot.dispositionRepairAttempt, - maxAttempts: input.contextSnapshot.dispositionRepairMaxAttempts, - sourceRunId: input.contextSnapshot.retryOfRunId, - instruction: input.contextSnapshot.dispositionRepairInstruction, - } : null, - livenessContinuation: - readNonEmptyString(input.contextSnapshot.livenessContinuationState) || - readNonEmptyString( - input.contextSnapshot.livenessContinuationInstruction, - ) || - readNonEmptyString( - input.contextSnapshot.livenessContinuationSourceRunId, - ) || - typeof input.contextSnapshot.livenessContinuationAttempt === "number" - ? { - attempt: input.contextSnapshot.livenessContinuationAttempt, - maxAttempts: input.contextSnapshot.livenessContinuationMaxAttempts, - sourceRunId: readNonEmptyString( - input.contextSnapshot.livenessContinuationSourceRunId, - ), - state: readNonEmptyString( - input.contextSnapshot.livenessContinuationState, - ), - reason: readNonEmptyString( - input.contextSnapshot.livenessContinuationReason, - ), - instruction: readNonEmptyString( - input.contextSnapshot.livenessContinuationInstruction, - ), - } - : null, - interactionKind, - interactionStatus, - interactionId, - sourceRunId: readNonEmptyString(input.contextSnapshot.sourceRunId), - externalChatQuestionResponse: externalChatProvider - ? (input.contextSnapshot[EXTERNAL_CHAT_QUESTION_RESPONSE_KEY] ?? null) - : null, - externalInteractionContinuation, - checkboxSelection: - Object.keys(checkboxSelection).length > 0 ? checkboxSelection : null, - checkedOutByHarness: - input.contextSnapshot[PAPERCLIP_HARNESS_CHECKOUT_KEY] === true, - externalChatExecutionBound: - input.contextSnapshot[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] === - true, - simplifiedEnglishInteractions: input.simplifiedEnglishInteractions === true, - dependencyBlockedInteraction: - input.contextSnapshot.dependencyBlockedInteraction === true, - treeHoldInteraction: input.contextSnapshot.treeHoldInteraction === true, - activeTreeHold: parseObject(input.contextSnapshot.activeTreeHold), - unresolvedBlockerIssueIds: Array.isArray( - input.contextSnapshot.unresolvedBlockerIssueIds, - ) - ? input.contextSnapshot.unresolvedBlockerIssueIds.filter( - (value): value is string => - typeof value === "string" && value.length > 0, - ) - : [], - unresolvedBlockerSummaries: Array.isArray( - input.contextSnapshot.unresolvedBlockerSummaries, - ) - ? input.contextSnapshot.unresolvedBlockerSummaries - : [], - executionStage: - Object.keys(executionStage).length > 0 ? executionStage : null, - taskWatchdog: (input.contextSnapshot.taskWatchdog ?? null) as unknown, - skillTest: (input.contextSnapshot.paperclipSkillTest ?? null) as unknown, - continuationSummary: safeContinuationSummary - ? { - key: safeContinuationSummary.key, - title: safeContinuationSummary.title, - body: - safeContinuationSummary.body.length > 4_000 - ? safeContinuationSummary.body.slice(0, 4_000) - : safeContinuationSummary.body, - bodyTruncated: safeContinuationSummary.body.length > 4_000, - sourceTrust: safeContinuationSummary.sourceTrust ?? null, - updatedAt: safeContinuationSummary.updatedAt.toISOString(), - } - : null, - commentIds, - latestCommentId: commentIds[commentIds.length - 1] ?? null, - comments, - annotationDeltas, - planReviewContext, - documentReviewContext, - commentWindow: { - requestedCount: commentIds.length, - includedCount: comments.length, - missingCount: missingCommentCount, - }, - truncated: payloadTruncated, - fallbackFetchNeeded: payloadTruncated || missingCommentCount > 0, - }; - return issueId - ? createRunSecretRedactionRegistry(input.db).redactForIssue( - input.companyId, - issueId, - payload, - ) - : payload; -} - function runTaskKey(run: typeof heartbeatRuns.$inferSelect) { return deriveTaskKey( run.contextSnapshot as Record | null, @@ -6054,6 +3446,17 @@ export function heartbeatService( db: Db, options: HeartbeatServiceOptions = {}, ) { + const { + toAgentOrgRow, + listCompanyAgentOrgRows, + groupAgentOrgRowsByCompany, + getIssueExecutionContext, + getPinnedSkillTestContext, + getRoutineEnvForExecutionIssue, + resolveResponsibleUserIdForRunSeed, + resolveResponsibleUserIdForRun, + resolveResponsibleUserIdForRunContext, + } = createHeartbeatRunPreparation(db); let shutdownInProgress = false; const instanceSettings = instanceSettingsService(db); const getCurrentUserRedactionOptions = async () => ({ @@ -7217,49 +4620,6 @@ export function heartbeatService( return evaluateAgentInvokabilityFromDb(db, agent); } - function toAgentOrgRow( - agent: Pick< - typeof agents.$inferSelect, - "id" | "companyId" | "name" | "reportsTo" | "status" - >, - ): AgentOrgRow { - return { - id: agent.id, - companyId: agent.companyId, - name: agent.name, - reportsTo: agent.reportsTo, - status: agent.status, - }; - } - - async function listCompanyAgentOrgRows( - companyId: string, - ): Promise { - return db - .select({ - id: agents.id, - companyId: agents.companyId, - name: agents.name, - reportsTo: agents.reportsTo, - status: agents.status, - }) - .from(agents) - .where(eq(agents.companyId, companyId)); - } - - function groupAgentOrgRowsByCompany(agentRows: AgentOrgRow[]) { - const byCompany = new Map(); - for (const agent of agentRows) { - const companyAgents = byCompany.get(agent.companyId); - if (companyAgents) { - companyAgents.push(agent); - } else { - byCompany.set(agent.companyId, [agent]); - } - } - return byCompany; - } - async function getRun( runId: string, opts?: { unsafeFullResultJson?: boolean; includeExecutionEvidence?: boolean }, @@ -7308,413 +4668,6 @@ export function heartbeatService( .then((rows) => rows[0] ?? null); } - async function getIssueExecutionContext(companyId: string, issueId: string) { - return db - .select({ - chatCommunicationGuidance: chatConversations.communicationGuidance, - chatAssignedAgentId: chatEndpoints.assignedAgentId, - // Select only the public command, never the rest of setup state. - chatSlackCommand: sql`case when ${chatEndpoints.status} in ('active', 'verifying') then ${chatEndpoints.setup}->>'command' end`, - externalConversationState: externalConversationStateSql(), - conversationAgentId: issues.conversationAgentId, - conversationUserId: issues.conversationUserId, - conversationState: issues.conversationState, - conversationSessionGeneration: issues.conversationSessionGeneration, - conversationBoundaryCommentId: issues.conversationBoundaryCommentId, - id: issues.id, - identifier: issues.identifier, - title: issues.title, - titleNeedsGeneration: issues.titleNeedsGeneration, - description: issues.description, - status: issues.status, - workMode: issues.workMode, - reviewPolicy: issues.reviewPolicy, - priority: issues.priority, - projectId: issues.projectId, - projectWorkspaceId: issues.projectWorkspaceId, - executionWorkspaceId: issues.executionWorkspaceId, - executionWorkspacePreference: issues.executionWorkspacePreference, - assigneeAgentId: issues.assigneeAgentId, - assigneeAdapterOverrides: issues.assigneeAdapterOverrides, - executionPolicy: issues.executionPolicy, - executionState: issues.executionState, - executionWorkspaceSettings: issues.executionWorkspaceSettings, - parentId: issues.parentId, - createdByUserId: issues.createdByUserId, - responsibleUserId: issues.responsibleUserId, - originKind: issues.originKind, - originId: issues.originId, - originRunId: issues.originRunId, - originIdentityContextId: issues.originIdentityContextId, - continuationIdentityContextId: issues.continuationIdentityContextId, - updatedAt: issues.updatedAt, - }) - .from(issues) - .leftJoin(chatConversations, and( - eq(chatConversations.companyId, issues.companyId), - eq(chatConversations.issueId, issues.id), - )) - .leftJoin(chatEndpoints, and( - eq(chatEndpoints.companyId, chatConversations.companyId), - eq(chatEndpoints.id, chatConversations.endpointId), - eq(chatEndpoints.provider, "slack"), - )) - .where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))) - .then((rows) => rows[0] ?? null); - } - - async function getPinnedSkillTestContext(companyId: string, issueId: string) { - const row = await db - .select({ - testRunId: companySkillTestRuns.id, - skillId: companySkillTestRuns.skillId, - inputId: companySkillTestRuns.inputId, - skillVersionId: companySkillTestRuns.skillVersionId, - outputDocumentKey: companySkillTestRuns.outputDocumentKey, - fileInventory: companySkillVersions.fileInventory, - revisionNumber: companySkillVersions.revisionNumber, - label: companySkillVersions.label, - }) - .from(companySkillTestRuns) - .innerJoin( - companySkillVersions, - and( - eq(companySkillVersions.id, companySkillTestRuns.skillVersionId), - eq(companySkillVersions.companyId, companySkillTestRuns.companyId), - ), - ) - .where( - and( - eq(companySkillTestRuns.companyId, companyId), - eq(companySkillTestRuns.issueId, issueId), - ), - ) - .then((rows) => rows[0] ?? null); - if (!row) return null; - const fileInventory = Array.isArray(row.fileInventory) - ? row.fileInventory.flatMap((entry) => { - if (!entry || typeof entry !== "object" || Array.isArray(entry)) - return []; - const record = entry as unknown as Record; - const path = typeof record.path === "string" ? record.path : ""; - if (!path) return []; - return [ - { - path, - kind: typeof record.kind === "string" ? record.kind : "other", - content: typeof record.content === "string" ? record.content : "", - }, - ]; - }) - : []; - return { - testRunId: row.testRunId, - skillId: row.skillId, - inputId: row.inputId ?? null, - skillVersionId: row.skillVersionId, - revisionNumber: row.revisionNumber, - label: row.label ?? null, - outputDocumentKey: row.outputDocumentKey, - fileInventory, - }; - } - - async function getRoutineEnvForExecutionIssue( - companyId: string, - issueContext: { originKind: string | null; originId: string | null; originRunId: string | null } | null, - ) { - if ( - !issueContext || - issueContext.originKind !== "routine_execution" || - !issueContext.originId - ) { - return { routineId: null, env: null, responsibleUserId: null }; - } - - const routineRun = issueContext.originRunId - ? await db - .select({ - routineRevisionId: routineRuns.routineRevisionId, - responsibleUserId: routineRuns.responsibleUserId, - }) - .from(routineRuns) - .where( - and( - eq(routineRuns.id, issueContext.originRunId), - eq(routineRuns.companyId, companyId), - eq(routineRuns.routineId, issueContext.originId), - ), - ) - .then((rows) => rows[0] ?? null) - : null; - - if (routineRun?.routineRevisionId) { - const revision = await db - .select({ - snapshot: routineRevisions.snapshot, - responsibleUserId: routineRevisions.responsibleUserId, - }) - .from(routineRevisions) - .where( - and( - eq(routineRevisions.id, routineRun.routineRevisionId), - eq(routineRevisions.companyId, companyId), - eq(routineRevisions.routineId, issueContext.originId), - ), - ) - .then((rows) => rows[0] ?? null); - const snapshot = revision?.snapshot as - RoutineRevisionSnapshotV1 | undefined; - if (snapshot?.version === 1) { - return { - routineId: issueContext.originId, - env: snapshot.routine.env ?? null, - responsibleUserId: - routineRun?.responsibleUserId ?? - revision?.responsibleUserId ?? - snapshot.routine.responsibleUserId ?? - null, - }; - } - } - - const routine = await db - .select({ - env: routines.env, - responsibleUserId: routines.responsibleUserId, - }) - .from(routines) - .where( - and( - eq(routines.id, issueContext.originId), - eq(routines.companyId, companyId), - ), - ) - .then((rows) => rows[0] ?? null); - return { - routineId: issueContext.originId, - env: routine?.env ?? null, - responsibleUserId: - routineRun?.responsibleUserId ?? routine?.responsibleUserId ?? null, - }; - } - - async function resolveCompanyDefaultResponsibleUserId(companyId: string) { - const company = await db - .select({ defaultResponsibleUserId: companies.defaultResponsibleUserId }) - .from(companies) - .where(eq(companies.id, companyId)) - .then((rows) => rows[0] ?? null); - const explicitDefault = readNonEmptyString( - company?.defaultResponsibleUserId, - ); - if (explicitDefault) return explicitDefault; - - const owner = await db - .select({ userId: companyMemberships.principalId }) - .from(companyMemberships) - .where( - and( - eq(companyMemberships.companyId, companyId), - eq(companyMemberships.principalType, "user"), - eq(companyMemberships.status, "active"), - eq(companyMemberships.membershipRole, "owner"), - ), - ) - .orderBy(asc(companyMemberships.createdAt), asc(companyMemberships.id)) - .limit(1) - .then((rows) => rows[0] ?? null); - if (owner?.userId) return owner.userId; - - const firstUser = await db - .select({ userId: companyMemberships.principalId }) - .from(companyMemberships) - .where( - and( - eq(companyMemberships.companyId, companyId), - eq(companyMemberships.principalType, "user"), - eq(companyMemberships.status, "active"), - ), - ) - .orderBy(asc(companyMemberships.createdAt), asc(companyMemberships.id)) - .limit(1) - .then((rows) => rows[0] ?? null); - return firstUser?.userId ?? null; - } - - async function resolveParentIssueResponsibleUserId( - companyId: string, - parentId: string | null | undefined, - ) { - if (!parentId) return null; - const parent = await db - .select({ - responsibleUserId: issues.responsibleUserId, - createdByUserId: issues.createdByUserId, - }) - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.id, parentId))) - .then((rows) => rows[0] ?? null); - return parent?.responsibleUserId ?? null; - } - - function isManualUserRun(input: { - contextSnapshot: Record; - requestedByActorType?: "user" | "agent" | "system" | null; - source?: WakeupOptions["source"] | null; - triggerDetail?: WakeupOptions["triggerDetail"] | null; - }) { - if (input.requestedByActorType !== "user") return false; - const wakeReason = readNonEmptyString(input.contextSnapshot.wakeReason); - if (wakeReason && ISSUE_RESPONSIBLE_USER_WAKE_REASONS.has(wakeReason)) - return false; - return input.source === "on_demand" || input.triggerDetail === "manual"; - } - - async function resolveResponsibleUserIdForRunSeed(input: { - companyId: string; - contextSnapshot: Record; - issueContext: { id: string; responsibleUserId: string | null; parentId: string | null } | null; - routineEnvContext: Awaited< - ReturnType - >; - requestedByActorType?: "user" | "agent" | "system" | null; - requestedByActorId?: string | null; - source?: WakeupOptions["source"] | null; - triggerDetail?: WakeupOptions["triggerDetail"] | null; - existingRunResponsibleUserId?: string | null; - }) { - const contextResponsibleUserId = readNonEmptyString( - input.contextSnapshot.responsibleUserId, - ); - const requestedUserId = - input.requestedByActorType === "user" - ? readNonEmptyString(input.requestedByActorId) - : null; - const messageIds = Array.isArray(input.contextSnapshot.wakeCommentIds) - ? input.contextSnapshot.wakeCommentIds.filter( - (id): id is string => typeof id === "string", - ) - : readNonEmptyString(input.contextSnapshot.wakeCommentId) - ? [String(input.contextSnapshot.wakeCommentId)] - : []; - if ( - input.issueContext && - messageIds.length && - !input.contextSnapshot.retryOfRunId - ) { - const messages = await db - .select({ - id: issueComments.id, - authorUserId: issueComments.authorUserId, - }) - .from(issueComments) - .where( - and( - eq(issueComments.companyId, input.companyId), - eq(issueComments.issueId, input.issueContext.id), - inArray(issueComments.id, messageIds), - ), - ); - for (const id of [...messageIds].reverse()) { - const author = messages.find( - (message) => message.id === id, - )?.authorUserId; - if (author) { - delete input.contextSnapshot.executionIdentityCause; - return author; - } - } - } - const retryOfRunId = readNonEmptyString(input.contextSnapshot.retryOfRunId); - if (retryOfRunId) { - const [origin] = await db - .select({ responsibleUserId: heartbeatRuns.responsibleUserId }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.id, retryOfRunId), - ), - ); - if (origin?.responsibleUserId) return origin.responsibleUserId; - } - if (contextResponsibleUserId) return contextResponsibleUserId; - if (input.existingRunResponsibleUserId) - return input.existingRunResponsibleUserId; - if (input.routineEnvContext.responsibleUserId) - return input.routineEnvContext.responsibleUserId; - if (isManualUserRun(input) && requestedUserId) return requestedUserId; - if (input.issueContext?.responsibleUserId) - return input.issueContext.responsibleUserId; - const parentResponsibleUserId = await resolveParentIssueResponsibleUserId( - input.companyId, - input.issueContext?.parentId, - ); - if (parentResponsibleUserId) return parentResponsibleUserId; - if (!input.issueContext && requestedUserId) return requestedUserId; - input.contextSnapshot.executionIdentityCause = "company_default"; - return resolveCompanyDefaultResponsibleUserId(input.companyId); - } - - async function resolveResponsibleUserIdForRun(input: { - run: typeof heartbeatRuns.$inferSelect; - contextSnapshot: Record; - issueContext: Awaited> | null; - routineEnvContext: Awaited< - ReturnType - >; - }) { - const operatorIdentity = await explicitOperatorRunIdentity(db, input.run); - const responsibleUserId = operatorIdentity?.actorId ?? await resolveResponsibleUserIdForRunSeed({ - companyId: input.run.companyId, - contextSnapshot: input.contextSnapshot, - issueContext: input.issueContext, - routineEnvContext: input.routineEnvContext, - existingRunResponsibleUserId: input.run.responsibleUserId, - source: input.run.invocationSource as WakeupOptions["source"], - triggerDetail: input.run.triggerDetail as WakeupOptions["triggerDetail"], - }); - if (!responsibleUserId) { - throw new HttpError( - 422, - "Unable to resolve responsible user for heartbeat run dispatch", - { - code: "responsible_user_unresolved", - runId: input.run.id, - agentId: input.run.agentId, - companyId: input.run.companyId, - issueId: input.issueContext?.id ?? null, - invocationSource: input.run.invocationSource, - triggerDetail: input.run.triggerDetail, - wakeReason: readNonEmptyString(input.contextSnapshot.wakeReason), - }, - ); - } - return responsibleUserId; - } - - async function resolveResponsibleUserIdForRunContext( - run: typeof heartbeatRuns.$inferSelect, - contextSnapshot: Record, - ) { - const issueId = - readNonEmptyString(contextSnapshot.issueId) ?? - readNonEmptyString(contextSnapshot.taskId); - const issueContext = issueId - ? await getIssueExecutionContext(run.companyId, issueId) - : null; - return resolveResponsibleUserIdForRun({ - run, - contextSnapshot, - issueContext, - routineEnvContext: await getRoutineEnvForExecutionIssue( - run.companyId, - issueContext, - ), - }); - } - async function getRuntimeState(agentId: string) { return db .select() @@ -12660,7 +9613,16 @@ export function heartbeatService( const now = new Date(); const cancelled = await setRunStatusIfRunning(run.id, "cancelled", { error: error.message, errorCode: AI_CONNECTION_BUSY_RETRY_REASON, finishedAt: now, - resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } }, + resultJson: { + executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false }, + cancellation: { + source: "control_plane", + expected: true, + initiator: { type: "system" }, + reason: "Waiting for shared AI credentials", + recordedAt: now.toISOString(), + }, + }, contextSnapshot: { ...parseObject(run.contextSnapshot), aiConnectionBusyDeferredWhileAssignee: wasIssueAssignee, @@ -12718,6 +9680,13 @@ export function heartbeatService( kind: "workspace_wait", providerWorkStarted: false, }, + cancellation: { + source: "control_plane", + expected: true, + initiator: { type: "system" }, + reason: "Waiting for the shared project workspace", + recordedAt: now.toISOString(), + }, workspaceBusy: { projectWorkspaceId: deferral.projectWorkspaceId, holderRunId: deferral.holder.runId, @@ -17268,7 +14237,21 @@ export function heartbeatService( if (error instanceof AiConnectionPoolExhausted && !routerHasPersistedInput) { const now = new Date(); context.aiConnectionBusyDeferredWhileAssignee = issueContext?.assigneeAgentId === agent.id; - const cancelled = await setRunStatusIfRunning(run.id, "cancelled", { error: error.message, errorCode: error.code, finishedAt: now, resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false }, retryAt: error.retryAt }, contextSnapshot: context }); + const cancelled = await setRunStatusIfRunning(run.id, "cancelled", { + error: error.message, errorCode: error.code, finishedAt: now, + resultJson: { + executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false }, + cancellation: { + source: "control_plane", + expected: true, + initiator: { type: "system" }, + reason: "Waiting for an AI connection pool account", + recordedAt: now.toISOString(), + }, + retryAt: error.retryAt, + }, + contextSnapshot: context, + }); if (cancelled.updated) { await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: now, error: error.message }); const retry = await scheduleBoundedRetryForRun(cancelled.run ?? run, agent, { now, retryReason: AI_CONNECTION_POOL_WAIT_RETRY_REASON, wakeReason: "ai_connection_pool_retry", maxAttempts: (run.scheduledRetryAttempt ?? 0) + 1, delayMs: Math.max(1000, Date.parse(error.retryAt) - Date.now()) }); @@ -22952,8 +19935,10 @@ export function heartbeatService( sandboxProviderPluginNotReadySetupFailure, ) : null); + const connectionFailure = readGitConnectionFailure(outerErr); const setupFailureResultJson = { ...setupFailureDetails, + ...(connectionFailure ? { connectionFailure } : {}), executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, }; const setupFailureWrite = await setRunStatusIfRunning(runId, "failed", { diff --git a/server/src/services/heartbeat/run-preparation.test.ts b/server/src/services/heartbeat/run-preparation.test.ts new file mode 100644 index 00000000000..10a447e1a74 --- /dev/null +++ b/server/src/services/heartbeat/run-preparation.test.ts @@ -0,0 +1,141 @@ +import type { Db } from "@paperclipai/db"; +import { PgDialect } from "drizzle-orm/pg-core"; +import type { SQL } from "drizzle-orm"; +import { describe, expect, it, vi } from "vitest"; +import { + buildPaperclipRuntimeMcpServers as legacyBuildRuntimeMcpServers, + buildPaperclipWakePayload as legacyBuildWakePayload, + ConfigurationIncompleteFailure as LegacyConfigurationIncompleteFailure, + resolveExecutionRunAdapterConfig as legacyResolveAdapterConfig, +} from "../heartbeat.js"; +import { + buildPaperclipRuntimeMcpServers, + buildPaperclipWakePayload, + ConfigurationIncompleteFailure, + createHeartbeatRunPreparation, + resolveExecutionRunAdapterConfig, +} from "./run-preparation.js"; + +// Existing integration suites exercise the real database and dispatch path. +// These results isolate the factory binding and preparation query order. +function databaseWithResults(...results: unknown[][]) { + const scopes: unknown[][] = []; + const select = vi.fn(() => { + const rows = results.shift() ?? []; + const query = { + from: () => query, + leftJoin: () => query, + where: (condition: SQL) => { + scopes.push(new PgDialect().sqlToQuery(condition).params); + return query; + }, + orderBy: () => query, + limit: () => query, + then: (resolve: (value: unknown[]) => unknown) => Promise.resolve(rows).then(resolve), + }; + return query; + }); + return { db: { select } as unknown as Db, select, scopes }; +} + +const emptyRoutineContext = { routineId: null, env: null, responsibleUserId: null }; + +describe("heartbeat run preparation module", () => { + it("preserves the legacy error class and public preparation functions", () => { + expect(LegacyConfigurationIncompleteFailure).toBe(ConfigurationIncompleteFailure); + expect(legacyResolveAdapterConfig).toBe(resolveExecutionRunAdapterConfig); + expect(legacyBuildWakePayload).toBe(buildPaperclipWakePayload); + expect(legacyBuildRuntimeMcpServers).toBe(buildPaperclipRuntimeMcpServers); + const failure = new ConfigurationIncompleteFailure("missing credentials", { reason: "test" }); + expect(failure).toBeInstanceOf(LegacyConfigurationIncompleteFailure); + expect(failure.code).toBe("configuration_incomplete"); + }); + + it("binds each context loader to its database and company scope", async () => { + const first = databaseWithResults([{ id: "issue-one", title: "First company" }]); + const second = databaseWithResults([{ id: "issue-two", title: "Second company" }]); + const firstPreparation = createHeartbeatRunPreparation(first.db); + const secondPreparation = createHeartbeatRunPreparation(second.db); + expect(first.select).not.toHaveBeenCalled(); + expect(second.select).not.toHaveBeenCalled(); + const [firstIssue, secondIssue] = await Promise.all([ + firstPreparation.getIssueExecutionContext("company-one", "issue-one"), + secondPreparation.getIssueExecutionContext("company-two", "issue-two"), + ]); + expect(firstIssue).toMatchObject({ id: "issue-one", title: "First company" }); + expect(secondIssue).toMatchObject({ id: "issue-two", title: "Second company" }); + expect(first.scopes[0]).toEqual(expect.arrayContaining(["company-one", "issue-one"])); + expect(second.scopes[0]).toEqual(expect.arrayContaining(["company-two", "issue-two"])); + expect(first.select).toHaveBeenCalledTimes(1); + expect(second.select).toHaveBeenCalledTimes(1); + }); + + it("keeps the pinned routine environment and original run identity", async () => { + const database = databaseWithResults( + [{ routineRevisionId: "revision-one", responsibleUserId: "run-owner" }], + [{ snapshot: { version: 1, routine: { env: { PINNED_FLAG: "original" }, responsibleUserId: "snapshot-owner" } }, responsibleUserId: "revision-owner" }], + ); + const result = await createHeartbeatRunPreparation(database.db).getRoutineEnvForExecutionIssue( + "company-one", + { originKind: "routine_execution", originId: "routine-one", originRunId: "routine-run-one" }, + ); + expect(result).toEqual({ routineId: "routine-one", env: { PINNED_FLAG: "original" }, responsibleUserId: "run-owner" }); + expect(database.scopes[0]).toEqual(expect.arrayContaining(["company-one", "routine-one", "routine-run-one"])); + expect(database.scopes[1]).toEqual(expect.arrayContaining(["company-one", "routine-one", "revision-one"])); + expect(database.select).toHaveBeenCalledTimes(2); + }); + + it("uses the latest wake comment's author even when query rows arrive out of order", async () => { + const database = databaseWithResults([ + { id: "comment-two", authorUserId: "latest-author" }, + { id: "comment-one", authorUserId: "older-author" }, + ]); + const contextSnapshot: Record = { + wakeCommentIds: ["comment-one", "comment-two"], + responsibleUserId: "previous-owner", + executionIdentityCause: "company_default", + }; + const owner = await createHeartbeatRunPreparation(database.db).resolveResponsibleUserIdForRunSeed({ + companyId: "company-one", contextSnapshot, + issueContext: { id: "issue-one", responsibleUserId: "issue-owner", parentId: null }, + routineEnvContext: emptyRoutineContext, + }); + expect(owner).toBe("latest-author"); + expect(contextSnapshot).not.toHaveProperty("executionIdentityCause"); + expect(database.scopes[0]).toEqual(expect.arrayContaining(["company-one", "issue-one"])); + expect(database.select).toHaveBeenCalledTimes(1); + }); + + it("resolves a system wake through the same company's active owner", async () => { + const database = databaseWithResults([{ defaultResponsibleUserId: null }], [{ userId: "company-owner" }]); + const contextSnapshot: Record = {}; + const owner = await createHeartbeatRunPreparation(database.db).resolveResponsibleUserIdForRunSeed({ + companyId: "company-one", contextSnapshot, issueContext: null, + routineEnvContext: emptyRoutineContext, requestedByActorType: "system", + }); + expect(owner).toBe("company-owner"); + expect(contextSnapshot.executionIdentityCause).toBe("company_default"); + expect(database.scopes[0]).toContain("company-one"); + expect(database.scopes[1]).toEqual(expect.arrayContaining(["company-one", "user", "active", "owner"])); + expect(database.select).toHaveBeenCalledTimes(2); + }); + + it("keeps missing scoped credentials as a pre-dispatch failure", async () => { + const secretsSvc = { + resolveAdapterConfigForRuntime: vi.fn(), + resolveEnvBindings: vi.fn(), + collectMissingRuntimeBindings: vi.fn(), + collectMissingAdapterConfigRuntimeBindings: vi.fn(), + }; + await expect(resolveExecutionRunAdapterConfig({ + companyId: "company-one", executionRunConfig: { env: {} }, projectEnv: null, + requiredScopedEnvBinding: { + keys: ["GH_TOKEN"], consumerScopes: ["agent", "project"], + reason: "github_credentials_missing", remediation: "Configure GitHub access.", + }, + secretsSvc, + })).rejects.toBeInstanceOf(LegacyConfigurationIncompleteFailure); + expect(secretsSvc.resolveAdapterConfigForRuntime).not.toHaveBeenCalled(); + expect(secretsSvc.resolveEnvBindings).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/heartbeat/run-preparation.ts b/server/src/services/heartbeat/run-preparation.ts new file mode 100644 index 00000000000..de22ffee6fa --- /dev/null +++ b/server/src/services/heartbeat/run-preparation.ts @@ -0,0 +1,3237 @@ +import { CONFIGURED_ENVIRONMENT_KEYS } from "../../vendor/paperclip-runner/index.js"; +import { ASSIGNED_MCP_SERVER_NAME } from "../mcp-tool-names.js"; +import { externalConversationStateSql } from "../slack-conversation-state.js"; +import { githubBotConnectionIdsForRun } from "../chat-github-tools.js"; +import { isBrowserUseConnection } from "../browser-use-client.js"; +import { explicitOperatorRunIdentity } from "../run-identity.js"; +import { + type DurableChatWakeupRequest, +} from "../durable-chat-wakeup.js"; +import { createHash } from "node:crypto"; +import { + and, + asc, + desc, + eq, + inArray, + isNull, + sql, +} from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { + CHAT_PROVIDERS, + CONNECTION_INTENT_AGENT_GUIDANCE, + CONNECTION_RUNTIME_TOOL_NAMES, + envBindingSchema, + isToolConnectionAttentionHealth, + type ChatProvider, + type RoutineRevisionSnapshotV1, + type SourceTrustMetadata, +} from "@paperclipai/shared"; +import { + agents, + assets, + chatConversations, + chatDeliveries, + chatEndpoints, + chatMessageLinks, + companyMemberships, + companySkillTestRuns, + companySkillVersions, + companySkills as companySkillsTable, + companies, + documentAnnotationComments, + documentAnnotationThreads, + heartbeatRuns, + issueAttachments, + issueComments, + issuePlanDecompositions, + issueRecoveryActions, + issues, + routineRevisions, + routineRuns, + routines, + toolMcpGateways, + toolMcpGatewayTokens, + toolCatalogEntries, + toolConnectionInstalls, + toolConnections, + toolProfileEntries, + toolProfiles, +} from "@paperclipai/db"; +import { HttpError } from "../../errors.js"; +import { filterResolvedGitHubConnectionsForRun } from "../git-credentials.js"; +import type { + AdapterRuntimeMcpAccess, + AdapterRuntimeMcpServer, + AdapterRuntimeToolAccess, +} from "../../adapters/index.js"; +import { createRuntimeToolsToken } from "../../runtime-tools-token.js"; +import { parseObject } from "../../adapters/utils.js"; +import { + authorizeChatConversationForBoundRun, + isExternalChatWaitAuthorizationContention, +} from "../native-runtime/chat-attachment-reuse.js"; +import { + EXTERNAL_CHAT_QUESTION_RESPONSE_KEY, + resolveExternalChatQuestionResponse, +} from "../native-runtime/external-chat-question-response.js"; +import { + secretService, + type MissingRuntimeBinding, +} from "../secrets.js"; +import { createToolGatewayService } from "../tool-gateway.js"; +import { toolAccessService } from "../tool-access.js"; +import { + buildDocumentReviewContext, + buildPlanReviewContext, +} from "../plan-review-context.js"; +import { + WAKE_COMMENT_IDS_KEY, + extractWakeCommentIds, + deriveCommentId, +} from "../../modules/run-dispatch/index.js"; +import { + DIRECT_NON_INVOKABLE_STATUSES, + type AgentOrgRow, +} from "../agent-invokability.js"; +import { + isLowTrustQuarantined, + redactQuarantinedBodyForHigherTrust, + sanitizeQuarantinedCommentForHigherTrust, +} from "../source-trust.js"; +import { redactSensitiveText } from "../../redaction.js"; +import { createRunSecretRedactionRegistry } from "../run-secret-redaction.js"; +import { + readPaperclipSkillSyncPreference, + writePaperclipSkillSyncPreference, +} from "@paperclipai/adapter-utils/server-utils"; +import { + extractSkillMentionIds, + isUuidLike, +} from "@paperclipai/shared"; +import { evaluateCodexCredentialReadiness } from "@paperclipai/adapter-codex-local/server"; +import type { TrustPresetResolution } from "../trust-preset-resolver.js"; + +const EXTERNAL_ATTACHMENT_OMISSIONS_KEY = "externalAttachmentOmissions"; + +export const PAPERCLIP_WAKE_PAYLOAD_KEY = "paperclipWake"; + +const PAPERCLIP_AGENT_MESSAGE_KEY = "paperclipAgentMessage"; + +export const PAPERCLIP_HARNESS_CHECKOUT_KEY = "paperclipHarnessCheckedOut"; + +export const PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY = + "paperclipExternalChatExecutionBound"; + +const MAX_INLINE_WAKE_COMMENTS = 8; + +const MAX_INLINE_WAKE_ATTACHMENTS = 20; + +const MAX_INLINE_WAKE_COMMENT_BODY_CHARS = 4_000; + +const MAX_INLINE_WAKE_COMMENT_BODY_TOTAL_CHARS = 12_000; + +const MAX_INLINE_WAKE_ISSUE_DESCRIPTION_CHARS = 12_000; + +const MAX_AGENT_SESSION_MESSAGE_CHARS = 12_000; + +export const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete"; + +// Pre-dispatch gate outcome: required secret/env bindings are missing, so the +// run must not be dispatched. Surfaced as a configuration-incomplete blocker +// routed to a human owner instead of N opaque dispatched-then-failed runs. +export class ConfigurationIncompleteFailure extends Error { + code = CONFIGURATION_INCOMPLETE_FAILURE_CODE; + resultJson: Record; + + constructor(message: string, resultJson: Record) { + super(message); + this.name = "ConfigurationIncompleteFailure"; + this.resultJson = resultJson; + } +} + +const ISSUE_RESPONSIBLE_USER_WAKE_REASONS = new Set([ + "issue_assigned", + "issue_checked_out", + "issue_commented", + "issue_comment_mentioned", + "issue_reopened_via_comment", + "issue_blockers_resolved", + "issue_children_completed", + "issue_status_changed", + "issue_tree_restored", + "issue_recovery_action_restored", + "execution_review_requested", + "execution_approval_requested", + "execution_changes_requested", + "approval_approved", +]); + +type RuntimeConfigSecretResolver = Pick< + ReturnType, + | "resolveAdapterConfigForRuntime" + | "resolveEnvBindings" + | "collectMissingRuntimeBindings" + | "collectMissingAdapterConfigRuntimeBindings" +>; + +function formatMissingBindingForOperator( + missing: MissingRuntimeBinding, +): string { + if (missing.bindingType === "user_secret_ref") { + const definitionLabel = missing.userSecretDefinitionName + ? `"${missing.userSecretDefinitionName}"` + : missing.userSecretDefinitionKey + ? `"${missing.userSecretDefinitionKey}"` + : "declared user secret"; + const ownerLabel = missing.responsibleUserId + ? ` for responsible user ${missing.responsibleUserId}` + : ""; + return `user secret ${definitionLabel}${ownerLabel} not available at ${missing.consumerType} ${missing.configPath}`; + } + const secretLabel = missing.secretName + ? `"${missing.secretName}"` + : (missing.secretId ?? "unknown"); + return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`; +} + +function isConfiguredEnvBindingValue(binding: unknown) { + const parsed = envBindingSchema.safeParse(binding); + if (!parsed.success) return false; + const value = parsed.data; + if (typeof value === "string") return value.trim().length > 0; + if (value.type === "plain") return value.value.trim().length > 0; + return true; +} + +const LOW_TRUST_SENSITIVE_ENV_KEY_RE = + /(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i; + +// PAPERCLIP_* env binding policy: +// 1. PAPERCLIP_API_KEY is never accepted from user/adapter/project/routine +// config — the harness-minted run token is the only source. +// 2. A PAPERCLIP_* runtime var the harness assigns for the run (RUN_ID, +// AGENT_ID, wake/workspace vars, ...) always wins over a same-named +// binding; adapters enforce this at env-merge time. +// 3. Any other PAPERCLIP_*-named binding is user data and flows through to +// the run env like any non-prefixed binding. +const FORBIDDEN_ENV_BINDING_KEYS = new Set([ + CONFIGURED_ENVIRONMENT_KEYS, + "PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY", + "PAPERCLIP_RUNNER_NETWORK_ACCESS", + "PAPERCLIP_RUNNER_NETWORK_ROOTS", + "PAPERCLIP_API_KEY", + "PAPERCLIP_GITHUB_AUTH_MODE", + "PAPERCLIP_GITHUB_HOST_HOME", + "PAPERCLIP_GIT_METADATA_ROOTS", + "PAPERCLIP_GITHUB_BROKER_TOKEN", + "PAPERCLIP_GITHUB_BROKER_URL", + "PAPERCLIP_GITHUB_BRIDGE_TOKEN", + "PAPERCLIP_GITHUB_LAUNCHER_DIR", +]); + +export const MANAGED_GITHUB_TOKEN_KEYS = new Set([ + "GH_TOKEN", + "GITHUB_TOKEN", + "GH_ENTERPRISE_TOKEN", + "GITHUB_ENTERPRISE_TOKEN", + "PAPERCLIP_GIT_TOKEN", +]); + +function stripForbiddenEnvBindings( + envValue: unknown, + managedGitHubCredentials = false, +): Record | null { + const record = parseObject(envValue); + const filtered = Object.fromEntries( + Object.entries(record).filter( + ([key]) => + !FORBIDDEN_ENV_BINDING_KEYS.has(key.toUpperCase()) && + !(managedGitHubCredentials && MANAGED_GITHUB_TOKEN_KEYS.has(key)), + ), + ); + return Object.keys(filtered).length > 0 ? filtered : null; +} + +function stripForbiddenEnvFromAdapterConfig( + config: Record, + managedGitHubCredentials = false, +): Record { + if (!Object.prototype.hasOwnProperty.call(config, "env")) return config; + return { + ...config, + env: stripForbiddenEnvBindings(config.env, managedGitHubCredentials) ?? {}, + }; +} + +function assertLowTrustEnvConfigAllowed(envValue: unknown, source: string) { + const record = stripForbiddenEnvBindings(envValue); + if (!record) return; + for (const [key, rawBinding] of Object.entries(record)) { + const parsed = envBindingSchema.safeParse(rawBinding); + if (!parsed.success) continue; + const binding = parsed.data; + const isPlainBinding = + typeof binding === "string" || + (typeof binding === "object" && + binding !== null && + binding.type === "plain"); + if (isPlainBinding && LOW_TRUST_SENSITIVE_ENV_KEY_RE.test(key)) { + throw new HttpError( + 422, + `Low-trust execution cannot use inline sensitive env value ${source}.${key}`, + { + code: "low_trust_inline_sensitive_env_denied", + }, + ); + } + } +} + +export async function resolveExecutionRunAdapterConfig(input: { + managedAiCredentials?: boolean; + companyId: string; + agentId?: string | null; + adapterType?: string | null; + issueId?: string | null; + heartbeatRunId?: string | null; + responsibleUserId?: string | null; + environmentId?: string | null; + environmentEnv?: unknown; + environmentDriver?: string | null; + projectId?: string | null; + routineId?: string | null; + executionRunConfig: Record; + projectEnv: unknown; + routineEnv?: unknown; + secretsSvc: RuntimeConfigSecretResolver; + trustPreset?: TrustPresetResolution; + requiredScopedEnvBinding?: { + keys: string[]; + consumerScopes: Array<"agent" | "project">; + reason: string; + remediation: string; + }; + /** Managed GitHub tokens are resolved only when operations start. */ + managedGitHubCredentials?: boolean; + /** Audited class-3 values resolved by an internal credential broker. */ + trustedEnvProjection?: Record; + trustedEnvSecretKeys?: string[]; +}) { + const executionRunConfig = stripForbiddenEnvFromAdapterConfig( + input.executionRunConfig, + input.managedGitHubCredentials, + ); + const environmentEnv = stripForbiddenEnvBindings( + input.environmentEnv, + input.managedGitHubCredentials, + ); + const projectEnv = stripForbiddenEnvBindings( + input.projectEnv, + input.managedGitHubCredentials, + ); + const routineEnv = stripForbiddenEnvBindings( + input.routineEnv, + input.managedGitHubCredentials, + ); + const agentEnv = parseObject(executionRunConfig.env); + const lowTrustAllowedBindingIds = + input.trustPreset?.kind === "low_trust_review" + ? (input.trustPreset.boundary.allowedSecretBindingIds ?? []) + : undefined; + const allowTrustedEnvProjection = + input.trustPreset?.kind !== "low_trust_review"; + if (input.trustPreset?.kind === "low_trust_review") { + assertLowTrustEnvConfigAllowed(environmentEnv, "environment.env"); + assertLowTrustEnvConfigAllowed(executionRunConfig.env, "agent.env"); + assertLowTrustEnvConfigAllowed(projectEnv, "project.env"); + assertLowTrustEnvConfigAllowed(routineEnv, "routine.env"); + } + const requiredScopedEnvBinding = input.requiredScopedEnvBinding ?? null; + const requiredScopedBindingsConfigured = requiredScopedEnvBinding + ? requiredScopedEnvBinding.keys.some( + (key) => + (allowTrustedEnvProjection && + typeof input.trustedEnvProjection?.[key] === "string") || + (requiredScopedEnvBinding.consumerScopes.includes("agent") && + isConfiguredEnvBindingValue(agentEnv[key])) || + (requiredScopedEnvBinding.consumerScopes.includes("project") && + isConfiguredEnvBindingValue(projectEnv?.[key])), + ) + : false; + if (requiredScopedEnvBinding && !requiredScopedBindingsConfigured) { + throw new ConfigurationIncompleteFailure( + `configuration incomplete: ${requiredScopedEnvBinding.remediation}`, + { + configurationIncomplete: { + reason: requiredScopedEnvBinding.reason, + companyId: input.companyId, + agentId: input.agentId ?? null, + issueId: input.issueId ?? null, + projectId: input.projectId ?? null, + routineId: input.routineId ?? null, + requiredEnvKeys: requiredScopedEnvBinding.keys, + requiredScopes: requiredScopedEnvBinding.consumerScopes, + missingBindings: [], + }, + }, + ); + } + // Pre-dispatch binding-validation gate: detect declared secret refs that have + // no binding before resolving any secret value. Missing bindings short-circuit + // to a configuration-incomplete blocker routed to a human owner instead of a + // dispatched-then-failed run (which previously surfaced as opaque setup_failed). + if (typeof input.secretsSvc.collectMissingRuntimeBindings === "function") { + const missingBindings: MissingRuntimeBinding[] = []; + if (environmentEnv && input.environmentId) { + missingBindings.push( + ...(await input.secretsSvc.collectMissingRuntimeBindings( + input.companyId, + environmentEnv, + { + consumerType: "environment", + consumerId: input.environmentId, + responsibleUserId: input.responsibleUserId ?? null, + }, + )), + ); + } + if (input.agentId) { + missingBindings.push( + ...(await input.secretsSvc.collectMissingRuntimeBindings( + input.companyId, + parseObject(executionRunConfig.env), + { + consumerType: "agent", + consumerId: input.agentId, + responsibleUserId: input.responsibleUserId ?? null, + }, + )), + ); + if ( + typeof input.secretsSvc.collectMissingAdapterConfigRuntimeBindings === + "function" + ) { + missingBindings.push( + ...(await input.secretsSvc.collectMissingAdapterConfigRuntimeBindings( + input.companyId, + executionRunConfig, + input.adapterType ?? null, + { + consumerType: "agent", + consumerId: input.agentId, + responsibleUserId: input.responsibleUserId ?? null, + }, + )), + ); + } + } + if (projectEnv && input.projectId) { + missingBindings.push( + ...(await input.secretsSvc.collectMissingRuntimeBindings( + input.companyId, + projectEnv, + { + consumerType: "project", + consumerId: input.projectId, + responsibleUserId: input.responsibleUserId ?? null, + }, + )), + ); + } + if (routineEnv && input.routineId) { + missingBindings.push( + ...(await input.secretsSvc.collectMissingRuntimeBindings( + input.companyId, + routineEnv, + { + consumerType: "routine", + consumerId: input.routineId, + responsibleUserId: input.responsibleUserId ?? null, + }, + )), + ); + } + if (requiredScopedEnvBinding) { + const requiredEnvKeys = new Set(requiredScopedEnvBinding.keys); + const requiredScopes = new Set(requiredScopedEnvBinding.consumerScopes); + const requiredMissingBindings = missingBindings.filter( + (binding) => + requiredScopes.has(binding.consumerType as "agent" | "project") && + requiredEnvKeys.has(binding.envKey), + ); + if (requiredMissingBindings.length > 0) { + const detail = requiredMissingBindings + .map(formatMissingBindingForOperator) + .join("; "); + throw new ConfigurationIncompleteFailure( + `configuration incomplete: ${requiredScopedEnvBinding.remediation}; ${detail}`, + { + configurationIncomplete: { + reason: requiredScopedEnvBinding.reason, + companyId: input.companyId, + agentId: input.agentId ?? null, + issueId: input.issueId ?? null, + projectId: input.projectId ?? null, + routineId: input.routineId ?? null, + requiredEnvKeys: requiredScopedEnvBinding.keys, + requiredScopes: requiredScopedEnvBinding.consumerScopes, + missingBindings: requiredMissingBindings, + }, + }, + ); + } + } + if (missingBindings.length > 0) { + const detail = missingBindings + .map(formatMissingBindingForOperator) + .join("; "); + throw new ConfigurationIncompleteFailure( + `configuration incomplete: ${detail}`, + { + configurationIncomplete: { + reason: "secret_binding_missing", + companyId: input.companyId, + agentId: input.agentId ?? null, + issueId: input.issueId ?? null, + projectId: input.projectId ?? null, + routineId: input.routineId ?? null, + missingBindings, + }, + }, + ); + } + } + const environmentEnvResolution = environmentEnv + ? await input.secretsSvc.resolveEnvBindings( + input.companyId, + environmentEnv, + input.environmentId + ? { + consumerType: "environment", + consumerId: input.environmentId, + actorType: "agent", + actorId: input.agentId ?? null, + responsibleUserId: input.responsibleUserId ?? null, + issueId: input.issueId ?? null, + heartbeatRunId: input.heartbeatRunId ?? null, + ...(lowTrustAllowedBindingIds !== undefined + ? { allowedBindingIds: lowTrustAllowedBindingIds } + : {}), + } + : undefined, + ) + : { env: {}, secretKeys: new Set(), manifest: [] }; + const { + config: resolvedConfig, + secretKeys, + manifest, + } = await input.secretsSvc.resolveAdapterConfigForRuntime( + input.companyId, + executionRunConfig, + input.agentId + ? { + consumerType: "agent", + consumerId: input.agentId, + actorType: "agent", + actorId: input.agentId, + responsibleUserId: input.responsibleUserId ?? null, + issueId: input.issueId ?? null, + heartbeatRunId: input.heartbeatRunId ?? null, + ...(lowTrustAllowedBindingIds !== undefined + ? { allowedBindingIds: lowTrustAllowedBindingIds } + : {}), + } + : undefined, + { adapterType: input.adapterType ?? null }, + ); + if (Object.keys(environmentEnvResolution.env).length > 0) { + resolvedConfig.env = { + ...environmentEnvResolution.env, + ...parseObject(resolvedConfig.env), + }; + for (const key of environmentEnvResolution.secretKeys) { + secretKeys.add(key); + } + } + const projectEnvResolution = projectEnv + ? await input.secretsSvc.resolveEnvBindings( + input.companyId, + projectEnv, + input.projectId + ? { + consumerType: "project", + consumerId: input.projectId, + actorType: "agent", + actorId: input.agentId ?? null, + responsibleUserId: input.responsibleUserId ?? null, + issueId: input.issueId ?? null, + heartbeatRunId: input.heartbeatRunId ?? null, + ...(lowTrustAllowedBindingIds !== undefined + ? { allowedBindingIds: lowTrustAllowedBindingIds } + : {}), + } + : undefined, + ) + : { env: {}, secretKeys: new Set(), manifest: [] }; + if (Object.keys(projectEnvResolution.env).length > 0) { + resolvedConfig.env = { + ...parseObject(resolvedConfig.env), + ...projectEnvResolution.env, + }; + for (const key of projectEnvResolution.secretKeys) { + secretKeys.add(key); + } + } + const routineEnvResolution = routineEnv + ? await input.secretsSvc.resolveEnvBindings( + input.companyId, + routineEnv, + input.routineId + ? { + consumerType: "routine", + consumerId: input.routineId, + actorType: "agent", + actorId: input.agentId ?? null, + responsibleUserId: input.responsibleUserId ?? null, + issueId: input.issueId ?? null, + heartbeatRunId: input.heartbeatRunId ?? null, + ...(lowTrustAllowedBindingIds !== undefined + ? { allowedBindingIds: lowTrustAllowedBindingIds } + : {}), + } + : undefined, + ) + : { env: {}, secretKeys: new Set(), manifest: [] }; + if (Object.keys(routineEnvResolution.env).length > 0) { + resolvedConfig.env = { + ...parseObject(resolvedConfig.env), + ...routineEnvResolution.env, + }; + for (const key of routineEnvResolution.secretKeys) { + secretKeys.add(key); + } + } + if ( + allowTrustedEnvProjection && + input.trustedEnvProjection && + Object.keys(input.trustedEnvProjection).length > 0 + ) { + resolvedConfig.env = { + ...parseObject(resolvedConfig.env), + ...input.trustedEnvProjection, + }; + for (const key of input.trustedEnvSecretKeys ?? []) secretKeys.add(key); + } + // Pre-dispatch credential gate for codex_local: a managed Codex home with no + // usable auth.json and an empty OPENAI_API_KEY would dispatch a run that + // immediately fails with "no Codex credentials provisioned" (adapter_failed), + // making a configuration problem look like a runtime failure. Surface it as a + // configuration-incomplete blocker instead, naming the missing credential + // action and owner without leaking any secret value. This runs after secret + // resolution so a per-agent OPENAI_API_KEY (plain or resolved secret) counts + // as satisfying the credential. It shares the exact readiness predicate the + // adapter uses at execute time, so the two cannot drift. + // + // Sandbox-destined runs are exempt: the sandbox image may carry its own + // Codex login (`~/.codex/auth.json` baked in at image setup), which only the + // adapter can probe once the sandbox is up — and on managed cloud hosts a + // host-side login never exists at all. The adapter's execute-time gate + // remains the authority there; it probes the sandbox before failing. + if ( + !input.managedAiCredentials && (input.adapterType ?? null) === "codex_local" && + (input.environmentDriver ?? null) !== "sandbox" + ) { + const resolvedEnv = parseObject(resolvedConfig.env); + const readiness = await evaluateCodexCredentialReadiness({ + env: process.env, + companyId: input.companyId, + configuredCodexHome: readNonEmptyString(resolvedEnv.CODEX_HOME), + configuredApiKey: readNonEmptyString(resolvedEnv.OPENAI_API_KEY), + }); + if (readiness.managed && !readiness.ready) { + throw new ConfigurationIncompleteFailure( + `configuration incomplete: no Codex credentials available for managed home "${readiness.effectiveHome}". ` + + `Sign in to Codex on the host with a ChatGPT subscription, or bind a per-agent OPENAI_API_KEY secret for this agent.`, + { + configurationIncomplete: { + reason: "codex_credentials_missing", + companyId: input.companyId, + agentId: input.agentId ?? null, + issueId: input.issueId ?? null, + projectId: input.projectId ?? null, + routineId: input.routineId ?? null, + responsibleUserId: input.responsibleUserId ?? null, + adapterType: "codex_local", + requiredEnvKeys: ["OPENAI_API_KEY"], + effectiveCodexHome: readiness.effectiveHome, + missingBindings: [], + }, + }, + ); + } + } + return { + resolvedConfig, + // Capture resolved task values before provider credential injection or host + // inheritance. Native-only launch limits are checked at native dispatch. + configuredTaskEnvironment: Object.fromEntries( + Object.entries(parseObject(resolvedConfig.env)).filter((entry): entry is [string, string] => typeof entry[1] === "string"), + ), + secretKeys, + secretManifest: [ + ...(environmentEnvResolution.manifest ?? []), + ...(manifest ?? []), + ...(projectEnvResolution.manifest ?? []), + ...(routineEnvResolution.manifest ?? []), + ], + }; +} + +export function extractMentionedSkillIdsFromSources( + sources: Array, +): string[] { + const mentionedIds = new Set(); + for (const source of sources) { + if (typeof source !== "string" || source.length === 0) continue; + for (const skillId of extractSkillMentionIds(source)) { + if (!isUuidLike(skillId)) continue; + mentionedIds.add(skillId); + } + } + return [...mentionedIds]; +} + +export function applyRunScopedMentionedSkillKeys( + config: Record, + skillKeys: string[], +): Record { + const normalizedSkillKeys = Array.from( + new Set(skillKeys.map((value) => value.trim()).filter(Boolean)), + ); + if (normalizedSkillKeys.length === 0) return config; + + const existingPreference = readPaperclipSkillSyncPreference(config); + return writePaperclipSkillSyncPreference(config, [ + ...existingPreference.desiredSkillEntries, + ...normalizedSkillKeys, + ]); +} + +export async function resolveRunScopedMentionedSkillKeys(input: { + db: Db; + companyId: string; + issueId: string | null; +}): Promise { + if (!input.issueId) return []; + + const issue = await input.db + .select({ + title: issues.title, + description: issues.description, + }) + .from(issues) + .where( + and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)), + ) + .then((rows) => rows[0] ?? null); + if (!issue) return []; + + const comments = await input.db + .select({ body: issueComments.body }) + .from(issueComments) + .where( + and( + eq(issueComments.issueId, input.issueId), + eq(issueComments.companyId, input.companyId), + isNull(issueComments.deletedAt), + ), + ); + const mentionedSkillIds = extractMentionedSkillIdsFromSources([ + issue.title, + issue.description ?? "", + ...comments.map((comment) => comment.body), + ]); + if (mentionedSkillIds.length === 0) return []; + + const skillRows = await input.db + .select({ + id: companySkillsTable.id, + key: companySkillsTable.key, + }) + .from(companySkillsTable) + .where( + and( + eq(companySkillsTable.companyId, input.companyId), + inArray(companySkillsTable.id, mentionedSkillIds), + ), + ); + const skillKeyById = new Map(skillRows.map((row) => [row.id, row.key])); + return mentionedSkillIds + .map((skillId) => skillKeyById.get(skillId) ?? null) + .filter((skillKey): skillKey is string => Boolean(skillKey)); +} + +export interface WakeupOptions { + /** Set only by authenticated board wake routes; never copied from caller payloads. */ + manualUserWake?: boolean; + /** Internal resume of a queue with persisted board interruption intent. */ + queuedCommentInterruptId?: string; + /** Internal delivery of an existing undelivered user comment. */ + queuedCommentRequestId?: string; + /** Exact failed run selected by an authenticated board Retry request. */ + failedRunId?: string | null; + durableChatRequest?: DurableChatWakeupRequest; + /** Server-owned Dot admission receipt; never accepted from API payloads. */ + durableDotRequest?: { + id: string; + companyId: string; + agentId: string; + issueId: string; + requestId: string; + idempotencyKey: string; + requestedAt: Date; + }; + source?: "timer" | "assignment" | "on_demand" | "automation"; + triggerDetail?: "manual" | "ping" | "callback" | "system"; + reason?: string | null; + payload?: Record | null; + idempotencyKey?: string | null; + requestedByActorType?: "user" | "agent" | "system"; + requestedByActorId?: string | null; + contextSnapshot?: Record; + issueStateGuard?: { + statuses: string[]; + assigneeAgentId: string; + statusVersion?: number; + monitorNextCheckAt?: string; + monitorWakeRequestedAt?: string; + }; + /** Keep causally distinct external chat continuations out of an existing run. */ + allowRunCoalescing?: boolean; +} + +function sanitizeAgentSessionMessageText(value: unknown): string | null { + const text = readNonEmptyString(value); + if (!text) return null; + const redacted = redactSensitiveText(text).slice( + 0, + MAX_AGENT_SESSION_MESSAGE_CHARS, + ); + return redacted.trim().length > 0 ? redacted : null; +} + +type ManagedMcpGatewayRunConfig = { + version: 1; + managedMcpOnly: boolean; + gateways: Array<{ + id: string; + name: string; + endpointPath: string; + bearerToken: string; + tokenPrefix: string; + }>; +}; + +export function configuredPaperclipApiBaseUrl(): string | null { + const configured = readNonEmptyString(process.env.PAPERCLIP_API_URL); + return configured + ? configured.replace(/\/+$/, "").replace(/\/api$/, "") + : null; +} + +export function paperclipApiBaseUrl(): string { + const configured = configuredPaperclipApiBaseUrl(); + if (!configured) { + throw new Error( + "PAPERCLIP_API_URL is required to deliver managed runtime MCP servers", + ); + } + return configured; +} + +export async function revokeHeartbeatRunGatewayTokens(input: { + db: Db; + companyId: string; + runId: string; +}): Promise { + const now = new Date(); + await input.db + .update(toolMcpGatewayTokens) + .set({ revokedAt: now, updatedAt: now }) + .where( + and( + eq(toolMcpGatewayTokens.companyId, input.companyId), + eq(toolMcpGatewayTokens.subjectType, "heartbeat_run"), + eq(toolMcpGatewayTokens.subjectId, input.runId), + isNull(toolMcpGatewayTokens.revokedAt), + ), + ); +} + +export async function buildPaperclipRuntimeMcpServers(input: { + db: Db; + agent: Pick; + runId: string; + expectedAssignmentDigest?: string | null; +}): Promise { + const access = toolAccessService(input.db); + const effective = await access.getEffectiveProfilesForAgent( + input.agent.companyId, + input.agent.id, + ); + const [runIdentity] = await input.db + .select({ + responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.agent.companyId), + eq(heartbeatRuns.agentId, input.agent.id), + ), + ) + .limit(1); + const resolvedInstalledConnections = runIdentity?.activeIdentityContextId + ? effective.installedConnections + : await filterResolvedGitHubConnectionsForRun({ + db: input.db, + companyId: input.agent.companyId, + agentId: input.agent.id, + responsibleUserId: runIdentity?.responsibleUserId ?? null, + connections: effective.installedConnections, + }); + const permittedConnectionIds = new Set([ + ...effective.entries + .filter((entry) => entry.effect === "include" && entry.connectionId) + .map((entry) => entry.connectionId!), + ...effective.allowedTools.map((tool) => tool.connectionId), + ]); + const allInstalledConnectionIds = new Set( + effective.installedConnections.map((connection) => connection.id), + ); + const permittedConnections = + permittedConnectionIds.size > 0 + ? await input.db + .select({ + id: toolConnections.id, + name: toolConnections.name, + transport: toolConnections.transport, + config: toolConnections.config, + }) + .from(toolConnections) + .where( + and( + eq(toolConnections.companyId, input.agent.companyId), + inArray(toolConnections.id, [...permittedConnectionIds]), + ), + ) + : []; + const permittedNotInstalledConnections = permittedConnections + .filter( + (connection) => + (connection.transport === "mcp_remote" || + connection.transport === "local_stdio" || isBrowserUseConnection(connection)) && + !allInstalledConnectionIds.has(connection.id), + ) + .map(({ id, name }) => ({ id, name })) + .sort((a, b) => a.name.localeCompare(b.name)); + const githubBotConnectionIds = await githubBotConnectionIdsForRun(input.db, input.agent.companyId, input.agent.id, input.runId); + const assignedConnections = resolvedInstalledConnections.filter( + (connection) => + permittedConnectionIds.has(connection.id) && + connection.status === "active" && + connection.enabled && + ((Boolean(runIdentity?.activeIdentityContextId) && + (connection.config?.sourceTemplateKey === "github" || + connection.transportConfig?.sourceTemplateKey === "github")) || + connection.credentialPolicy === "per_user" || + !isToolConnectionAttentionHealth(connection.healthStatus)) && + (connection.transport === "mcp_remote" || + connection.transport === "local_stdio" || isBrowserUseConnection(connection) || githubBotConnectionIds.has(connection.id)), + ); + const assignedConnectionIds = new Set( + assignedConnections.map((connection) => connection.id), + ); + const assignedTools = effective.allowedTools.filter((tool) => + assignedConnectionIds.has(tool.connectionId), + ); + const service = createToolGatewayService(input.db); + if (assignedConnections.length === 0) { + await service.recordRuntimeMcpDeliveryDiagnostic({ + companyId: input.agent.companyId, + agentId: input.agent.id, + runId: input.runId, + permittedNotInstalledConnections, + }); + return []; + } + const assignment = { + version: 1, + agentId: input.agent.id, + connections: assignedConnections.map((connection) => connection.id).sort(), + tools: assignedTools.map((tool) => tool.id).sort(), + }; + const assignmentDigest = createHash("sha256") + .update(JSON.stringify(assignment)) + .digest("hex"); + // Native runs may lose access after their immutable context is captured, but + // they must never gain a new or changed assignment during dispatch. + if ( + input.expectedAssignmentDigest !== undefined && + input.expectedAssignmentDigest !== assignmentDigest + ) { + return []; + } + const profileKey = `native:${input.agent.id}:${assignmentDigest}`; + let [profile] = await input.db + .select() + .from(toolProfiles) + .where( + and( + eq(toolProfiles.companyId, input.agent.companyId), + eq(toolProfiles.profileKey, profileKey), + ), + ) + .limit(1); + + if (!profile) { + const fullConnectionIds = new Set( + effective.entries + .filter( + (entry) => + entry.effect === "include" && + entry.selectorType === "connection" && + entry.connectionId, + ) + .map((entry) => entry.connectionId!), + ); + const entries = [ + ...assignedConnections + .filter((connection) => fullConnectionIds.has(connection.id)) + .map((connection) => ({ + selectorType: "connection" as const, + effect: "include" as const, + applicationId: connection.applicationId, + connectionId: connection.id, + })), + ...assignedTools + .filter((tool) => !fullConnectionIds.has(tool.connectionId)) + .map((tool) => ({ + selectorType: "catalog_entry" as const, + effect: "include" as const, + applicationId: tool.applicationId, + connectionId: tool.connectionId, + catalogEntryId: tool.id, + })), + ]; + // The 250-entry limit bounds a public profile-edit request, not the + // effective assignment assembled from existing profiles. Keep every exact + // selector here: truncating or replacing them with connection-wide grants + // would either lose assigned tools or authorize tools outside this snapshot. + try { + const created = await access.createProfile(input.agent.companyId, { + profileKey, + name: `Native ${input.agent.id.slice(0, 8)} ${assignmentDigest.slice(0, 12)}`, + description: "Immutable Paperclip Runner MCP assignment profile.", + status: "active", + defaultAction: "deny", + metadata: { + source: "paperclip_runner", + agentId: input.agent.id, + assignmentDigest, + }, + entries, + }); + [profile] = await input.db + .select() + .from(toolProfiles) + .where(eq(toolProfiles.id, created.id)) + .limit(1); + } catch (error) { + [profile] = await input.db + .select() + .from(toolProfiles) + .where( + and( + eq(toolProfiles.companyId, input.agent.companyId), + eq(toolProfiles.profileKey, profileKey), + ), + ) + .limit(1); + if (!profile) throw error; + } + } + + if (profile!.metadata?.source !== "paperclip_runner" || profile!.metadata?.agentId !== input.agent.id || + profile!.metadata?.assignmentDigest !== assignmentDigest) { + throw new Error("Invalid native runtime profile provenance"); + } + + let [gateway] = ( + await input.db + .select() + .from(toolMcpGateways) + .where( + and( + eq(toolMcpGateways.companyId, input.agent.companyId), + eq(toolMcpGateways.status, "active"), + isNull(toolMcpGateways.archivedAt), + ), + ) + ).filter( + (candidate) => + candidate.metadata?.nativeRuntimeAssignmentDigest === assignmentDigest && + candidate.metadata?.agentId === input.agent.id && + candidate.profileId === profile!.id && + (!candidate.agentId || candidate.agentId === input.agent.id), + ); + // Gateways created before the agent binding existed have a null agentId, and + // named-gateway auth only rejects another agent's run token when it is set. + // The assignment digest includes the agent id, so a reused gateway is this + // agent's own: bind it here instead of waiting for the assignment to change. + if (gateway && !gateway.agentId) { + await input.db + .update(toolMcpGateways) + .set({ agentId: input.agent.id, contextScopeType: "agent", contextScopeId: input.agent.id, updatedAt: new Date() }) + .where( + and( + eq(toolMcpGateways.id, gateway.id), + eq(toolMcpGateways.companyId, input.agent.companyId), + isNull(toolMcpGateways.agentId), + ), + ); + [gateway] = await input.db + .select() + .from(toolMcpGateways) + .where(eq(toolMcpGateways.id, gateway.id)) + .limit(1); + } + if (!gateway) { + const slug = `native-${input.agent.id.replaceAll("-", "").slice(0, 12)}-${assignmentDigest.slice(0, 16)}`; + try { + const created = await service.createNamedGateway({ + companyId: input.agent.companyId, + body: { + name: `Native ${input.agent.name} ${assignmentDigest.slice(0, 8)}`, + slug, + description: "Run-scoped Paperclip Runner MCP gateway.", + profileId: profile!.id, + defaultProfileMode: "gateway_only", + agentId: input.agent.id, + contextScopeType: "agent", + contextScopeId: input.agent.id, + metadata: { + nativeRuntimeAssignmentDigest: assignmentDigest, + agentId: input.agent.id, + }, + }, + actor: { agentId: input.agent.id }, + }); + [gateway] = await input.db + .select() + .from(toolMcpGateways) + .where(eq(toolMcpGateways.id, created.id)) + .limit(1); + } catch (error) { + [gateway] = await input.db + .select() + .from(toolMcpGateways) + .where( + and( + eq(toolMcpGateways.companyId, input.agent.companyId), + eq(toolMcpGateways.slug, slug), + ), + ) + .limit(1); + if (!gateway) throw error; + } + } + + if (gateway!.agentId !== input.agent.id || gateway!.profileId !== profile!.id || + gateway!.metadata?.agentId !== input.agent.id || + gateway!.metadata?.nativeRuntimeAssignmentDigest !== assignmentDigest) { + throw new Error("Invalid native runtime gateway provenance"); + } + + const token = await service.createNamedGatewayToken({ + companyId: input.agent.companyId, + gatewayId: gateway!.id, + body: { + name: `Run ${input.runId.slice(0, 8)}`, + subjectType: "heartbeat_run", + subjectId: input.runId, + clientLabel: `${input.agent.name} heartbeat run`, + ownerNote: `Short-lived runtime MCP token for heartbeat run ${input.runId}.`, + allowedActions: ["tools/list", "tools/call"], + expiresAt: new Date(Date.now() + 60 * 60 * 1_000), + }, + actor: { agentId: input.agent.id }, + }); + + return [ + { + name: ASSIGNED_MCP_SERVER_NAME, + url: `${paperclipApiBaseUrl()}/mcp/gateways/${gateway!.gatewayPublicId}`, + token: token.token, + connectionId: `assignment:${assignmentDigest}`, + }, + ]; +} + +export function createAdapterRuntimeMcpAccess( + servers: AdapterRuntimeMcpServer[], +): AdapterRuntimeMcpAccess | undefined { + if (servers.length === 0) return undefined; + const snapshot = servers.map((server) => Object.freeze({ ...server })); + return Object.freeze({ + getServers: () => snapshot.map((server) => ({ ...server })), + }); +} + +export function createAdapterRuntimeToolAccess(input: { + agentId: string; + companyId: string; + runId: string; + responsibleUserId: string | null; +}): AdapterRuntimeToolAccess | undefined { + if (!input.responsibleUserId) return undefined; + const minted = createRuntimeToolsToken({ + agentId: input.agentId, + companyId: input.companyId, + runId: input.runId, + responsibleUserId: input.responsibleUserId, + }); + if (!minted) return undefined; + // The normal server bootstrap always exports PAPERCLIP_API_URL. Some service + // tests invoke heartbeat execution without booting an HTTP server, however; + // in that context there is no reachable endpoint to advertise and runtime + // tools should simply remain unavailable instead of failing the run. + const baseUrl = configuredPaperclipApiBaseUrl(); + if (!baseUrl) return undefined; + return Object.freeze({ + version: 1, + guidance: CONNECTION_INTENT_AGENT_GUIDANCE, + mcpEndpoint: `${baseUrl}/mcp/runtime-tools`, + rest: { + connectionsSearch: `${baseUrl}/runtime-tools/connections/search`, + connectionRequest: `${baseUrl}/runtime-tools/connections/request`, + }, + bearerToken: minted.token, + expiresAt: minted.expiresAt, + tools: CONNECTION_RUNTIME_TOOL_NAMES, + }); +} + +const MANAGED_MCP_LOCAL_ADAPTERS = new Set(["codex_local"]); + +function adapterSupportsManagedMcpConfig(adapterType: string): boolean { + return MANAGED_MCP_LOCAL_ADAPTERS.has(adapterType); +} + +function gatewayAppliesToRun(input: { + gateway: typeof toolMcpGateways.$inferSelect; + agentId: string; + projectId: string | null; + issueId: string | null; +}): boolean { + const { gateway, agentId, projectId, issueId } = input; + if (gateway.agentId && gateway.agentId !== agentId) return false; + if (gateway.projectId && gateway.projectId !== projectId) return false; + if (gateway.issueId && gateway.issueId !== issueId) return false; + if ( + gateway.contextScopeType === "agent" && + gateway.contextScopeId && + gateway.contextScopeId !== agentId + ) + return false; + if ( + gateway.contextScopeType === "project" && + gateway.contextScopeId && + gateway.contextScopeId !== projectId + ) + return false; + if ( + gateway.contextScopeType === "issue" && + gateway.contextScopeId && + gateway.contextScopeId !== issueId + ) + return false; + return true; +} + +async function gatewayConnectionIds(input: { + db: Db; + companyId: string; + gateway: typeof toolMcpGateways.$inferSelect; +}): Promise> { + const managedRuntimeConnectionId = readNonEmptyString( + input.gateway.metadata?.managedRuntimeConnectionId, + ); + if (managedRuntimeConnectionId) return new Set([managedRuntimeConnectionId]); + + const [profile, entries, catalog, connections] = await Promise.all([ + input.db + .select({ defaultAction: toolProfiles.defaultAction }) + .from(toolProfiles) + .where( + and( + eq(toolProfiles.companyId, input.companyId), + eq(toolProfiles.id, input.gateway.profileId), + ), + ) + .then((rows) => rows[0] ?? null), + input.db + .select() + .from(toolProfileEntries) + .where( + and( + eq(toolProfileEntries.companyId, input.companyId), + eq(toolProfileEntries.profileId, input.gateway.profileId), + ), + ), + input.db + .select({ + id: toolCatalogEntries.id, + connectionId: toolCatalogEntries.connectionId, + applicationId: toolCatalogEntries.applicationId, + toolName: toolCatalogEntries.toolName, + riskLevel: toolCatalogEntries.riskLevel, + }) + .from(toolCatalogEntries) + .where( + and( + eq(toolCatalogEntries.companyId, input.companyId), + eq(toolCatalogEntries.status, "active"), + ), + ), + input.db + .select({ + id: toolConnections.id, + applicationId: toolConnections.applicationId, + }) + .from(toolConnections) + .where(eq(toolConnections.companyId, input.companyId)), + ]); + if (!profile) return new Set(); + if (profile.defaultAction === "allow") + return new Set(catalog.map((entry) => entry.connectionId)); + + const connectionIds = new Set(); + for (const entry of entries) { + if (entry.effect !== "include") continue; + if (entry.connectionId) connectionIds.add(entry.connectionId); + if (entry.applicationId) { + for (const connection of connections) { + if (connection.applicationId === entry.applicationId) + connectionIds.add(connection.id); + } + } + for (const catalogEntry of catalog) { + if ( + (entry.catalogEntryId && entry.catalogEntryId === catalogEntry.id) || + (entry.toolName && entry.toolName === catalogEntry.toolName) || + (entry.riskLevel && entry.riskLevel === catalogEntry.riskLevel) + ) { + connectionIds.add(catalogEntry.connectionId); + } + } + } + return connectionIds; +} + +export async function createManagedMcpRunConfig(input: { + db: Db; + agent: Pick< + typeof agents.$inferSelect, + "id" | "companyId" | "name" | "adapterType" + >; + runId: string; + config: Record; + projectId: string | null; + issueId: string | null; +}): Promise { + if (!adapterSupportsManagedMcpConfig(input.agent.adapterType)) return null; + if (input.config.managedMcpOnly === false) return null; + + const rows = await input.db + .select() + .from(toolMcpGateways) + .where( + and( + eq(toolMcpGateways.companyId, input.agent.companyId), + eq(toolMcpGateways.status, "active"), + isNull(toolMcpGateways.archivedAt), + // A native profile remains an immutable assignment even if gateway + // metadata is cleared. Explicit shared gateways use ordinary profiles. + sql`not exists ( + select 1 from ${toolProfiles} + where ${toolProfiles.id} = ${toolMcpGateways.profileId} + and ${toolProfiles.companyId} = ${toolMcpGateways.companyId} + and (${toolProfiles.profileKey} like 'native:%' + or ${toolProfiles.metadata}->>'source' = 'paperclip_runner') + )`, + ), + ) + .orderBy(asc(toolMcpGateways.name)); + + const installRows = await input.db + .select({ + connectionId: toolConnectionInstalls.connectionId, + enabled: toolConnections.enabled, + status: toolConnections.status, + healthStatus: toolConnections.healthStatus, + config: toolConnections.config, + transportConfig: toolConnections.transportConfig, + }) + .from(toolConnectionInstalls) + .innerJoin( + toolConnections, + and( + eq(toolConnections.id, toolConnectionInstalls.connectionId), + eq(toolConnections.companyId, toolConnectionInstalls.companyId), + ), + ) + .where( + and( + eq(toolConnectionInstalls.companyId, input.agent.companyId), + sql`((${toolConnectionInstalls.targetType} = 'company' and ${toolConnectionInstalls.targetId} = ${input.agent.companyId}) or (${toolConnectionInstalls.targetType} = 'agent' and ${toolConnectionInstalls.targetId} = ${input.agent.id}))`, + ), + ); + const [runIdentity] = await input.db + .select({ + responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.agent.companyId), + eq(heartbeatRuns.agentId, input.agent.id), + ), + ) + .limit(1); + const resolvedAvailableInstalls = runIdentity?.activeIdentityContextId + ? installRows + .filter((install) => install.enabled && install.status === "active") + .map((install) => ({ id: install.connectionId })) + : await filterResolvedGitHubConnectionsForRun({ + db: input.db, + companyId: input.agent.companyId, + agentId: input.agent.id, + responsibleUserId: runIdentity?.responsibleUserId ?? null, + connections: installRows + .filter( + (install) => + install.enabled && + install.status === "active" && + !["degraded", "failed", "error", "missing_secret"].includes( + install.healthStatus, + ), + ) + .map((install) => ({ + id: install.connectionId, + config: install.config, + transportConfig: install.transportConfig, + })), + }); + const availableInstalledConnectionIds = new Set( + resolvedAvailableInstalls.map((install) => install.id), + ); + + const applicableGateways = rows.filter((gateway) => + // The immutable native assignment is delivered separately. Including any + // historical native gateway here duplicates and broadens that assignment. + !Object.hasOwn(gateway.metadata ?? {}, "nativeRuntimeAssignmentDigest") && gatewayAppliesToRun({ + gateway, + agentId: input.agent.id, + projectId: input.projectId, + issueId: input.issueId, + }), + ); + const gateways = ( + await Promise.all( + applicableGateways.map(async (gateway) => ({ + gateway, + connectionIds: await gatewayConnectionIds({ + db: input.db, + companyId: input.agent.companyId, + gateway, + }), + })), + ) + ) + .filter( + ({ connectionIds }) => + connectionIds.size > 0 && + [...connectionIds].every((connectionId) => + availableInstalledConnectionIds.has(connectionId), + ), + ) + .map(({ gateway }) => gateway); + if (gateways.length === 0) return null; + + const service = createToolGatewayService(input.db); + const expiresAt = new Date(Date.now() + 60 * 60 * 1000); + const managedGateways: ManagedMcpGatewayRunConfig["gateways"] = []; + for (const gateway of gateways) { + const token = await service.createNamedGatewayToken({ + companyId: input.agent.companyId, + gatewayId: gateway.id, + body: { + name: `Managed ${input.agent.name} ${input.runId.slice(0, 8)}`, + subjectType: "heartbeat_run", + subjectId: input.runId, + clientLabel: `${input.agent.name} managed local adapter`, + ownerNote: `Short-lived Paperclip-managed MCP token for heartbeat run ${input.runId}.`, + allowedActions: ["tools/list", "tools/call"], + expiresAt, + }, + actor: { agentId: input.agent.id }, + }); + managedGateways.push({ + id: gateway.id, + name: gateway.name, + // This path must bypass the normal /api agent-JWT middleware. The MCP + // gateway performs its own bearer validation for the run-scoped token. + endpointPath: `/mcp/gateways/${gateway.gatewayPublicId}`, + bearerToken: token.token, + tokenPrefix: token.tokenPrefix, + }); + } + + return { + version: 1, + managedMcpOnly: true, + gateways: managedGateways, + }; +} + +export function deriveTaskKey( + contextSnapshot: Record | null | undefined, + payload: Record | null | undefined, +) { + return ( + readNonEmptyString(contextSnapshot?.taskKey) ?? + readNonEmptyString(contextSnapshot?.taskId) ?? + readNonEmptyString(contextSnapshot?.issueId) ?? + readNonEmptyString(payload?.taskKey) ?? + readNonEmptyString(payload?.taskId) ?? + readNonEmptyString(payload?.issueId) ?? + null + ); +} + +function mergeWakeCommentIds(...values: Array): string[] { + const merged: string[] = []; + const append = (value: unknown) => { + const normalized = readNonEmptyString(value); + if (!normalized || merged.includes(normalized)) return; + merged.push(normalized); + }; + + for (const value of values) { + if (Array.isArray(value)) { + for (const entry of value) append(entry); + continue; + } + if (typeof value === "object" && value !== null) { + const candidate = value as Record; + const batched = extractWakeCommentIds(candidate); + if (batched.length > 0) { + for (const entry of batched) append(entry); + continue; + } + append(candidate.wakeCommentId); + append(candidate.commentId); + continue; + } + append(value); + } + + return merged; +} + +const EXTERNAL_ATTACHMENT_OMISSION_REASONS = new Set([ + "attachment_limit", + "storage_unavailable", + "declared_too_large", + "download_unavailable", + "unsupported_type", + "empty_download", + "downloaded_too_large", + "processing_failed", +]); + +type ExternalAttachmentOmission = { + commentId: string; + reasons: Record; +}; + +function readExternalAttachmentOmissions( + value: unknown, +): ExternalAttachmentOmission[] { + if (!Array.isArray(value)) return []; + const byCommentId = new Map(); + for (const candidate of value) { + const record = parseObject(candidate); + const commentId = readNonEmptyString(record.commentId); + if (!commentId) continue; + const reasons = Object.fromEntries( + Object.entries(parseObject(record.reasons)).flatMap(([reason, count]) => + EXTERNAL_ATTACHMENT_OMISSION_REASONS.has(reason) && + typeof count === "number" && + Number.isSafeInteger(count) && + count > 0 + ? [[reason, count]] + : [], + ), + ); + if (Object.keys(reasons).length === 0) continue; + byCommentId.set(commentId, { commentId, reasons }); + } + return [...byCommentId.values()].slice(-50); +} + +function mergeExternalAttachmentOmissions( + existing: Record, + incoming: Record, +) { + return readExternalAttachmentOmissions([ + ...readExternalAttachmentOmissions( + existing[EXTERNAL_ATTACHMENT_OMISSIONS_KEY], + ), + ...readExternalAttachmentOmissions( + incoming[EXTERNAL_ATTACHMENT_OMISSIONS_KEY], + ), + ]); +} + +function externalAttachmentOmissionNotice( + omission: ExternalAttachmentOmission, +) { + const entries = Object.entries(omission.reasons); + const omitted = entries.reduce((total, [, count]) => total + count, 0); + const reasons = entries + .map(([reason, count]) => `${reason.replaceAll("_", " ")}: ${count}`) + .join(", "); + return `Paperclip could not import every attachment from this exact external message: ${omitted} attachment${omitted === 1 ? " was" : "s were"} omitted (${reasons}). Treat omitted attachments as unavailable; do not infer their contents or substitute an older workspace file.`; +} + +export function enrichWakeContextSnapshot(input: { + contextSnapshot: Record; + reason: string | null; + source: WakeupOptions["source"]; + triggerDetail: WakeupOptions["triggerDetail"] | null; + payload: Record | null; +}) { + const { contextSnapshot, reason, source, triggerDetail, payload } = input; + const issueIdFromPayload = + readNonEmptyString(payload?.["issueId"]) ?? + readNonEmptyString(payload?.["taskId"]); + const commentIdFromPayload = readNonEmptyString(payload?.["commentId"]); + const taskKey = deriveTaskKey(contextSnapshot, payload); + const wakeCommentId = deriveCommentId(contextSnapshot, payload); + const wakeCommentIds = mergeWakeCommentIds( + contextSnapshot, + commentIdFromPayload, + ); + + if (!readNonEmptyString(contextSnapshot["wakeReason"]) && reason) { + contextSnapshot.wakeReason = reason; + } + if (!readNonEmptyString(contextSnapshot["issueId"]) && issueIdFromPayload) { + contextSnapshot.issueId = issueIdFromPayload; + } + if (!readNonEmptyString(contextSnapshot["taskId"]) && issueIdFromPayload) { + contextSnapshot.taskId = issueIdFromPayload; + } + if (!readNonEmptyString(contextSnapshot["taskKey"]) && taskKey) { + contextSnapshot.taskKey = taskKey; + } + if ( + !readNonEmptyString(contextSnapshot["commentId"]) && + commentIdFromPayload + ) { + contextSnapshot.commentId = commentIdFromPayload; + } + if (wakeCommentIds.length > 0) { + const latestCommentId = wakeCommentIds[wakeCommentIds.length - 1]; + contextSnapshot[WAKE_COMMENT_IDS_KEY] = wakeCommentIds; + contextSnapshot.commentId = latestCommentId; + contextSnapshot.wakeCommentId = latestCommentId; + // Once comment ids are normalized into the snapshot, rebuild the structured + // wake payload from those ids later instead of carrying forward stale data. + delete contextSnapshot[PAPERCLIP_WAKE_PAYLOAD_KEY]; + } else if ( + !readNonEmptyString(contextSnapshot["wakeCommentId"]) && + wakeCommentId + ) { + contextSnapshot.wakeCommentId = wakeCommentId; + } + if (!readNonEmptyString(contextSnapshot["wakeSource"]) && source) { + contextSnapshot.wakeSource = source; + } + if ( + !readNonEmptyString(contextSnapshot["wakeTriggerDetail"]) && + triggerDetail + ) { + contextSnapshot.wakeTriggerDetail = triggerDetail; + } + normalizeInteractionContinuationWakeContext(contextSnapshot, payload); + + return { + contextSnapshot, + issueIdFromPayload, + commentIdFromPayload, + taskKey, + wakeCommentId, + }; +} + +const INTERACTION_CONTINUATION_CONTEXT_KEYS = [ + "interactionId", + "interactionKind", + "interactionStatus", + "continuationPolicy", + "checkboxSelection", + "itemVerdicts", + "newlyResolvedItemIds", +] as const; + +export function isInteractionResolutionWakePayload( + payload: Record | null | undefined, +) { + return readNonEmptyString(payload?.mutation) === "interaction"; +} + +export function clearInteractionContinuationWakeContext( + contextSnapshot: Record, +) { + for (const key of INTERACTION_CONTINUATION_CONTEXT_KEYS) { + delete contextSnapshot[key]; + } +} + +export function hasInteractionContinuationWakeContext( + contextSnapshot: Record, +) { + return INTERACTION_CONTINUATION_CONTEXT_KEYS.some((key) => + readNonEmptyString(contextSnapshot[key]), + ); +} + +function normalizeInteractionContinuationWakeContext( + contextSnapshot: Record, + payload: Record | null | undefined, +) { + if (isInteractionResolutionWakePayload(payload)) return; + clearInteractionContinuationWakeContext(contextSnapshot); +} + +type AcceptedPlanWakeRoutingDecision = { + otherActiveClaimIssueId: string; + otherActiveClaimIdentifier: string | null; + otherActiveClaimTitle: string; + forceFreshSession: boolean; + suppressAcceptedContinuation: boolean; +}; + +export async function resolveAcceptedPlanWakeRoutingDecision(args: { + db: Db; + companyId: string; + agentId: string; + issueId: string | null; + acceptedPlanContinuationWake: boolean; + contextSnapshot: Record; +}): Promise { + if (args.issueId === null) return null; + if (!args.acceptedPlanContinuationWake) return null; + + const activeClaims = await args.db + .select({ + sourceIssueId: issuePlanDecompositions.sourceIssueId, + identifier: issues.identifier, + title: issues.title, + }) + .from(issuePlanDecompositions) + .innerJoin(issues, eq(issues.id, issuePlanDecompositions.sourceIssueId)) + .where( + and( + eq(issuePlanDecompositions.companyId, args.companyId), + eq(issuePlanDecompositions.ownerAgentId, args.agentId), + eq(issuePlanDecompositions.status, "in_flight"), + ), + ) + .orderBy( + desc(issuePlanDecompositions.updatedAt), + asc(issuePlanDecompositions.createdAt), + ); + + if (activeClaims.length === 0) return null; + if (activeClaims.some((claim) => claim.sourceIssueId === args.issueId)) + return null; + + const otherActiveClaim = activeClaims[0]; + if (!otherActiveClaim) return null; + + const hasAcceptedContinuationWake = + readNonEmptyString(args.contextSnapshot.interactionKind) === + "request_confirmation" && + readNonEmptyString(args.contextSnapshot.interactionStatus) === "accepted"; + + return { + otherActiveClaimIssueId: otherActiveClaim.sourceIssueId, + otherActiveClaimIdentifier: otherActiveClaim.identifier ?? null, + otherActiveClaimTitle: otherActiveClaim.title, + forceFreshSession: true, + suppressAcceptedContinuation: hasAcceptedContinuationWake, + }; +} + +export function mergeCoalescedContextSnapshot( + existingRaw: unknown, + incoming: Record, + options: { preserveExistingInteractionContinuation?: boolean } = {}, +) { + const existing = parseObject(existingRaw); + const existingSource = readNonEmptyString(existing.source); + const incomingSource = readNonEmptyString(incoming.source); + const preservesExternalChatOrigin = + existingSource?.startsWith("chat:") === true && + readNonEmptyString(existing.issueId) !== null && + existing.issueId === incoming.issueId && + incomingSource === "native_status_decision" && + readNonEmptyString(incoming.statusDecisionSource) === + "native_status_decision"; + const merged: Record = { + ...existing, + ...incoming, + }; + // Only executeRun can mint this proof. Coalescence may retain an unchanged + // admitted proof, but must never accept a new marker from an incoming wake. + delete merged[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY]; + delete merged[EXTERNAL_CHAT_QUESTION_RESPONSE_KEY]; + const mergedAttachmentOmissions = mergeExternalAttachmentOmissions( + existing, + incoming, + ); + if (mergedAttachmentOmissions.length > 0) { + merged[EXTERNAL_ATTACHMENT_OMISSIONS_KEY] = mergedAttachmentOmissions; + } else { + delete merged[EXTERNAL_ATTACHMENT_OMISSIONS_KEY]; + } + // A native status wake is control-flow metadata, not a new user-input + // provenance. When it coalesces into the live run, retain the verified chat + // source so the eventual terminal presentation can still prove its route. + // Fresh status-decision runs keep their native_status_decision source. + if (preservesExternalChatOrigin) { + merged.source = existingSource; + } + if ( + existing.forceFreshSession === true || + incoming.forceFreshSession === true + ) { + merged.forceFreshSession = true; + } + if (existing.refreshTools === true || incoming.refreshTools === true) { + merged.refreshTools = true; + } + const mergedCommentIds = mergeWakeCommentIds(existing, incoming); + if (mergedCommentIds.length > 0) { + const latestCommentId = mergedCommentIds[mergedCommentIds.length - 1]; + merged[WAKE_COMMENT_IDS_KEY] = mergedCommentIds; + merged.commentId = latestCommentId; + merged.wakeCommentId = latestCommentId; + // The merged context should carry canonical comment ids; the next wake will + // regenerate any structured payload from those ids. + delete merged[PAPERCLIP_WAKE_PAYLOAD_KEY]; + } + const existingWake = parseObject(existing[PAPERCLIP_WAKE_PAYLOAD_KEY]); + const existingCommentIds = extractWakeCommentIds(existing); + const payloadCommentIds = Array.isArray(existingWake.commentIds) + ? existingWake.commentIds + : []; + const preservesAdmittedWake = + preservesExternalChatOrigin && + parseObject(existingWake.issue).id === existing.issueId && + CHAT_PROVIDERS.some( + (provider) => + existingWake.externalChatProvider === provider && + (existingSource === `chat:${provider}` || + existingSource === `chat:${provider}:recovery`), + ) && + existingCommentIds.length > 0 && + mergedCommentIds.length === existingCommentIds.length && + mergedCommentIds.every((id, index) => id === existingCommentIds[index]) && + payloadCommentIds.length === existingCommentIds.length && + payloadCommentIds.every((id, index) => id === existingCommentIds[index]) && + ((existing[PAPERCLIP_HARNESS_CHECKOUT_KEY] === true && + existingWake.checkedOutByHarness === true) || + (existing[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] === true && + existingWake.externalChatExecutionBound === true)); + if (preservesAdmittedWake) { + merged[PAPERCLIP_WAKE_PAYLOAD_KEY] = existingWake; + merged.wakeReason = existing.wakeReason; + if (existing[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] === true) { + merged[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] = true; + } + } + if ( + !hasInteractionContinuationWakeContext(incoming) && + !( + options.preserveExistingInteractionContinuation === true && + hasInteractionContinuationWakeContext(existing) + ) + ) { + clearInteractionContinuationWakeContext(merged); + } + return merged; +} + +// This is a prompt optimization, not an authorization grant. Verify the durable +// chat binding rather than trusting an arbitrary caller's source/context marker. +export async function resolveExternalChatWakeProvider(input: { + db: Db; + companyId: string; + agentId?: string | null; + runId?: string | null; + issueId: string | null; + contextSnapshot: Record; +}): Promise { + if (input.contextSnapshot.source === "issue.interaction.respond") { + if (!input.runId || !input.agentId || !input.issueId) return null; + const answer = await resolveExternalChatQuestionResponse( + input.db, + { + companyId: input.companyId, + agentId: input.agentId, + issueId: input.issueId, + runId: input.runId, + }, + input.contextSnapshot, + "read", + ); + return answer?.provider ?? null; + } + const source = readNonEmptyString(input.contextSnapshot.source); + const provider = CHAT_PROVIDERS.find( + (candidate) => + source === `chat:${candidate}` || source === `chat:${candidate}:recovery`, + ); + const commentIds = extractWakeCommentIds(input.contextSnapshot); + if ( + !provider || + !input.agentId || + !input.issueId || + commentIds.length === 0 || + (input.contextSnapshot[PAPERCLIP_HARNESS_CHECKOUT_KEY] !== true && + input.contextSnapshot[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] !== + true) + ) { + return null; + } + + const links = await input.db + .select({ + commentId: chatMessageLinks.commentId, + conversationId: chatConversations.id, + }) + .from(chatMessageLinks) + .innerJoin( + chatConversations, + and( + eq(chatConversations.companyId, chatMessageLinks.companyId), + eq(chatConversations.id, chatMessageLinks.conversationId), + eq(chatConversations.endpointId, chatMessageLinks.endpointId), + ), + ) + .innerJoin( + chatEndpoints, + and( + eq(chatEndpoints.companyId, chatConversations.companyId), + eq(chatEndpoints.id, chatConversations.endpointId), + ), + ) + .where( + and( + eq(chatMessageLinks.companyId, input.companyId), + eq(chatMessageLinks.direction, "inbound"), + inArray(chatMessageLinks.commentId, commentIds), + eq(chatConversations.issueId, input.issueId), + inArray(chatConversations.state, ["active", "waiting"]), + eq(chatEndpoints.provider, provider), + eq(chatEndpoints.assignedAgentId, input.agentId), + inArray(chatEndpoints.status, ["active", "verifying"]), + ), + ); + const linkedCommentIds = new Set(links.map((link) => link.commentId)); + return new Set(links.map((link) => link.conversationId)).size === 1 && + commentIds.every((id) => linkedCommentIds.has(id)) + ? provider + : null; +} + +/** Bind an already-claimed chat/reply turn without checking out or approving it. */ +export async function attestReviewedExternalChatRun(input: { + db: Db; + companyId: string; + agentId: string; + issueId: string; + runId: string; + contextSnapshot: Record; + /** Diagnostic-only value from the committed, authorized answer; never a wake field. */ + onQuestionResponseAttested?: (answeredAtMs: number) => void; +}): Promise { + const attempt = () => + input.db.transaction( + async ( + transaction, + ): Promise => { + const tx = transaction as unknown as Db; + const [issue] = await tx + .select() + .from(issues) + .where( + and( + eq(issues.id, input.issueId), + eq(issues.companyId, input.companyId), + ), + ) + .for("update", { noWait: true }) + .limit(1); + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), + ), + ) + .for("update", { noWait: true }) + .limit(1); + const [actor] = await tx + .select({ status: agents.status }) + .from(agents) + .where( + and( + eq(agents.id, input.agentId), + eq(agents.companyId, input.companyId), + ), + ) + .for("update", { noWait: true }) + .limit(1); + if ( + !issue || + !run || + !actor || + !( + issue.status === "in_review" || + (input.contextSnapshot.source === "issue.interaction.respond" && + issue.status === "in_progress") + ) || + issue.assigneeAgentId !== input.agentId || + issue.executionRunId !== input.runId || + run.status !== "running" || + (run.nativeIssueId !== null && run.nativeIssueId !== input.issueId) || + // This attestation precedes execution-start's transition to running. + // A prior run's error projection is invokable, unlike an operator + // pause, termination, or pending approval; all binding checks remain. + DIRECT_NON_INVOKABLE_STATUSES.has(actor.status) + ) + return false; + const admittedContext = parseObject(run.contextSnapshot); + const admittedIds = extractWakeCommentIds(admittedContext); + const suppliedIds = extractWakeCommentIds(input.contextSnapshot); + if ( + admittedContext.issueId !== input.issueId || + input.contextSnapshot.issueId !== input.issueId || + admittedContext.source !== input.contextSnapshot.source || + admittedIds.length === 0 || + admittedIds.length !== suppliedIds.length || + admittedIds.some((id, index) => id !== suppliedIds[index]) + ) + return false; + try { + const answer = + admittedContext.source === "issue.interaction.respond" + ? await resolveExternalChatQuestionResponse( + tx, + input, + admittedContext, + "nonblocking", + true, + ) + : null; + if (admittedContext.source === "issue.interaction.respond" && !answer) + return false; + // The marker is built here only after proving the real execution owner. + // The shared boundary then verifies current provider/resource/principal + // access for every admitted message; no lifecycle state is mutated. + await authorizeChatConversationForBoundRun( + tx, + input, + { + ...(answer?.authorizationContext ?? admittedContext), + [PAPERCLIP_HARNESS_CHECKOUT_KEY]: false, + [PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY]: true, + }, + "nonblocking", + ); + if (answer) + input.contextSnapshot[EXTERNAL_CHAT_QUESTION_RESPONSE_KEY] = + answer.marker; + return answer ? { answeredAtMs: answer.answeredAtMs } : true; + } catch (error) { + if ( + error instanceof Error && + error.message === "paperclip_runner_chat_attachment_binding_denied" + ) { + // Inbound processing commits the message/link before dispatching its + // wake, but completes subscription and marks delivery processed after + // dispatch. Wait for that exact committed batch, never admit it early. + const links = await tx + .select({ + commentId: chatMessageLinks.commentId, + state: chatDeliveries.state, + }) + .from(chatMessageLinks) + .innerJoin( + chatDeliveries, + and( + eq(chatDeliveries.id, chatMessageLinks.deliveryId), + eq(chatDeliveries.companyId, chatMessageLinks.companyId), + eq(chatDeliveries.endpointId, chatMessageLinks.endpointId), + eq( + chatDeliveries.conversationId, + chatMessageLinks.conversationId, + ), + ), + ) + .where( + and( + eq(chatMessageLinks.companyId, input.companyId), + eq(chatMessageLinks.direction, "inbound"), + inArray(chatMessageLinks.commentId, admittedIds), + ), + ); + if ( + admittedIds.every((id) => + links.some((link) => link.commentId === id), + ) && + links.some((link) => link.state === "processing") && + links.every( + (link) => + link.state === "processing" || link.state === "processed", + ) + ) { + return "pending_delivery"; + } + } + if ( + error instanceof Error && + [ + "paperclip_runner_chat_attachment_binding_denied", + "paperclip_runner_chat_attachment_destination_denied", + "paperclip_runner_chat_attachment_principal_denied", + ].includes(error.message) + ) + return false; + throw error; + } + }, + ); + // Release every lock before retrying and re-prove the current execution + // owner, admitted batch, and policy. Routine control-plane contention is not + // evidence that the user lost access; neither is it permission to bypass it. + for (let attemptNumber = 0; attemptNumber < 51; attemptNumber += 1) { + try { + const result = await attempt(); + if (typeof result === "object") { + // The transaction (including COMMIT) must succeed before any attempt- + // local timing is accepted. A retry/rollback cannot publish this value. + input.onQuestionResponseAttested?.(result.answeredAtMs); + return true; + } + if (result !== "pending_delivery") return result; + } catch (error) { + if (!isExternalChatWaitAuthorizationContention(error)) throw error; + } + if (attemptNumber < 50) + await new Promise((resolve) => setTimeout(resolve, 100)); + } + throw new Error("reviewed_chat_execution_binding_not_ready"); +} + +export async function buildPaperclipWakePayload(input: { + db: Db; + companyId: string; + agentId?: string | null; + runId?: string | null; + contextSnapshot: Record; + continuationSummary?: { + key: string; + title: string | null; + body: string; + sourceTrust?: SourceTrustMetadata | null; + updatedAt: Date; + } | null; + issueSummary?: { + id: string; + identifier: string | null; + title: string; + description: string | null; + status: string; + priority: string; + workMode: string; + projectId?: string | null; + executionPolicy?: unknown; + } | null; + exposeLowTrustRaw?: boolean; + // Experimental: agents write user-interaction content in ASD-STE100 + // Simplified Technical English (rendered as a prompt directive downstream). + simplifiedEnglishInteractions?: boolean; +}) { + const executionStage = parseObject(input.contextSnapshot.executionStage); + const commentIds = extractWakeCommentIds(input.contextSnapshot); + const annotationCommentId = readNonEmptyString( + input.contextSnapshot.annotationCommentId, + ); + const issueId = readNonEmptyString(input.contextSnapshot.issueId); + const conversationMode = input.contextSnapshot.conversationMode === true; + const continuationSummary = conversationMode ? null : input.continuationSummary ?? null; + const agentMessage = parseObject( + input.contextSnapshot[PAPERCLIP_AGENT_MESSAGE_KEY], + ); + const agentMessageText = sanitizeAgentSessionMessageText(agentMessage.text); + const issueSummary = + input.issueSummary ?? + (issueId + ? await input.db + .select({ + id: issues.id, + identifier: issues.identifier, + title: issues.title, + description: issues.description, + status: issues.status, + priority: issues.priority, + workMode: issues.workMode, + }) + .from(issues) + .where( + and(eq(issues.id, issueId), eq(issues.companyId, input.companyId)), + ) + .then((rows) => rows[0] ?? null) + : null); + if ( + commentIds.length === 0 && + Object.keys(executionStage).length === 0 && + !issueSummary && + !agentMessageText + ) + return null; + + const commentRows = + commentIds.length === 0 + ? [] + : await input.db + .select({ + id: issueComments.id, + issueId: issueComments.issueId, + body: issueComments.body, + authorType: issueComments.authorType, + authorAgentId: issueComments.authorAgentId, + authorUserId: issueComments.authorUserId, + presentation: issueComments.presentation, + metadata: issueComments.metadata, + deletedAt: issueComments.deletedAt, + deletedByType: issueComments.deletedByType, + deletedByAgentId: issueComments.deletedByAgentId, + deletedByUserId: issueComments.deletedByUserId, + deletedByRunId: issueComments.deletedByRunId, + sourceTrust: issueComments.sourceTrust, + createdAt: issueComments.createdAt, + }) + .from(issueComments) + .where( + and( + eq(issueComments.companyId, input.companyId), + issueId ? eq(issueComments.issueId, issueId) : undefined, + inArray(issueComments.id, commentIds), + ), + ); + + const commentsById = new Map( + commentRows.map((comment) => [comment.id, comment]), + ); + const issueDescription = conversationMode ? null : issueSummary?.description ?? null; + const issueDescriptionTruncated = + issueDescription !== null && + issueDescription.length > MAX_INLINE_WAKE_ISSUE_DESCRIPTION_CHARS; + const inlineIssueDescription = issueDescriptionTruncated + ? issueDescription.slice(0, MAX_INLINE_WAKE_ISSUE_DESCRIPTION_CHARS) + : issueDescription; + const comments: Array> = []; + let remainingBodyChars = MAX_INLINE_WAKE_COMMENT_BODY_TOTAL_CHARS; + let truncated = false; + let missingCommentCount = 0; + const safeContinuationSummary = + continuationSummary && !input.exposeLowTrustRaw + ? redactQuarantinedBodyForHigherTrust(continuationSummary) + : continuationSummary; + + for (const commentId of commentIds) { + const row = commentsById.get(commentId); + if (!row) { + truncated = true; + missingCommentCount += 1; + continue; + } + if (comments.length >= MAX_INLINE_WAKE_COMMENTS) { + truncated = true; + break; + } + + const deletedAt = row.deletedAt ?? null; + const safeRow = + deletedAt || input.exposeLowTrustRaw + ? row + : sanitizeQuarantinedCommentForHigherTrust(row); + const fullBody = deletedAt ? "" : safeRow.body; + const allowedBodyChars = Math.min( + MAX_INLINE_WAKE_COMMENT_BODY_CHARS, + remainingBodyChars, + ); + if (allowedBodyChars <= 0) { + truncated = true; + break; + } + + const body = + fullBody.length > allowedBodyChars + ? fullBody.slice(0, allowedBodyChars) + : fullBody; + const bodyTruncated = body.length < fullBody.length; + if (bodyTruncated) truncated = true; + remainingBodyChars -= body.length; + + comments.push({ + id: row.id, + issueId: row.issueId, + authorType: + row.authorType ?? + (row.authorAgentId ? "agent" : row.authorUserId ? "user" : "system"), + body, + bodyTruncated, + presentation: deletedAt ? null : (safeRow.presentation ?? null), + metadata: deletedAt ? null : (safeRow.metadata ?? null), + deletedAt: deletedAt ? deletedAt.toISOString() : null, + deletedByType: deletedAt ? (row.deletedByType ?? null) : null, + deletedByAgentId: deletedAt ? (row.deletedByAgentId ?? null) : null, + deletedByUserId: deletedAt ? (row.deletedByUserId ?? null) : null, + deletedByRunId: deletedAt ? (row.deletedByRunId ?? null) : null, + sourceTrust: row.sourceTrust ?? null, + createdAt: row.createdAt.toISOString(), + author: row.authorAgentId + ? { type: "agent", id: row.authorAgentId } + : row.authorUserId + ? { type: "user", id: row.authorUserId } + : { type: "system", id: null }, + }); + } + + const attachmentCommentIds = comments.flatMap((comment) => + typeof comment.id === "string" && + comment.deletedAt === null && + (input.exposeLowTrustRaw || + !isLowTrustQuarantined(comment.sourceTrust as SourceTrustMetadata | null)) + ? [comment.id] + : [], + ); + const attachmentRows = + !issueId || attachmentCommentIds.length === 0 + ? [] + : await input.db + .select({ + id: issueAttachments.id, + issueCommentId: issueAttachments.issueCommentId, + filename: assets.originalFilename, + contentType: assets.contentType, + byteSize: assets.byteSize, + }) + .from(issueAttachments) + .innerJoin( + assets, + and( + eq(issueAttachments.assetId, assets.id), + eq(assets.companyId, input.companyId), + ), + ) + .where( + and( + eq(issueAttachments.companyId, input.companyId), + eq(issueAttachments.issueId, issueId), + inArray(issueAttachments.issueCommentId, attachmentCommentIds), + ), + ) + .orderBy(asc(issueAttachments.createdAt), asc(issueAttachments.id)) + .limit(MAX_INLINE_WAKE_ATTACHMENTS + 1); + if (attachmentRows.length > MAX_INLINE_WAKE_ATTACHMENTS) truncated = true; + const attachmentsByCommentId = new Map< + string, + Array<{ + id: string; + filename: string; + contentType: string; + byteSize: number; + contentPath: string; + }> + >(); + for (const attachment of attachmentRows.slice( + 0, + MAX_INLINE_WAKE_ATTACHMENTS, + )) { + if (!attachment.issueCommentId) continue; + const descriptors = + attachmentsByCommentId.get(attachment.issueCommentId) ?? []; + descriptors.push({ + id: attachment.id, + filename: attachment.filename?.trim() || "attachment", + contentType: attachment.contentType, + byteSize: attachment.byteSize, + contentPath: `/api/attachments/${attachment.id}/content`, + }); + attachmentsByCommentId.set(attachment.issueCommentId, descriptors); + } + for (const comment of comments) { + if (typeof comment.id !== "string") continue; + const attachments = attachmentsByCommentId.get(comment.id); + if (attachments?.length) comment.attachments = attachments; + } + + const annotationDeltas = + annotationCommentId && issueId + ? await input.db + .select({ + id: documentAnnotationComments.id, + issueId: documentAnnotationComments.issueId, + threadId: documentAnnotationComments.threadId, + body: documentAnnotationComments.body, + authorType: documentAnnotationComments.authorType, + authorAgentId: documentAnnotationComments.authorAgentId, + authorUserId: documentAnnotationComments.authorUserId, + createdAt: documentAnnotationComments.createdAt, + documentKey: documentAnnotationThreads.documentKey, + status: documentAnnotationThreads.status, + anchorState: documentAnnotationThreads.anchorState, + anchorConfidence: documentAnnotationThreads.anchorConfidence, + currentRevisionNumber: + documentAnnotationThreads.currentRevisionNumber, + selectedText: documentAnnotationThreads.selectedText, + prefixText: documentAnnotationThreads.prefixText, + suffixText: documentAnnotationThreads.suffixText, + }) + .from(documentAnnotationComments) + .innerJoin( + documentAnnotationThreads, + eq( + documentAnnotationComments.threadId, + documentAnnotationThreads.id, + ), + ) + .where( + and( + eq(documentAnnotationComments.companyId, input.companyId), + eq(documentAnnotationComments.issueId, issueId), + eq(documentAnnotationComments.id, annotationCommentId), + eq(documentAnnotationThreads.companyId, input.companyId), + eq(documentAnnotationThreads.issueId, issueId), + ), + ) + .then((rows) => + rows.map((row) => ({ + id: row.id, + issueId: row.issueId, + threadId: row.threadId, + documentKey: row.documentKey, + revisionNumber: row.currentRevisionNumber, + quote: row.selectedText, + prefix: row.prefixText, + suffix: row.suffixText, + threadStatus: row.status, + anchorState: row.anchorState, + anchorConfidence: row.anchorConfidence, + body: + row.body.length > MAX_INLINE_WAKE_COMMENT_BODY_CHARS + ? row.body.slice(0, MAX_INLINE_WAKE_COMMENT_BODY_CHARS) + : row.body, + bodyTruncated: + row.body.length > MAX_INLINE_WAKE_COMMENT_BODY_CHARS, + createdAt: row.createdAt.toISOString(), + author: row.authorAgentId + ? { type: "agent", id: row.authorAgentId } + : row.authorUserId + ? { type: "user", id: row.authorUserId } + : { type: row.authorType, id: null }, + })), + ) + : []; + const interactionId = readNonEmptyString(input.contextSnapshot.interactionId); + const interactionKind = readNonEmptyString( + input.contextSnapshot.interactionKind, + ); + const interactionStatus = readNonEmptyString( + input.contextSnapshot.interactionStatus, + ); + const interactionContinuationSource = readNonEmptyString( + input.contextSnapshot.source, + ); + const externalInteractionContinuation = + (interactionStatus === "answered" || interactionStatus === "accepted") && + (input.contextSnapshot.externalChatContinuation === true || + interactionContinuationSource === "external_chat.interaction.resolve"); + const checkboxSelection = parseObject( + input.contextSnapshot.checkboxSelection, + ); + // A resolved plan review is new user input, including in chat. Ordinary chat + // wakes must still exclude historical plan context across /new boundaries. + const resolvedPlanInteraction = interactionId && interactionKind === "request_confirmation" && + (interactionStatus === "accepted" || interactionStatus === "rejected"); + const planReviewContext = issueId && (!conversationMode || resolvedPlanInteraction) + ? await buildPlanReviewContext({ + db: input.db, + companyId: input.companyId, + issueId, + issueWorkMode: conversationMode ? null : issueSummary?.workMode ?? null, + includeForIssueComment: !conversationMode && commentIds.length > 0, + includeForAnnotationDelta: !conversationMode && annotationDeltas.length > 0, + interactionId, + }) + : null; + const documentReviewContext = issueId && !conversationMode + ? await buildDocumentReviewContext({ + db: input.db, + companyId: input.companyId, + issueId, + includeForIssueComment: commentIds.length > 0, + includeForAnnotationDelta: annotationDeltas.length > 0, + }) + : null; + const payloadTruncated = + truncated || + issueDescriptionTruncated || + planReviewContext?.truncated === true || + documentReviewContext?.truncated === true; + const recoveryActionId = readNonEmptyString( + input.contextSnapshot.recoveryActionId, + ); + const recoveryCause = readNonEmptyString(input.contextSnapshot.recoveryCause); + const recoveryAction = recoveryActionId + ? await input.db + .select() + .from(issueRecoveryActions) + .where( + and( + eq(issueRecoveryActions.id, recoveryActionId), + eq(issueRecoveryActions.companyId, input.companyId), + ), + ) + .then((rows) => rows[0] ?? null) + : null; + const recoveryEvidence = parseObject(recoveryAction?.evidence); + // A restored task resumes its deliverable, not the completed repair. A + // referenced settled/missing action must not fall back to a stale cause. + const recoveryStillActive = recoveryAction + ? ["active", "escalated"].includes(recoveryAction.status) + : !recoveryActionId && Boolean(recoveryCause); + const originalAssigneeId = + recoveryAction?.returnOwnerAgentId ?? + recoveryAction?.previousOwnerAgentId ?? + null; + const originalAssignee = originalAssigneeId + ? await input.db + .select({ id: agents.id, name: agents.name }) + .from(agents) + .where( + and( + eq(agents.id, originalAssigneeId), + eq(agents.companyId, input.companyId), + ), + ) + .then((rows) => rows[0] ?? null) + : null; + + const externalChatProvider = await resolveExternalChatWakeProvider({ + db: input.db, + companyId: input.companyId, + agentId: input.agentId, + runId: input.runId, + issueId: issueSummary?.id === issueId ? issueId : null, + contextSnapshot: input.contextSnapshot, + }); + const attachmentOmissions = externalChatProvider + ? readExternalAttachmentOmissions( + input.contextSnapshot[EXTERNAL_ATTACHMENT_OMISSIONS_KEY], + ) + .filter((omission) => commentIds.includes(omission.commentId)) + .map((omission) => ({ + commentId: omission.commentId, + reasons: omission.reasons, + notice: externalAttachmentOmissionNotice(omission), + })) + : []; + const payload = { + reason: readNonEmptyString(input.contextSnapshot.wakeReason), + executionContinuation: input.contextSnapshot.executionContinuation ?? null, + chatCompletionUpdates: input.contextSnapshot.chatCompletionUpdates ?? null, + attachmentOmissions, + externalChatProvider, + recovery: + recoveryStillActive + ? { + cause: recoveryAction?.cause ?? recoveryCause, + failureSummary: readNonEmptyString(recoveryEvidence.failureSummary), + originalAssignee: originalAssignee + ? { id: originalAssignee.id, name: originalAssignee.name } + : originalAssigneeId + ? { id: originalAssigneeId, name: null } + : null, + attemptCount: recoveryAction?.attemptCount ?? null, + maxAttempts: recoveryAction?.maxAttempts ?? null, + nextAction: recoveryAction?.nextAction ?? null, + routingFallbackReason: readNonEmptyString( + recoveryEvidence.routingFallbackReason, + ), + } + : null, + issue: issueSummary + ? { + id: issueSummary.id, + identifier: issueSummary.identifier, + title: issueSummary.title, + description: inlineIssueDescription, + descriptionTruncated: issueDescriptionTruncated, + status: issueSummary.status, + priority: issueSummary.priority, + workMode: issueSummary.workMode, + } + : null, + agentMessage: agentMessageText + ? { + text: agentMessageText, + source: readNonEmptyString(agentMessage.source), + pluginKey: readNonEmptyString(agentMessage.pluginKey), + sessionId: readNonEmptyString(agentMessage.sessionId), + ...(Array.isArray(agentMessage.untrustedToolResults) + ? { + untrustedToolResults: agentMessage.untrustedToolResults + .slice(0, 8) + .map((value) => { + const result = parseObject(value); + return { + actionRequestId: + sanitizeAgentSessionMessageText( + result.actionRequestId, + ) ?? "", + toolName: + sanitizeAgentSessionMessageText(result.toolName) ?? "", + resultSummary: + sanitizeAgentSessionMessageText(result.resultSummary) ?? + "", + error: sanitizeAgentSessionMessageText(result.error), + declineReason: sanitizeAgentSessionMessageText( + result.declineReason, + ), + }; + }), + } + : {}), + } + : null, + childIssueSummaries: Array.isArray( + input.contextSnapshot.childIssueSummaries, + ) + ? input.contextSnapshot.childIssueSummaries + : [], + childIssueSummaryTruncated: + input.contextSnapshot.childIssueSummaryTruncated === true, + dispositionRepair: input.contextSnapshot.legacyDispositionEpisode ? { + attempt: input.contextSnapshot.dispositionRepairAttempt, + maxAttempts: input.contextSnapshot.dispositionRepairMaxAttempts, + sourceRunId: input.contextSnapshot.retryOfRunId, + instruction: input.contextSnapshot.dispositionRepairInstruction, + } : null, + livenessContinuation: + readNonEmptyString(input.contextSnapshot.livenessContinuationState) || + readNonEmptyString( + input.contextSnapshot.livenessContinuationInstruction, + ) || + readNonEmptyString( + input.contextSnapshot.livenessContinuationSourceRunId, + ) || + typeof input.contextSnapshot.livenessContinuationAttempt === "number" + ? { + attempt: input.contextSnapshot.livenessContinuationAttempt, + maxAttempts: input.contextSnapshot.livenessContinuationMaxAttempts, + sourceRunId: readNonEmptyString( + input.contextSnapshot.livenessContinuationSourceRunId, + ), + state: readNonEmptyString( + input.contextSnapshot.livenessContinuationState, + ), + reason: readNonEmptyString( + input.contextSnapshot.livenessContinuationReason, + ), + instruction: readNonEmptyString( + input.contextSnapshot.livenessContinuationInstruction, + ), + } + : null, + interactionKind, + interactionStatus, + interactionId, + sourceRunId: readNonEmptyString(input.contextSnapshot.sourceRunId), + externalChatQuestionResponse: externalChatProvider + ? (input.contextSnapshot[EXTERNAL_CHAT_QUESTION_RESPONSE_KEY] ?? null) + : null, + externalInteractionContinuation, + checkboxSelection: + Object.keys(checkboxSelection).length > 0 ? checkboxSelection : null, + checkedOutByHarness: + input.contextSnapshot[PAPERCLIP_HARNESS_CHECKOUT_KEY] === true, + externalChatExecutionBound: + input.contextSnapshot[PAPERCLIP_EXTERNAL_CHAT_EXECUTION_BOUND_KEY] === + true, + simplifiedEnglishInteractions: input.simplifiedEnglishInteractions === true, + dependencyBlockedInteraction: + input.contextSnapshot.dependencyBlockedInteraction === true, + treeHoldInteraction: input.contextSnapshot.treeHoldInteraction === true, + activeTreeHold: parseObject(input.contextSnapshot.activeTreeHold), + unresolvedBlockerIssueIds: Array.isArray( + input.contextSnapshot.unresolvedBlockerIssueIds, + ) + ? input.contextSnapshot.unresolvedBlockerIssueIds.filter( + (value): value is string => + typeof value === "string" && value.length > 0, + ) + : [], + unresolvedBlockerSummaries: Array.isArray( + input.contextSnapshot.unresolvedBlockerSummaries, + ) + ? input.contextSnapshot.unresolvedBlockerSummaries + : [], + executionStage: + Object.keys(executionStage).length > 0 ? executionStage : null, + taskWatchdog: (input.contextSnapshot.taskWatchdog ?? null) as unknown, + skillTest: (input.contextSnapshot.paperclipSkillTest ?? null) as unknown, + continuationSummary: safeContinuationSummary + ? { + key: safeContinuationSummary.key, + title: safeContinuationSummary.title, + body: + safeContinuationSummary.body.length > 4_000 + ? safeContinuationSummary.body.slice(0, 4_000) + : safeContinuationSummary.body, + bodyTruncated: safeContinuationSummary.body.length > 4_000, + sourceTrust: safeContinuationSummary.sourceTrust ?? null, + updatedAt: safeContinuationSummary.updatedAt.toISOString(), + } + : null, + commentIds, + latestCommentId: commentIds[commentIds.length - 1] ?? null, + comments, + annotationDeltas, + planReviewContext, + documentReviewContext, + commentWindow: { + requestedCount: commentIds.length, + includedCount: comments.length, + missingCount: missingCommentCount, + }, + truncated: payloadTruncated, + fallbackFetchNeeded: payloadTruncated || missingCommentCount > 0, + }; + return issueId + ? createRunSecretRedactionRegistry(input.db).redactForIssue( + input.companyId, + issueId, + payload, + ) + : payload; +} + +function readNonEmptyString(value: unknown): string | null { + return typeof value === "string" && value.trim().length > 0 ? value : null; +} + +/** Issue context and responsible-user resolution bound to one service database. */ +export function createHeartbeatRunPreparation(db: Db) { + function toAgentOrgRow( + agent: Pick< + typeof agents.$inferSelect, + "id" | "companyId" | "name" | "reportsTo" | "status" + >, + ): AgentOrgRow { + return { + id: agent.id, + companyId: agent.companyId, + name: agent.name, + reportsTo: agent.reportsTo, + status: agent.status, + }; + } + + async function listCompanyAgentOrgRows( + companyId: string, + ): Promise { + return db + .select({ + id: agents.id, + companyId: agents.companyId, + name: agents.name, + reportsTo: agents.reportsTo, + status: agents.status, + }) + .from(agents) + .where(eq(agents.companyId, companyId)); + } + + function groupAgentOrgRowsByCompany(agentRows: AgentOrgRow[]) { + const byCompany = new Map(); + for (const agent of agentRows) { + const companyAgents = byCompany.get(agent.companyId); + if (companyAgents) { + companyAgents.push(agent); + } else { + byCompany.set(agent.companyId, [agent]); + } + } + return byCompany; + } + + async function getIssueExecutionContext(companyId: string, issueId: string) { + return db + .select({ + chatCommunicationGuidance: chatConversations.communicationGuidance, + chatAssignedAgentId: chatEndpoints.assignedAgentId, + // Select only the public command, never the rest of setup state. + chatSlackCommand: sql`case when ${chatEndpoints.status} in ('active', 'verifying') then ${chatEndpoints.setup}->>'command' end`, + externalConversationState: externalConversationStateSql(), + conversationAgentId: issues.conversationAgentId, + conversationUserId: issues.conversationUserId, + conversationState: issues.conversationState, + conversationSessionGeneration: issues.conversationSessionGeneration, + conversationBoundaryCommentId: issues.conversationBoundaryCommentId, + id: issues.id, + identifier: issues.identifier, + title: issues.title, + titleNeedsGeneration: issues.titleNeedsGeneration, + description: issues.description, + status: issues.status, + workMode: issues.workMode, + reviewPolicy: issues.reviewPolicy, + priority: issues.priority, + projectId: issues.projectId, + projectWorkspaceId: issues.projectWorkspaceId, + executionWorkspaceId: issues.executionWorkspaceId, + executionWorkspacePreference: issues.executionWorkspacePreference, + assigneeAgentId: issues.assigneeAgentId, + assigneeAdapterOverrides: issues.assigneeAdapterOverrides, + executionPolicy: issues.executionPolicy, + executionState: issues.executionState, + executionWorkspaceSettings: issues.executionWorkspaceSettings, + parentId: issues.parentId, + createdByUserId: issues.createdByUserId, + responsibleUserId: issues.responsibleUserId, + originKind: issues.originKind, + originId: issues.originId, + originRunId: issues.originRunId, + originIdentityContextId: issues.originIdentityContextId, + continuationIdentityContextId: issues.continuationIdentityContextId, + updatedAt: issues.updatedAt, + }) + .from(issues) + .leftJoin(chatConversations, and( + eq(chatConversations.companyId, issues.companyId), + eq(chatConversations.issueId, issues.id), + )) + .leftJoin(chatEndpoints, and( + eq(chatEndpoints.companyId, chatConversations.companyId), + eq(chatEndpoints.id, chatConversations.endpointId), + eq(chatEndpoints.provider, "slack"), + )) + .where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))) + .then((rows) => rows[0] ?? null); + } + + async function getPinnedSkillTestContext(companyId: string, issueId: string) { + const row = await db + .select({ + testRunId: companySkillTestRuns.id, + skillId: companySkillTestRuns.skillId, + inputId: companySkillTestRuns.inputId, + skillVersionId: companySkillTestRuns.skillVersionId, + outputDocumentKey: companySkillTestRuns.outputDocumentKey, + fileInventory: companySkillVersions.fileInventory, + revisionNumber: companySkillVersions.revisionNumber, + label: companySkillVersions.label, + }) + .from(companySkillTestRuns) + .innerJoin( + companySkillVersions, + and( + eq(companySkillVersions.id, companySkillTestRuns.skillVersionId), + eq(companySkillVersions.companyId, companySkillTestRuns.companyId), + ), + ) + .where( + and( + eq(companySkillTestRuns.companyId, companyId), + eq(companySkillTestRuns.issueId, issueId), + ), + ) + .then((rows) => rows[0] ?? null); + if (!row) return null; + const fileInventory = Array.isArray(row.fileInventory) + ? row.fileInventory.flatMap((entry) => { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) + return []; + const record = entry as unknown as Record; + const path = typeof record.path === "string" ? record.path : ""; + if (!path) return []; + return [ + { + path, + kind: typeof record.kind === "string" ? record.kind : "other", + content: typeof record.content === "string" ? record.content : "", + }, + ]; + }) + : []; + return { + testRunId: row.testRunId, + skillId: row.skillId, + inputId: row.inputId ?? null, + skillVersionId: row.skillVersionId, + revisionNumber: row.revisionNumber, + label: row.label ?? null, + outputDocumentKey: row.outputDocumentKey, + fileInventory, + }; + } + + async function getRoutineEnvForExecutionIssue( + companyId: string, + issueContext: { originKind: string | null; originId: string | null; originRunId: string | null } | null, + ) { + if ( + !issueContext || + issueContext.originKind !== "routine_execution" || + !issueContext.originId + ) { + return { routineId: null, env: null, responsibleUserId: null }; + } + + const routineRun = issueContext.originRunId + ? await db + .select({ + routineRevisionId: routineRuns.routineRevisionId, + responsibleUserId: routineRuns.responsibleUserId, + }) + .from(routineRuns) + .where( + and( + eq(routineRuns.id, issueContext.originRunId), + eq(routineRuns.companyId, companyId), + eq(routineRuns.routineId, issueContext.originId), + ), + ) + .then((rows) => rows[0] ?? null) + : null; + + if (routineRun?.routineRevisionId) { + const revision = await db + .select({ + snapshot: routineRevisions.snapshot, + responsibleUserId: routineRevisions.responsibleUserId, + }) + .from(routineRevisions) + .where( + and( + eq(routineRevisions.id, routineRun.routineRevisionId), + eq(routineRevisions.companyId, companyId), + eq(routineRevisions.routineId, issueContext.originId), + ), + ) + .then((rows) => rows[0] ?? null); + const snapshot = revision?.snapshot as + RoutineRevisionSnapshotV1 | undefined; + if (snapshot?.version === 1) { + return { + routineId: issueContext.originId, + env: snapshot.routine.env ?? null, + responsibleUserId: + routineRun?.responsibleUserId ?? + revision?.responsibleUserId ?? + snapshot.routine.responsibleUserId ?? + null, + }; + } + } + + const routine = await db + .select({ + env: routines.env, + responsibleUserId: routines.responsibleUserId, + }) + .from(routines) + .where( + and( + eq(routines.id, issueContext.originId), + eq(routines.companyId, companyId), + ), + ) + .then((rows) => rows[0] ?? null); + return { + routineId: issueContext.originId, + env: routine?.env ?? null, + responsibleUserId: + routineRun?.responsibleUserId ?? routine?.responsibleUserId ?? null, + }; + } + + async function resolveCompanyDefaultResponsibleUserId(companyId: string) { + const company = await db + .select({ defaultResponsibleUserId: companies.defaultResponsibleUserId }) + .from(companies) + .where(eq(companies.id, companyId)) + .then((rows) => rows[0] ?? null); + const explicitDefault = readNonEmptyString( + company?.defaultResponsibleUserId, + ); + if (explicitDefault) return explicitDefault; + + const owner = await db + .select({ userId: companyMemberships.principalId }) + .from(companyMemberships) + .where( + and( + eq(companyMemberships.companyId, companyId), + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.status, "active"), + eq(companyMemberships.membershipRole, "owner"), + ), + ) + .orderBy(asc(companyMemberships.createdAt), asc(companyMemberships.id)) + .limit(1) + .then((rows) => rows[0] ?? null); + if (owner?.userId) return owner.userId; + + const firstUser = await db + .select({ userId: companyMemberships.principalId }) + .from(companyMemberships) + .where( + and( + eq(companyMemberships.companyId, companyId), + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.status, "active"), + ), + ) + .orderBy(asc(companyMemberships.createdAt), asc(companyMemberships.id)) + .limit(1) + .then((rows) => rows[0] ?? null); + return firstUser?.userId ?? null; + } + + async function resolveParentIssueResponsibleUserId( + companyId: string, + parentId: string | null | undefined, + ) { + if (!parentId) return null; + const parent = await db + .select({ + responsibleUserId: issues.responsibleUserId, + createdByUserId: issues.createdByUserId, + }) + .from(issues) + .where(and(eq(issues.companyId, companyId), eq(issues.id, parentId))) + .then((rows) => rows[0] ?? null); + return parent?.responsibleUserId ?? null; + } + + function isManualUserRun(input: { + contextSnapshot: Record; + requestedByActorType?: "user" | "agent" | "system" | null; + source?: WakeupOptions["source"] | null; + triggerDetail?: WakeupOptions["triggerDetail"] | null; + }) { + if (input.requestedByActorType !== "user") return false; + const wakeReason = readNonEmptyString(input.contextSnapshot.wakeReason); + if (wakeReason && ISSUE_RESPONSIBLE_USER_WAKE_REASONS.has(wakeReason)) + return false; + return input.source === "on_demand" || input.triggerDetail === "manual"; + } + + async function resolveResponsibleUserIdForRunSeed(input: { + companyId: string; + contextSnapshot: Record; + issueContext: { id: string; responsibleUserId: string | null; parentId: string | null } | null; + routineEnvContext: Awaited< + ReturnType + >; + requestedByActorType?: "user" | "agent" | "system" | null; + requestedByActorId?: string | null; + source?: WakeupOptions["source"] | null; + triggerDetail?: WakeupOptions["triggerDetail"] | null; + existingRunResponsibleUserId?: string | null; + }) { + const contextResponsibleUserId = readNonEmptyString( + input.contextSnapshot.responsibleUserId, + ); + const requestedUserId = + input.requestedByActorType === "user" + ? readNonEmptyString(input.requestedByActorId) + : null; + const messageIds = Array.isArray(input.contextSnapshot.wakeCommentIds) + ? input.contextSnapshot.wakeCommentIds.filter( + (id): id is string => typeof id === "string", + ) + : readNonEmptyString(input.contextSnapshot.wakeCommentId) + ? [String(input.contextSnapshot.wakeCommentId)] + : []; + if ( + input.issueContext && + messageIds.length && + !input.contextSnapshot.retryOfRunId + ) { + const messages = await db + .select({ + id: issueComments.id, + authorUserId: issueComments.authorUserId, + }) + .from(issueComments) + .where( + and( + eq(issueComments.companyId, input.companyId), + eq(issueComments.issueId, input.issueContext.id), + inArray(issueComments.id, messageIds), + ), + ); + for (const id of [...messageIds].reverse()) { + const author = messages.find( + (message) => message.id === id, + )?.authorUserId; + if (author) { + delete input.contextSnapshot.executionIdentityCause; + return author; + } + } + } + const retryOfRunId = readNonEmptyString(input.contextSnapshot.retryOfRunId); + if (retryOfRunId) { + const [origin] = await db + .select({ responsibleUserId: heartbeatRuns.responsibleUserId }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.id, retryOfRunId), + ), + ); + if (origin?.responsibleUserId) return origin.responsibleUserId; + } + if (contextResponsibleUserId) return contextResponsibleUserId; + if (input.existingRunResponsibleUserId) + return input.existingRunResponsibleUserId; + if (input.routineEnvContext.responsibleUserId) + return input.routineEnvContext.responsibleUserId; + if (isManualUserRun(input) && requestedUserId) return requestedUserId; + if (input.issueContext?.responsibleUserId) + return input.issueContext.responsibleUserId; + const parentResponsibleUserId = await resolveParentIssueResponsibleUserId( + input.companyId, + input.issueContext?.parentId, + ); + if (parentResponsibleUserId) return parentResponsibleUserId; + if (!input.issueContext && requestedUserId) return requestedUserId; + input.contextSnapshot.executionIdentityCause = "company_default"; + return resolveCompanyDefaultResponsibleUserId(input.companyId); + } + + async function resolveResponsibleUserIdForRun(input: { + run: typeof heartbeatRuns.$inferSelect; + contextSnapshot: Record; + issueContext: Awaited> | null; + routineEnvContext: Awaited< + ReturnType + >; + }) { + const operatorIdentity = await explicitOperatorRunIdentity(db, input.run); + const responsibleUserId = operatorIdentity?.actorId ?? await resolveResponsibleUserIdForRunSeed({ + companyId: input.run.companyId, + contextSnapshot: input.contextSnapshot, + issueContext: input.issueContext, + routineEnvContext: input.routineEnvContext, + existingRunResponsibleUserId: input.run.responsibleUserId, + source: input.run.invocationSource as WakeupOptions["source"], + triggerDetail: input.run.triggerDetail as WakeupOptions["triggerDetail"], + }); + if (!responsibleUserId) { + throw new HttpError( + 422, + "Unable to resolve responsible user for heartbeat run dispatch", + { + code: "responsible_user_unresolved", + runId: input.run.id, + agentId: input.run.agentId, + companyId: input.run.companyId, + issueId: input.issueContext?.id ?? null, + invocationSource: input.run.invocationSource, + triggerDetail: input.run.triggerDetail, + wakeReason: readNonEmptyString(input.contextSnapshot.wakeReason), + }, + ); + } + return responsibleUserId; + } + + async function resolveResponsibleUserIdForRunContext( + run: typeof heartbeatRuns.$inferSelect, + contextSnapshot: Record, + ) { + const issueId = + readNonEmptyString(contextSnapshot.issueId) ?? + readNonEmptyString(contextSnapshot.taskId); + const issueContext = issueId + ? await getIssueExecutionContext(run.companyId, issueId) + : null; + return resolveResponsibleUserIdForRun({ + run, + contextSnapshot, + issueContext, + routineEnvContext: await getRoutineEnvForExecutionIssue( + run.companyId, + issueContext, + ), + }); + } + + return { + toAgentOrgRow, + listCompanyAgentOrgRows, + groupAgentOrgRowsByCompany, + getIssueExecutionContext, + getPinnedSkillTestContext, + getRoutineEnvForExecutionIssue, + resolveResponsibleUserIdForRunSeed, + resolveResponsibleUserIdForRun, + resolveResponsibleUserIdForRunContext, + }; +} diff --git a/server/src/services/heartbeat/workspaces.ts b/server/src/services/heartbeat/workspaces.ts index 5893ca1e7c6..69b879f3c28 100644 --- a/server/src/services/heartbeat/workspaces.ts +++ b/server/src/services/heartbeat/workspaces.ts @@ -1,3 +1,5 @@ +import { readConnectionFailure, type GitConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; +import { classifyGitCloneFailure, GitConnectionFailureError, readGitConnectionFailure } from "../git-connection-failure.js"; import { managedAiSessionFingerprintConfig } from "../ai-connection-runtime.js"; import { PROJECT_REPOSITORIES_DIR, @@ -549,22 +551,30 @@ async function materializeManagedProjectWorkspace( : null; const cloneTmpDir = await fs.mkdtemp(`${cwd}.clone-`); try { - await execFile( - "git", - [...(auth?.configArgs ?? []), "clone", "--no-hardlinks", "--", input.localSource ?? input.repoUrl, cloneTmpDir], - { - env: { - // Spread order matters: the sanitizer strips PAPERCLIP_*, which would remove the - // credential-helper token env if it came first. GIT_TERMINAL_PROMPT=0 fails a - // credential-less private clone immediately instead of hanging on a prompt until - // the clone timeout. - ...sanitizeRuntimeServiceBaseEnv(process.env), - GIT_TERMINAL_PROMPT: "0", - ...(auth?.env ?? {}), + try { + await execFile( + "git", + [...(auth?.configArgs ?? []), "clone", "--no-hardlinks", "--", input.localSource ?? input.repoUrl, cloneTmpDir], + { + env: { + // Spread order matters: the sanitizer strips PAPERCLIP_*, which would remove the + // credential-helper token env if it came first. GIT_TERMINAL_PROMPT=0 fails a + // credential-less private clone immediately instead of hanging on a prompt until + // the clone timeout. + ...sanitizeRuntimeServiceBaseEnv(process.env), + GIT_TERMINAL_PROMPT: "0", + ...(auth?.env ?? {}), + }, + timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, }, - timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, - }, - ); + ); + } catch (error) { + const connectionFailure = !input.localSource ? classifyGitCloneFailure(input.repoUrl, error) : null; + if (connectionFailure) throw new GitConnectionFailureError( + error instanceof Error ? error.message : "Git clone failed", connectionFailure, + ); + throw error; + } if (input.localSource) { const snapshot = await readGitWorkspaceSnapshot(input.localSource, false); if (!snapshot) throw new Error("Configured repository folder is not a Git checkout"); @@ -595,6 +605,8 @@ async function materializeManagedProjectWorkspace( // Preserve the closed failure code without copying subprocess output or // credentials into the durable run. Setup recovery needs the actual cause. if (isWorkspaceGitScanError(error)) throw new WorkspaceGitScanError(error.code, message); + const connectionFailure = readGitConnectionFailure(error); + if (connectionFailure) throw new GitConnectionFailureError(message, connectionFailure); throw new Error(message); } @@ -1060,8 +1072,10 @@ export async function assertGitWorktreeBaseWorkspaceReady(input: { reason: string, message: string, extra: Record = {}, + connectionFailure?: GitConnectionFailure, ) => { throw new WorkspaceValidationFailure(message, { + ...(connectionFailure ? { connectionFailure } : {}), workspaceValidation: { reason, issueId: input.issue!.id, @@ -1098,10 +1112,15 @@ export async function assertGitWorktreeBaseWorkspaceReady(input: { const materializationFailures = input.anchor?.materializationFailures ?? []; if (input.anchor?.baseCwdFallback && materializationFailures.length > 0) { const failureDetail = `: ${materializationFailures[0].error.replace(/\s+/g, " ")}`; + const connections = materializationFailures.map((failure) => readConnectionFailure(failure.connectionFailure)); + // Mixed connection/local failures must retain their application-error report. + const connectionFailure = connections.every((failure) => failure?.provider === "git") + ? connections[0] as GitConnectionFailure : undefined; fail( "git_worktree_base_materialization_failed", `Issue ${issueLabel} requested ${input.requestedExecutionWorkspaceMode} with git_worktree, but the project workspace checkout could not be prepared${failureDetail}. Repair the project workspace repository URL, clone access, or configured local cwd, then retry.`, { baseCwdFallback: true, materializationFailures }, + connectionFailure, ); } @@ -1410,6 +1429,7 @@ export type WorkspaceMaterializationFailure = { projectWorkspaceId: string | null; repoUrl: string | null; error: string; + connectionFailure?: GitConnectionFailure; }; export type ResolvedWorkspaceForRun = { @@ -3526,12 +3546,14 @@ export function createHeartbeatWorkspaceResolver(db: Db) { error instanceof Error ? error.message : String(error), ); const workspaceRepoUrl = readNonEmptyString(workspace.repoUrl); + const connectionFailure = readGitConnectionFailure(error); materializationFailures.push({ projectWorkspaceId: workspace.id, repoUrl: workspaceRepoUrl ? scrubGitCredentialText(workspaceRepoUrl) : null, error: scrubbedError, + ...(connectionFailure ? { connectionFailure } : {}), }); if (preferredWorkspace?.id === workspace.id) { preferredWorkspaceWarning = scrubbedError; diff --git a/server/src/services/issue-execution-policy.ts b/server/src/services/issue-execution-policy.ts index 859dc5fe1e3..ed38f91fb9b 100644 --- a/server/src/services/issue-execution-policy.ts +++ b/server/src/services/issue-execution-policy.ts @@ -401,7 +401,8 @@ export function normalizeIssueExecutionPolicy(input: unknown): IssueExecutionPol const reviewPreset = parsed.data.reviewPreset; const authorizationPolicy = parsed.data.authorizationPolicy; - if (stages.length === 0 && !monitor && !reviewPreset && !authorizationPolicy) return null; + if (stages.length === 0 && !monitor && !reviewPreset && !authorizationPolicy + && parsed.data.maxReviewRounds == null) return null; return { mode: parsed.data.mode ?? "normal", diff --git a/server/src/services/mcp-connection-failure.ts b/server/src/services/mcp-connection-failure.ts new file mode 100644 index 00000000000..b3cbb816e0b --- /dev/null +++ b/server/src/services/mcp-connection-failure.ts @@ -0,0 +1,63 @@ +import type { ConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; +import { HttpError } from "../errors.js"; +import { classifyRemoteConnectionError, hasUnclassifiedRemoteConnectionAttempts, readRemoteConnectionFailure } from "./remote-connection-failure.js"; + +type McpConnectionFailure = Extract; +type Reason = McpConnectionFailure["reason"]; + +// These are local provenance receipts, never properties supplied by a provider, +// request body, or persisted error. They affect Sentry reporting only. +const failures = new WeakMap(); + +function remember(error: Error, reason: Reason): void { + failures.set(error, { schemaVersion: 1, provider: "mcp_http", operation: "discover_tools", reason }); +} + +/** Call only around the outbound MCP request or its response-body read. */ +export async function withMcpConnectionFailure(request: () => Promise): Promise { + try { + return await request(); + } catch (error) { + if (error instanceof Error) { + if (hasUnclassifiedRemoteConnectionAttempts(error)) failures.delete(error); + const reason = readRemoteConnectionFailure(error) ?? classifyRemoteConnectionError(error); + if (reason) remember(error, reason); + } + throw error; + } +} + +/** Preserve the existing HTTP failure; classify only the actual remote status. */ +export function mcpDiscoveryHttpFailure(response: Response, message: string): HttpError { + return markMcpHttpResponseFailure(response, new HttpError(502, message, { status: response.status })); +} + +/** Only an actual MCP response can attach this evidence to a protocol wrapper. */ +export function markMcpHttpResponseFailure(response: Response, error: T): T { + const status = response.status; + const reason: Reason | null = status === 401 || status === 403 ? "authentication_failed" + : status === 404 ? "endpoint_not_found" + : status === 429 ? "rate_limited" + : status >= 500 && status <= 599 ? "remote_unavailable" : null; + if (reason) remember(error, reason); + return error; +} + +/** Rewrapping a known error must not classify unrelated persistence failures. */ +export function retainMcpConnectionFailure(source: unknown, target: T): T { + if (hasUnclassifiedRemoteConnectionAttempts(source)) { + failures.delete(target); + return target; + } + const failure = source instanceof Error ? failures.get(source) : undefined; + if (failure) failures.set(target, failure); + else { + const reason = readRemoteConnectionFailure(source); + if (reason) remember(target, reason); + } + return target; +} + +export function isExpectedMcpConnectionFailure(error: unknown): boolean { + return error instanceof HttpError && error.status === 502 && !hasUnclassifiedRemoteConnectionAttempts(error) && failures.has(error); +} diff --git a/server/src/services/mcp-http.ts b/server/src/services/mcp-http.ts index 445d26b93ea..80db7f9c17e 100644 --- a/server/src/services/mcp-http.ts +++ b/server/src/services/mcp-http.ts @@ -1,4 +1,5 @@ import { createHash } from "node:crypto"; +import { markMcpHttpResponseFailure, retainMcpConnectionFailure, withMcpConnectionFailure } from "./mcp-connection-failure.js"; // Helpers for talking to remote MCP servers over the Streamable HTTP transport. // // The MCP Streamable HTTP spec requires the client to advertise that it accepts @@ -72,18 +73,18 @@ export async function initializeMcpHttpSession(input: { }), }); if (!initializeResponse.ok) { - throw new McpHttpInitializationError( + throw markMcpHttpResponseFailure(initializeResponse, new McpHttpInitializationError( `Remote MCP initialization returned HTTP ${initializeResponse.status}`, "initialize", initializeResponse.status, initializeResponse, - ); + )); } let payload: unknown; try { - payload = await readMcpHttpResponse(initializeResponse, `${input.requestId}-initialize`); - } catch { - throw new McpHttpInitializationError("Remote MCP initialization returned an invalid response", "initialize", null); + payload = await withMcpConnectionFailure(() => readMcpHttpResponse(initializeResponse, `${input.requestId}-initialize`)); + } catch (error) { + throw retainMcpConnectionFailure(error, new McpHttpInitializationError("Remote MCP initialization returned an invalid response", "initialize", null)); } const result = payload && typeof payload === "object" && "result" in payload ? (payload as { result?: unknown }).result @@ -109,11 +110,11 @@ export async function initializeMcpHttpSession(input: { }), }); if (!initializedResponse.ok) { - throw new McpHttpInitializationError( + throw markMcpHttpResponseFailure(initializedResponse, new McpHttpInitializationError( `Remote MCP initialized notification returned HTTP ${initializedResponse.status}`, "initialized_notification", initializedResponse.status, - ); + )); } return sessionHeaders; } diff --git a/server/src/services/remote-connection-failure.ts b/server/src/services/remote-connection-failure.ts new file mode 100644 index 00000000000..25b1e0269b3 --- /dev/null +++ b/server/src/services/remote-connection-failure.ts @@ -0,0 +1,80 @@ +/** Server-local provenance of a failure at a remote connection boundary. */ +export type RemoteConnectionFailureReason = "connection_refused" | "dns_failure" | "network_unreachable" + | "connection_reset" | "connection_timeout" | "tls_failure"; +type Reason = RemoteConnectionFailureReason; +const failures = new WeakMap(); +const failedAttempts = new WeakMap(); + +function attemptFailureReason(reasons: readonly (Reason | null)[]): Reason | null { + return reasons.length > 0 && reasons.every((reason) => reason !== null) ? reasons[reasons.length - 1]! : null; +} + +/** Keep every address's classification without changing the final thrown value. */ +export function recordRemoteConnectionAttempts(error: T, causes: readonly unknown[]): T { + if (error instanceof Error) { + const reasons = causes.map((cause) => readRemoteConnectionFailure(cause) ?? classifyRemoteConnectionError(cause)); + failedAttempts.set(error, Object.freeze(reasons)); + } + return error; +} + +export function hasUnclassifiedRemoteConnectionAttempts(error: unknown): boolean { + const attempts = error instanceof Error ? failedAttempts.get(error) : undefined; + return attempts !== undefined && attemptFailureReason(attempts) === null; +} + +export function markRemoteConnectionFailure(error: T, reason: Reason): T { + failures.set(error, reason); + return error; +} + +export function readRemoteConnectionFailure(error: unknown): Reason | null { + if (!(error instanceof Error)) return null; + const attempts = failedAttempts.get(error); + // An explicit unknown attempt overrides even a known final owned deadline. + return attempts ? attemptFailureReason(attempts) : failures.get(error) ?? null; +} + +const transportReasons: Readonly> = { + ECONNREFUSED: "connection_refused", + ENOTFOUND: "dns_failure", + ENODATA: "dns_failure", + EAI_AGAIN: "dns_failure", + EHOSTUNREACH: "network_unreachable", + ENETUNREACH: "network_unreachable", + ECONNRESET: "connection_reset", + EPIPE: "connection_reset", + UND_ERR_SOCKET: "connection_reset", + ETIMEDOUT: "connection_timeout", + UND_ERR_CONNECT_TIMEOUT: "connection_timeout", + UND_ERR_HEADERS_TIMEOUT: "connection_timeout", + UND_ERR_BODY_TIMEOUT: "connection_timeout", + CERT_HAS_EXPIRED: "tls_failure", + CERT_NOT_YET_VALID: "tls_failure", + DEPTH_ZERO_SELF_SIGNED_CERT: "tls_failure", + SELF_SIGNED_CERT_IN_CHAIN: "tls_failure", + UNABLE_TO_VERIFY_LEAF_SIGNATURE: "tls_failure", + UNABLE_TO_GET_ISSUER_CERT_LOCALLY: "tls_failure", + ERR_TLS_CERT_ALTNAME_INVALID: "tls_failure", +}; + +export function classifyRemoteConnectionError(value: unknown, depth = 0): Reason | null { + if (!value || typeof value !== "object" || Array.isArray(value) || depth > 2) return null; + const attempts = value instanceof Error ? failedAttempts.get(value) : undefined; + // Do not fall back to the last error's code/cause after mixed attempts. + if (attempts) return attemptFailureReason(attempts); + const error = value as Record; + if (error.name === "AbortError") return null; + const reason = typeof error.code === "string" && Object.hasOwn(transportReasons, error.code) + ? transportReasons[error.code]! + : null; + if (error.code !== undefined && !reason) return null; + if (error.errors !== undefined) { + if (!Array.isArray(error.errors) || error.errors.length === 0 || error.errors.length > 8) return null; + const reasons = error.errors.map((child) => classifyRemoteConnectionError(child, depth + 1)); + const first = reasons[0]; + return first && reasons.every((child) => child === first) && (!reason || reason === first) ? first : null; + } + if (reason) return reason; + return error.cause !== undefined ? classifyRemoteConnectionError(error.cause, depth + 1) : null; +} diff --git a/server/src/services/remote-http-endpoint-guard.ts b/server/src/services/remote-http-endpoint-guard.ts index 0c5c0b3ce34..a120718b1ea 100644 --- a/server/src/services/remote-http-endpoint-guard.ts +++ b/server/src/services/remote-http-endpoint-guard.ts @@ -1,5 +1,6 @@ import { lookup as dnsLookup } from "node:dns/promises"; import { isIP } from "node:net"; +import { classifyRemoteConnectionError, markRemoteConnectionFailure, readRemoteConnectionFailure } from "./remote-connection-failure.js"; const DEFAULT_DNS_TIMEOUT_MS = 5_000; @@ -83,11 +84,14 @@ export async function resolveApprovedRemoteHttpAddresses( options.lookup ?? defaultLookup, options.dnsTimeoutMs ?? DEFAULT_DNS_TIMEOUT_MS, ); - } catch { - throw error("Remote MCP connection hostname could not be resolved", "remote_http_dns_failed"); + } catch (cause) { + const failure = error("Remote MCP connection hostname could not be resolved", "remote_http_dns_failed"); + const reason = readRemoteConnectionFailure(cause) ?? classifyRemoteConnectionError(cause); + if (reason === "dns_failure" || reason === "connection_timeout") markRemoteConnectionFailure(failure, reason); + throw failure; } if (results.length === 0) { - throw error("Remote MCP connection hostname did not resolve", "remote_http_dns_failed"); + throw markRemoteConnectionFailure(error("Remote MCP connection hostname did not resolve", "remote_http_dns_failed"), "dns_failure"); } if (results.some((result) => isAlwaysDeniedLinkLocalIp(result.address) @@ -123,7 +127,7 @@ async function lookupWithTimeout(hostname: string, lookup: RemoteHttpEndpointLoo lookup(hostname), new Promise((_, reject) => { timeout = setTimeout(() => { - reject(new Error(`DNS lookup timed out for ${hostname}`)); + reject(markRemoteConnectionFailure(new Error(`DNS lookup timed out for ${hostname}`), "connection_timeout")); }, timeoutMs); timeout.unref?.(); }), diff --git a/server/src/services/remote-http-fetch.ts b/server/src/services/remote-http-fetch.ts index c6efa351344..b02d8769fa3 100644 --- a/server/src/services/remote-http-fetch.ts +++ b/server/src/services/remote-http-fetch.ts @@ -4,6 +4,7 @@ import { connect as netConnect, isIP, type Socket } from "node:net"; import { Readable } from "node:stream"; import { connect as tlsConnect, type TLSSocket } from "node:tls"; import { createBrotliDecompress, createGunzip, createInflate } from "node:zlib"; +import { classifyRemoteConnectionError, markRemoteConnectionFailure, recordRemoteConnectionAttempts } from "./remote-connection-failure.js"; import { isAlwaysDeniedLinkLocalIp, @@ -104,10 +105,12 @@ export async function guardedRemoteHttpFetch( return await platformFetch(endpoint.toString(), { ...init, redirect: "manual" }); } catch (error) { if (isDnsResolutionError(error)) { - throw options.error( + const failure = options.error( "Remote MCP connection hostname could not be resolved", "remote_http_dns_failed", ); + if (classifyRemoteConnectionError(error) === "dns_failure") markRemoteConnectionFailure(failure, "dns_failure"); + throw failure; } throw error; } @@ -188,17 +191,21 @@ async function dialApprovedAddress(input: { options: GuardedRemoteHttpFetchOptions; }): Promise { let lastReason: unknown; + const failedAttempts: unknown[] = []; for (const address of input.approved) { input.signal?.throwIfAborted?.(); try { return await openVerifiedSocket({ ...input, address }); } catch (error) { - if (!(error instanceof UnreachableAddressError)) throw error; + if (!(error instanceof UnreachableAddressError)) { + throw recordRemoteConnectionAttempts(error, [...failedAttempts, error]); + } lastReason = error.reason; + failedAttempts.push(error.reason); } } - throw lastReason - ?? input.options.error("Remote MCP endpoint could not be reached", "remote_http_connect_failed"); + throw recordRemoteConnectionAttempts(lastReason + ?? input.options.error("Remote MCP endpoint could not be reached", "remote_http_connect_failed"), failedAttempts); } /** @@ -318,7 +325,7 @@ async function sendRequest(input: { const headersTimer = setTimeout(() => { // Destroying the request tears the socket down too, so a silent peer costs // neither a pending handler nor a leaked descriptor. - req.destroy(error("Remote MCP endpoint did not respond in time", "remote_http_response_timeout")); + req.destroy(markRemoteConnectionFailure(error("Remote MCP endpoint did not respond in time", "remote_http_response_timeout"), "connection_timeout")); }, responseTimeoutMs); headersTimer.unref?.(); req.on("response", () => clearTimeout(headersTimer)); @@ -330,7 +337,7 @@ async function sendRequest(input: { // Body idle deadline, mirroring undici's `bodyTimeout`: a stalled stream is // destroyed so `response.text()` rejects instead of hanging the caller. message.setTimeout(responseTimeoutMs, () => { - message.destroy(error("Remote MCP endpoint stalled mid-response", "remote_http_response_timeout")); + message.destroy(markRemoteConnectionFailure(error("Remote MCP endpoint stalled mid-response", "remote_http_response_timeout"), "connection_timeout")); }); // A finished response must not leave an armed socket timer behind, or a later // reader of the same socket inherits a deadline it never asked for. @@ -395,7 +402,7 @@ function once( ): Promise { return new Promise((resolve, reject) => { const timer = setTimeout(() => { - settle(new Error(`Timed out trying to ${input.what} the remote MCP endpoint`)); + settle(markRemoteConnectionFailure(new Error(`Timed out trying to ${input.what} the remote MCP endpoint`), "connection_timeout")); }, input.timeoutMs); timer.unref?.(); diff --git a/server/src/services/run-failure-report.ts b/server/src/services/run-failure-report.ts index 33f6fbe33e8..d44a1329280 100644 --- a/server/src/services/run-failure-report.ts +++ b/server/src/services/run-failure-report.ts @@ -11,6 +11,7 @@ import { } from "./run-failure-diagnostics.js"; import { logger } from "../middleware/logger.js"; import { isUnexpectedRunCancellation } from "./run-cancellation.js"; +import { readConnectionFailure, type ConnectionFailure } from "@paperclipai/adapter-utils/connection-failure"; type HeartbeatRun = typeof heartbeatRuns.$inferSelect; @@ -103,6 +104,49 @@ function isLocalPathWorkspaceConfigurationBlocker(run: HeartbeatRun, options: Ru recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false; } +/** Only the managed-clone producer can attach this bounded pre-dispatch evidence. */ +function isGitConnectionFailure(run: HeartbeatRun, options: RunFailureReportOptions): boolean { + const connection = readConnectionFailure(run.resultJson?.connectionFailure); + if (connection?.provider !== "git" || run.status !== "failed" || run.executionStage !== "preparing" || + options.phase !== "setup" || run.exitCode != null || run.signal != null) return false; + const recovery = asRecord(run.resultJson?.executionRecovery); + if (recovery?.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return false; + const validation = asRecord(run.resultJson?.workspaceValidation); + if (run.errorCode === "setup_failed") return !validation; + if (run.errorCode !== "workspace_validation_failed" || validation?.reason !== "git_worktree_base_materialization_failed") return false; + const failures = validation.materializationFailures; + return Array.isArray(failures) && failures.length > 0 && failures.every((failure) => { + const marker = readConnectionFailure(asRecord(failure)?.connectionFailure); + return marker?.provider === "git"; + }) && readConnectionFailure(asRecord(failures[0])?.connectionFailure)?.reason === connection.reason; +} + +const hermesTransportReasons = new Set([ + "connection_refused", "dns_failure", "network_unreachable", "connection_reset", "connection_timeout", "tls_failure", +]); + +/** An adapter result remains failed; this only excludes its proven external cause. */ +function isHermesConnectionFailure(run: HeartbeatRun, options: RunFailureReportOptions, adapterType: string | undefined): boolean { + if (adapterType !== "hermes_gateway" || run.status !== "failed" || run.exitCode !== 1 || run.signal != null || options.phase === "setup") return false; + const connection = readConnectionFailure(run.resultJson?.connectionFailure); + if (connection?.provider !== "hermes_gateway") return false; + const code = run.errorCode; + if (connection.operation === "configuration") { + const codes: Record["reason"], string> = { + endpoint_missing: "hermes_gateway_api_base_url_missing", + endpoint_invalid: "hermes_gateway_api_base_url_invalid", + insecure_transport: "hermes_gateway_plain_http_remote_denied", + credentials_missing: "hermes_gateway_api_key_missing", + }; + return code === codes[connection.reason]; + } + if (hermesTransportReasons.has(connection.reason)) return code === "hermes_gateway_connect_failed" || code === "hermes_gateway_protocol_error"; + return (connection.reason === "authentication_failed" && code === "hermes_gateway_auth_failed") || + (connection.reason === "endpoint_not_found" && code === "hermes_gateway_runs_unsupported") || + (connection.reason === "rate_limited" && code === "hermes_gateway_rate_limited") || + (connection.reason === "remote_unavailable" && code === "hermes_gateway_upstream_error"); +} + /** * Report a terminal run failure to Sentry. Returns at once for any status * other than failures and unexpected started cancellations. Never throws — a Sentry failure or a @@ -119,6 +163,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR if (run.status === "cancelled" && !isUnexpectedRunCancellation(run)) return Promise.resolve(); if (isMissingSecretConfigurationBlocker(run, options)) return Promise.resolve(); if (isLocalPathWorkspaceConfigurationBlocker(run, options)) return Promise.resolve(); + if (isGitConnectionFailure(run, options)) return Promise.resolve(); const runStatus = run.status; const report = captureTerminalRunFailure(db, run, runStatus, options); pendingRunFailureReports.add(report); @@ -147,6 +192,8 @@ async function captureTerminalRunFailure( .where(and(eq(agents.id, run.agentId), eq(agents.companyId, run.companyId))) .then((rows) => rows[0] ?? null); + if (isHermesConnectionFailure(run, options, agent?.adapterType)) return; + const taskId = readTaskId(run); if (!taskId) { logger.warn({ runId: run.id }, "run failure report has no task id, skipping Sentry report"); diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index b01550aea82..fb473843888 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -10,6 +10,7 @@ import { composioAppAccounts, composioAppSetupResult } from "./composio-app-setu import { honchoManagedArguments } from "./honcho-connection.js"; import { defaultConnectionAgentInstructions } from "@paperclipai/shared"; import { isInsufficientConnectionScope, INSUFFICIENT_CONNECTION_SCOPE_MESSAGE } from "./connection-permission-errors.js"; +import { mcpDiscoveryHttpFailure, retainMcpConnectionFailure, withMcpConnectionFailure } from "./mcp-connection-failure.js"; import { ASANA_CONNECTOR_SCOPES, isAsanaConnectorProfileId, type AsanaConnectorProfileId } from "@paperclipai/shared"; import { BROWSER_USE_TOOLS } from "@paperclipai/shared"; import { browserUseClient, isBrowserUseConnection } from "./browser-use-client.js"; @@ -2885,6 +2886,7 @@ function healthFailureHttpStatus(failure: { if (failure.code === "oauth_challenge") return 422; if (failure.code === "oauth_refresh_missing") return 422; if (failure.code === "oauth_reauthorization_required") return 422; + if (failure.code === "oauth_insufficient_scope") return 422; if (failure.code === "slack_mcp_access_disabled") return 422; if (failure.code === "user_authorization_required") return 422; if (failure.code === "composio_broker_retired") return 422; @@ -7141,7 +7143,7 @@ export function toolAccessService( // PAP-17098 closed for the OAuth endpoints. let listRequestId = "paperclip-catalog-refresh"; let sessionHeaders = headers; - const sendRemote = (init: RequestInit) => requestRemoteHttpEndpoint(new URL(endpoint), init); + const sendRemote = (init: RequestInit) => withMcpConnectionFailure(() => requestRemoteHttpEndpoint(new URL(endpoint), init)); const sendToolsList = (requestHeaders: Record, cursor?: string) => { sessionHeaders = requestHeaders; return sendRemote({ method: "POST", headers: mcpHttpRequestHeaders(requestHeaders), @@ -7329,14 +7331,12 @@ export function toolAccessService( oauthSupported: Boolean(endpoints), }); } - throw new HttpError(502, `Remote app returned HTTP ${response.status}`, { - status: response.status, - }); + throw mcpDiscoveryHttpFailure(response, `Remote app returned HTTP ${response.status}`); } const descriptors: McpToolDescriptor[] = []; const seenCursors = new Set(); for (let page = 0; ; page += 1) { - const payload = await readMcpHttpResponse(response, listRequestId); + const payload = await withMcpConnectionFailure(() => readMcpHttpResponse(response, listRequestId)); const record = asRecord(payload); if (record.error) throw new HttpError(502, "Remote MCP tool discovery failed", { code: "mcp_catalog_error" }); const result = asRecord(record.result); @@ -7348,7 +7348,7 @@ export function toolAccessService( seenCursors.add(cursor); listRequestId = `paperclip-catalog-refresh-${page + 1}`; response = await sendToolsList(sessionHeaders, cursor); - if (!response.ok) throw new HttpError(502, "Remote MCP catalog page could not be read", { status: response.status }); + if (!response.ok) throw mcpDiscoveryHttpFailure(response, "Remote MCP catalog page could not be read"); } if (!isRailwayConnection(connection)) return descriptors; if (descriptors.some((tool) => normalizeRailwayToolName(tool.name).startsWith(RAILWAY_TOOL_PREFIX))) { @@ -7699,13 +7699,13 @@ export function toolAccessService( actor, details: { status: failure.status, transport: connection.transport }, }); - throw new HttpError(healthFailureHttpStatus(failure), failure.message, { + throw retainMcpConnectionFailure(error, new HttpError(healthFailureHttpStatus(failure), failure.message, { code: failure.code, connection: toConnection(updated), runtimeSlot, setupUrl: connectionSetupUrl(connection), reconnectUrl: connectionReconnectUrl(connection), - }); + })); } } @@ -7752,11 +7752,11 @@ export function toolAccessService( details: { status: failure.status }, actor, }); - throw new HttpError(healthFailureHttpStatus(failure), failure.message, { + throw retainMcpConnectionFailure(error, new HttpError(healthFailureHttpStatus(failure), failure.message, { code: failure.code, setupUrl: connectionSetupUrl(connection), reconnectUrl: connectionReconnectUrl(connection), - }); + })); } const existingRows = await db diff --git a/tests/runner-e2e/daytona-image-content.ts b/tests/runner-e2e/daytona-image-content.ts index 2c79b6d52a1..0bb601acf08 100644 --- a/tests/runner-e2e/daytona-image-content.ts +++ b/tests/runner-e2e/daytona-image-content.ts @@ -37,6 +37,7 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [ "packages/paperclip-runner/runner/crates", "packages/paperclip-runner/scripts/acpx-sidecar-contract.mjs", "packages/paperclip-runner/scripts/build-provider-pack.mjs", + "packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs", "packages/paperclip-runner/scripts/build-node-startup-timeout.mjs", "packages/paperclip-runner/scripts/candidate-provider-pack.mjs", "packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs", diff --git a/tests/runner-e2e/daytona-image.test.ts b/tests/runner-e2e/daytona-image.test.ts index 25696be09f0..bb38e762cbf 100644 --- a/tests/runner-e2e/daytona-image.test.ts +++ b/tests/runner-e2e/daytona-image.test.ts @@ -334,9 +334,11 @@ describe("runner E2E Daytona image contract", () => { } }); - it("invalidates the image when the Cursor runtime isolation patch changes", async () => { - const root = await mkdtemp(path.join(tmpdir(), "paperclip-cursor-image-id-")); - const patchPath = "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs"; + it.each([ + "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs", + "packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs", + ])("invalidates the image when %s changes", async (patchPath) => { + const root = await mkdtemp(path.join(tmpdir(), "paperclip-provider-image-id-")); expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(patchPath); const options = { repositoryRoot: root, diff --git a/ui/src/components/ExecutionParticipantPicker.test.tsx b/ui/src/components/ExecutionParticipantPicker.test.tsx new file mode 100644 index 00000000000..98fa99a5aab --- /dev/null +++ b/ui/src/components/ExecutionParticipantPicker.test.tsx @@ -0,0 +1,70 @@ +// @vitest-environment jsdom +import { act, type ReactNode } from "react"; +import { createRoot } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { issueExecutionPolicySchema, type Issue } from "@paperclipai/shared"; +import { describe, expect, it, vi } from "vitest"; +import { ExecutionParticipantPicker } from "./ExecutionParticipantPicker"; + +vi.mock("../api/access", () => ({ accessApi: { listUserDirectory: async () => ({ users: [] }) } })); +vi.mock("../lib/sentry", () => ({ captureBrowserException: vi.fn() })); +vi.mock("@/components/ui/popover", () => ({ + Popover: ({ children }: { children: ReactNode }) =>
{children}
, + PopoverTrigger: ({ children }: { children: ReactNode }) => <>{children}, + PopoverContent: ({ children }: { children: ReactNode }) =>
{children}
, +})); + +function renderPicker(policy: unknown) { + const container = document.createElement("div"); + const root = createRoot(container); + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + const onUpdate = vi.fn(); + const issue = { companyId: "company-fixture", executionPolicy: policy } as Issue; + act(() => root.render( + + )); + return { container, onUpdate, cleanup: () => { act(() => root.unmount()); client.clear(); } }; +} + +describe("ExecutionParticipantPicker", () => { + it("adds a reviewer to a valid policy that omits stages without dropping authorization", () => { + const policy = { authorizationPolicy: { trustPreset: "standard" }, maxReviewRounds: 4 }; + const { container, onUpdate, cleanup } = renderPicker(policy); + expect(container.textContent).not.toContain("Execution policy unavailable"); + const assign = Array.from(container.querySelectorAll("button")).find((button) => button.textContent === "Assign to me"); + expect(assign).toBeTruthy(); + act(() => assign!.click()); + expect(onUpdate).toHaveBeenCalledTimes(1); + const updated = onUpdate.mock.calls[0][0].executionPolicy; + expect(issueExecutionPolicySchema.safeParse(updated).success).toBe(true); + expect(updated).toMatchObject({ ...policy, stages: [{ type: "review", participants: [{ type: "user", userId: "board-reviewer" }] }] }); + expect(policy).not.toHaveProperty("stages"); + cleanup(); + }); + + it("shows an existing reviewer and preserves other policy settings when that reviewer is removed", () => { + const policy = { + authorizationPolicy: { trustPreset: "standard" }, maxReviewRounds: 4, + stages: [{ type: "review", participants: [{ type: "user", userId: "board-reviewer" }] }], + }; + const { container, onUpdate, cleanup } = renderPicker(policy); + expect(container.querySelector("button")?.textContent).toBe("You"); + const remove = Array.from(container.querySelectorAll("button")).find((button) => button.textContent === "No reviewers"); + expect(remove).toBeTruthy(); + act(() => remove!.click()); + expect(onUpdate).toHaveBeenCalledWith({ executionPolicy: { + mode: "normal", commentRequired: true, stages: [], + authorizationPolicy: policy.authorizationPolicy, maxReviewRounds: 4, + } }); + expect(policy.stages[0].participants).toHaveLength(1); + cleanup(); + }); + + it("keeps a malformed policy unavailable and exposes no mutation controls", () => { + const { container, onUpdate, cleanup } = renderPicker({ stages: [{ type: "review", participants: {} }] }); + expect(container.textContent).toBe("Execution policy unavailable. Refresh to try again."); + expect(container.querySelector("button")).toBeNull(); + expect(onUpdate).not.toHaveBeenCalled(); + cleanup(); + }); +}); diff --git a/ui/src/components/ExecutionParticipantPicker.tsx b/ui/src/components/ExecutionParticipantPicker.tsx index b13d60748f7..b5aba28216a 100644 --- a/ui/src/components/ExecutionParticipantPicker.tsx +++ b/ui/src/components/ExecutionParticipantPicker.tsx @@ -11,6 +11,7 @@ import { buildExecutionPolicy, stageParticipantValues, } from "../lib/issue-execution-policy"; +import { useExecutionPolicy } from "../hooks/useExecutionPolicy"; import { cn } from "../lib/utils"; import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover"; import { User, Eye, ShieldCheck } from "lucide-react"; @@ -34,6 +35,7 @@ export function ExecutionParticipantPicker({ }: ExecutionParticipantPickerProps) { const [open, setOpen] = useState(false); const [search, setSearch] = useState(""); + const policyAvailable = useExecutionPolicy(issue.executionPolicy).success; const reviewerValues = stageParticipantValues(issue.executionPolicy, "review"); const approverValues = stageParticipantValues(issue.executionPolicy, "approval"); @@ -73,6 +75,7 @@ export function ExecutionParticipantPicker({ }; const updatePolicy = (nextValues: string[]) => { + if (!policyAvailable) return; onUpdate({ executionPolicy: buildExecutionPolicy({ existingPolicy: issue.executionPolicy ?? null, @@ -92,6 +95,10 @@ export function ExecutionParticipantPicker({ const label = stageType === "review" ? "Reviewers" : "Approvers"; const Icon = stageType === "review" ? Eye : ShieldCheck; + if (!policyAvailable) { + return Execution policy unavailable. Refresh to try again.; + } + return ( { setOpen(o); if (!o) setSearch(""); }}> diff --git a/ui/src/components/IssueProperties.test.tsx b/ui/src/components/IssueProperties.test.tsx index 683acb658ec..dd19e8712b6 100644 --- a/ui/src/components/IssueProperties.test.tsx +++ b/ui/src/components/IssueProperties.test.tsx @@ -2735,10 +2735,10 @@ describe("IssueProperties", () => { executionPolicy: createExecutionPolicy({ stages: [ { - id: "review-stage", + id: "00000000-0000-4000-8000-000000000010", type: "review", approvalsNeeded: 1, - participants: [{ id: "participant-1", type: "agent", agentId: "agent-1", userId: null }], + participants: [{ id: "00000000-0000-4000-8000-000000000012", type: "agent", agentId: "00000000-0000-4000-8000-000000000013", userId: null }], }, ], }), @@ -2767,10 +2767,10 @@ describe("IssueProperties", () => { executionPolicy: createExecutionPolicy({ stages: [ { - id: "approval-stage", + id: "00000000-0000-4000-8000-000000000011", type: "approval", approvalsNeeded: 1, - participants: [{ id: "participant-2", type: "user", agentId: null, userId: "user-1" }], + participants: [{ id: "00000000-0000-4000-8000-000000000014", type: "user", agentId: null, userId: "user-1" }], }, ], }), @@ -2793,10 +2793,10 @@ describe("IssueProperties", () => { executionPolicy: createExecutionPolicy({ stages: [ { - id: "review-stage", + id: "00000000-0000-4000-8000-000000000010", type: "review", approvalsNeeded: 1, - participants: [{ id: "participant-1", type: "agent", agentId: "agent-1", userId: null }], + participants: [{ id: "00000000-0000-4000-8000-000000000012", type: "agent", agentId: "00000000-0000-4000-8000-000000000013", userId: null }], }, ], }), @@ -2819,10 +2819,10 @@ describe("IssueProperties", () => { executionPolicy: createExecutionPolicy({ stages: [ { - id: "review-stage", + id: "00000000-0000-4000-8000-000000000010", type: "review", approvalsNeeded: 1, - participants: [{ id: "participant-1", type: "agent", agentId: "agent-1", userId: null }], + participants: [{ id: "00000000-0000-4000-8000-000000000012", type: "agent", agentId: "00000000-0000-4000-8000-000000000013", userId: null }], }, ], }), @@ -2924,6 +2924,60 @@ describe("IssueProperties", () => { dateNowSpy.mockRestore(); }); + it.each([true, false])("clears a monitor with omitted stages while preserving independent limits (authorization: %s)", async (withAuthorization) => { + const onUpdate = vi.fn(); + const executionPolicy = { + ...(withAuthorization ? { authorizationPolicy: { trustPreset: "standard" } } : {}), + maxReviewRounds: 4, + monitor: { nextCheckAt: "2099-01-01T12:00:00.000Z", notes: "Check service", scheduledBy: "board" }, + } as IssueExecutionPolicy; + const before = structuredClone(executionPolicy); + const root = renderProperties(container, { + issue: createIssue({ executionPolicy }), childIssues: [], onUpdate, inline: true, + }); + await flush(); + + expect(container.textContent).toContain("Reviewers"); + expect(container.textContent).not.toContain("Execution policy unavailable"); + const monitorTrigger = container.querySelector('[data-testid="monitor-row-trigger"]')?.closest("button"); + expect(monitorTrigger).toBeTruthy(); + await act(async () => { monitorTrigger!.dispatchEvent(new MouseEvent("click", { bubbles: true })); }); + const clearButton = Array.from(container.querySelectorAll("button")).find((button) => button.textContent === "Clear"); + expect(clearButton).toBeTruthy(); + await act(async () => { clearButton!.dispatchEvent(new MouseEvent("click", { bubbles: true })); }); + + expect(onUpdate).toHaveBeenCalledWith({ executionPolicy: { + mode: "normal", commentRequired: true, stages: [], + ...(withAuthorization ? { authorizationPolicy: before.authorizationPolicy } : {}), maxReviewRounds: 4, + } }); + expect(executionPolicy).toEqual(before); + act(() => root.unmount()); + }); + + it.each([ + { stages: {} }, + { stages: [null] }, + { stages: [{ type: "review", participants: {} }] }, + { monitor: { nextCheckAt: {}, notes: {} } }, + ])("keeps malformed policies unavailable without exposing destructive edits: %j", async (executionPolicy) => { + const onUpdate = vi.fn(); + const root = renderProperties(container, { + issue: createIssue({ executionPolicy: executionPolicy as unknown as IssueExecutionPolicy }), + childIssues: [], onUpdate, inline: true, + }); + await flush(); + + expect(container.textContent).toContain("Execution policy unavailable. Refresh to try again."); + expect(container.textContent).toContain("Assignee"); + expect(container.textContent).not.toContain("No reviewers"); + expect(container.textContent).not.toContain("No approvers"); + expect(container.textContent).not.toContain("Run review now"); + expect(container.querySelector('[data-testid="monitor-row-trigger"]')).toBeNull(); + expect(container.querySelector('input[type="datetime-local"]')).toBeNull(); + expect(onUpdate).not.toHaveBeenCalled(); + act(() => root.unmount()); + }); + it("renders scheduled, retrying, due, overdue, cleared, and empty monitor row states", async () => { // Anchored to a fixed *local* time, not a fixed UTC instant. The row renders // these in the machine's timezone and labels them "Today" only while they diff --git a/ui/src/components/issue-properties/IssueProperties.tsx b/ui/src/components/issue-properties/IssueProperties.tsx index db6de10d240..2b00c76abae 100644 --- a/ui/src/components/issue-properties/IssueProperties.tsx +++ b/ui/src/components/issue-properties/IssueProperties.tsx @@ -47,6 +47,7 @@ import { getRecentProjectIds } from "../../lib/recent-projects"; import { orderItemsBySelectedAndRecent } from "../../lib/recent-selections"; import { formatAssigneeUserLabel, formatUserLabel } from "../../lib/assignees"; import { buildExecutionPolicy, stageParticipantValues } from "../../lib/issue-execution-policy"; +import { useExecutionPolicy } from "../../hooks/useExecutionPolicy"; import { formatMonitorAbsolute, formatMonitorAbsoluteFull, @@ -246,6 +247,9 @@ export function IssueProperties({ documentDeepLink, sidePanelContentOnly = false, }: IssuePropertiesProps) { + const policyResult = useExecutionPolicy(issue.executionPolicy); + const policyAvailable = policyResult.success; + const executionPolicy = policyResult.success ? policyResult.data : null; const { selectedCompanyId } = useCompany(); const { isMobile } = useSidebar(); const queryClient = useQueryClient(); @@ -392,9 +396,9 @@ export function IssueProperties({ const [newLabelName, setNewLabelName] = useState(""); // token-extraction: allowlisted — color-picker seed state, persisted into label-create payload; a var() string would break that payload. const [newLabelColor, setNewLabelColor] = useState("#6366f1"); - const [monitorAtInput, setMonitorAtInput] = useState(() => toDateTimeLocalValue(issue.executionPolicy?.monitor?.nextCheckAt)); - const [monitorNotesInput, setMonitorNotesInput] = useState(issue.executionPolicy?.monitor?.notes ?? ""); - const [monitorServiceInput, setMonitorServiceInput] = useState(issue.executionPolicy?.monitor?.serviceName ?? ""); + const [monitorAtInput, setMonitorAtInput] = useState(() => toDateTimeLocalValue(executionPolicy?.monitor?.nextCheckAt)); + const [monitorNotesInput, setMonitorNotesInput] = useState(executionPolicy?.monitor?.notes ?? ""); + const [monitorServiceInput, setMonitorServiceInput] = useState(executionPolicy?.monitor?.serviceName ?? ""); const [runtimeActionMessage, setRuntimeActionMessage] = useState(null); const [runtimeActionErrorMessage, setRuntimeActionErrorMessage] = useState(null); const [unarchiveErrorMessage, setUnarchiveErrorMessage] = useState(null); @@ -999,6 +1003,7 @@ export function IssueProperties({ applyAssignee(next, track); }; const updateExecutionPolicy = (nextReviewers: string[], nextApprovers: string[]) => { + if (!policyAvailable) return; onUpdate({ executionPolicy: buildExecutionPolicy({ existingPolicy: issue.executionPolicy ?? null, @@ -1073,13 +1078,13 @@ export function IssueProperties({ return `${stageLabel} pending${participantLabel ? ` with ${participantLabel}` : ""}`; })(); useEffect(() => { - setMonitorAtInput(toDateTimeLocalValue(issue.executionPolicy?.monitor?.nextCheckAt)); - setMonitorNotesInput(issue.executionPolicy?.monitor?.notes ?? ""); - setMonitorServiceInput(issue.executionPolicy?.monitor?.serviceName ?? ""); + setMonitorAtInput(toDateTimeLocalValue(executionPolicy?.monitor?.nextCheckAt)); + setMonitorNotesInput(executionPolicy?.monitor?.notes ?? ""); + setMonitorServiceInput(executionPolicy?.monitor?.serviceName ?? ""); }, [ - issue.executionPolicy?.monitor?.nextCheckAt, - issue.executionPolicy?.monitor?.notes, - issue.executionPolicy?.monitor?.serviceName, + executionPolicy?.monitor?.nextCheckAt, + executionPolicy?.monitor?.notes, + executionPolicy?.monitor?.serviceName, ]); // Re-sync watchdog editor inputs when the persisted watchdog changes (and reset on close). useEffect(() => { @@ -1261,6 +1266,7 @@ export function IssueProperties({ ? M | null : never : never) => { + if (!policyAvailable) return; const basePolicy = buildExecutionPolicy({ existingPolicy: issue.executionPolicy ?? null, reviewerValues, @@ -1270,8 +1276,10 @@ export function IssueProperties({ onUpdate({ executionPolicy: null }); return; } + const { monitor: _previousMonitor, ...policyWithoutMonitor } = basePolicy ?? {}; onUpdate({ executionPolicy: { + ...policyWithoutMonitor, mode: basePolicy?.mode ?? issue.executionPolicy?.mode ?? "normal", commentRequired: true, stages: basePolicy?.stages ?? [], @@ -1299,11 +1307,11 @@ export function IssueProperties({ setMonitorOpen(false); }; const monitorState = issue.executionState?.monitor ?? null; - const monitorNextCheckAt = monitorState?.nextCheckAt ?? issue.monitorNextCheckAt ?? issue.executionPolicy?.monitor?.nextCheckAt ?? null; + const monitorNextCheckAt = monitorState?.nextCheckAt ?? issue.monitorNextCheckAt ?? executionPolicy?.monitor?.nextCheckAt ?? null; const monitorAttemptCount = issue.monitorAttemptCount ?? monitorState?.attemptCount ?? 0; const monitorLastTriggeredAt = issue.monitorLastTriggeredAt ?? monitorState?.lastTriggeredAt ?? null; - const monitorServiceName = issue.executionPolicy?.monitor?.serviceName ?? monitorState?.serviceName ?? null; - const monitorNotes = issue.executionPolicy?.monitor?.notes ?? monitorState?.notes ?? null; + const monitorServiceName = executionPolicy?.monitor?.serviceName ?? monitorState?.serviceName ?? null; + const monitorNotes = executionPolicy?.monitor?.notes ?? monitorState?.notes ?? null; const monitorNow = useMonitorCountdown(monitorNextCheckAt); const monitorRelative = monitorNextCheckAt ? formatMonitorEta(monitorNextCheckAt, monitorNow) : null; const monitorIsDueNow = monitorRelative === "due now"; @@ -1585,7 +1593,7 @@ export function IssueProperties({ > Schedule - {issue.executionPolicy?.monitor ? ( + {executionPolicy?.monitor ? (