mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 01:50:13 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A plugin can provide Runner execution through an environment driver that declares `supportsTasks: true`. Agent computers use this path. > - Core let an agent without a managed AI connection run on that path. The run then used the account that was installed on the plugin's execution host. > - That is not a supported credential source. When the host Runner did not match, the run failed three times, 30 seconds apart, and then blocked the task with a generic error. The error did not name the fix. > - The plugin cannot refuse these runs itself. `describe` has no connection data, and Core replaces plugin error text before the user sees it. > - This pull request makes a managed AI connection a requirement for every plugin Runner-task environment. Core refuses a run without one before it acquires a lease or calls the plugin. > - The benefit is one clear notice that tells the operator to choose an AI connection, and no wasted retries or plugin work. ## Linked Issues or Issue Description Refs #19326. That pull request makes a Runner revision mismatch retry. It noted that runs without a managed AI connection reach the Runner check on a different path, which still blocks after 3 attempts. This pull request removes that path for plugin Runner-task environments. **Problem or motivation** A `paperclip_runner` agent with no managed AI connection can run in a plugin environment that provides Runner tasks. The run uses the account installed on the execution host. If the host Runner does not match, the native session retries 3 times and then blocks the task with "Plugin-provided Runner execution operation unavailable". The operator cannot tell what to fix. **Proposed solution** Refuse these runs in Core before any lease, plugin call, secret resolution, or provider work. Use one stable reason, `ai_connection_required`, for run admission, the setup probe, and agent verification. Show a task notice that names the fix. **Alternatives considered** A refusal in the plugin. It cannot see the connection at `describe`, which runs first. Its error text is also replaced twice before the user sees it. A new structured plugin error code would still not cover `describe`. ## What Changed - Add `server/src/services/plugin-runner-ai-connection.ts`. It holds the `ai_connection_required` code and message, the probe result, and the manifest lookup for a Runner-task driver. The lookup moves here from `agent-runner-selection.ts`, so run admission and runner selection use the same rule. - `heartbeat/execution-configuration.ts`: refuse a new `paperclip_runner` run in a Runner-task environment when it has no managed AI connection. This also applies when `homeMode: "preconfigured"` drops the connection. The run fails as `configuration_incomplete` with reason `ai_connection_required` and an action URL to the agent's runtime settings. The check runs before observability, secrets, the managed runtime, the workspace, and the lease. A resumed native run keeps its admitted contract. - `recovery/stranded-notice.ts`: add an "AI connection required" task notice. - `run-failure-report.ts`: treat this pre-dispatch refusal as an owner action, not an unexpected failure. - `adapters/registry.ts`: the unmanaged probe for a Runner-task target now returns a failed `ai_connection_required` check. It calls no task operation. Managed probes still go through `testManagedEnvironment`. - `agent-harness-verification.ts` and `agent-lifecycle/application/failure.ts`: classify the check as `ai_connection_required` with a fixed message. Verification waits for the configuration to change instead of retrying. - Docs: `PLUGIN_RUNNER_EXECUTION.md` states the requirement. `PLUGIN_ENVIRONMENT_TASKS.md` no longer mentions installed accounts. `agent-config-ui.md` notes the requirement. - Tests: the "plugin-owned account" heartbeat test described the behavior that this change removes. It now checks the assembled heartbeat refusal: no plugin call, no native boundary, no adapter, no workspace files. ## Verification - `npx vitest run` in `server/` for these files. All tests passed: - `src/__tests__/heartbeat-execution-configuration.test.ts`. New cases: refused without a connection, refused for a preconfigured home, managed passes, resumed native input passes, non-task plugin driver passes. The refused cases also check that observability, secrets, and the managed runtime did not start. - `src/__tests__/heartbeat-plugin-environment.test.ts`. One test, "inherits the instance default environment across companies…", also fails on unmodified `master` in my environment. This change does not touch it, because its driver does not declare `supportsTasks`. - `src/adapters/registry.test.ts`, `src/__tests__/agent-harness-verification.test.ts`, `src/modules/agent-lifecycle/__tests__/diagnostics.test.ts`, `src/services/recovery/stranded-notice.test.ts`, and `src/services/__tests__/run-failure-report.test.ts`. - 17 more suites that use plugin Runner-task environments, with 705 tests: `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `task-creation-recovery.integration`, `native-startup-cleanup-recovery`, `environment-task-authority.integration`, `plugin-task-credential-writeback`, agent-lifecycle `lifecycle`, `setup`, and `environment-plugins`, `workspace-preparation`, `agent-runner-selection`, `agent-harness-verification-provider-api`, `agent-harness-verification-remote`, `plugin-environment-driver-seam`, `environment-task-runtime`, `agent-test-environment-routes`, and `adapter-registry`. - `pnpm typecheck` in `server/` reported no errors in the changed files. Its only errors came from sandbox-provider packages that my local install did not link. - Not run: `plugin-runner-probe.test.ts`, because it needs a Cargo build. This change does not modify `probePluginRunnerEnvironment`. Also not run: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Behavior change: an agent in a plugin Runner-task environment with no managed AI connection now stops before dispatch. Before, it ran with the host's installed account. The agent's owner must choose an AI connection. This includes agents with `homeMode: "preconfigured"` in these environments. - Runs that were already admitted and resume from saved native input are not refused. They keep the earlier behavior until they finish. - The check reads the plugin manifest on each new run admission for plugin environments. This is the same lookup that runner selection already does. - This pull request and #19326 both edit `stranded-notice.ts` and `failure.ts`. The changes are additive, but whichever merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (file edits, shell, test runs). Default context window, extended thinking on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1496 lines
72 KiB
TypeScript
1496 lines
72 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import os from "node:os";
|
|
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { agents, applyPendingMigrations, authUsers, closeRegisteredClients, companies, companySecrets, createDb, dotAgentBindings,
|
|
dotRunnerAssignments, grokBotDispatches, grokBotRoutines, heartbeatRuns, issues, mcpOauthClients, mcpOauthGrants, nativeRunFinalizations,
|
|
type Db } from "@paperclipai/db";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "../../__tests__/helpers/embedded-postgres.js";
|
|
|
|
const mockCaptureRunFailure = vi.hoisted(() => vi.fn());
|
|
const mockRedactCurrentUserText = vi.hoisted(() => vi.fn());
|
|
const mockResolveSecret = vi.hoisted(() => vi.fn());
|
|
|
|
vi.mock("../../secrets/provider-registry.js", () => ({
|
|
getSecretProvider: () => ({ resolveVersion: mockResolveSecret }),
|
|
}));
|
|
|
|
vi.mock("../../sentry.js", () => ({
|
|
captureRunFailure: mockCaptureRunFailure,
|
|
}));
|
|
// Wrap the real function instead of a fake, so tests can assert the actual
|
|
// redacted output while still spying on the call. A fake output would hide
|
|
// whether the composed redaction in run-failure-report.ts is correct.
|
|
vi.mock("../../log-redaction.js", async (importOriginal) => {
|
|
const actual = await importOriginal<typeof import("../../log-redaction.js")>();
|
|
mockRedactCurrentUserText.mockImplementation(actual.redactCurrentUserText);
|
|
return { ...actual, redactCurrentUserText: mockRedactCurrentUserText };
|
|
});
|
|
|
|
import { isNewTerminalRunFailure, reportRunFailure, waitForPendingRunFailureReports } from "../run-failure-report.js";
|
|
import { controllerLeaseLossCancellation } from "../run-cancellation.js";
|
|
import { REDACTED_EVENT_VALUE } from "../../redaction.js";
|
|
|
|
const testDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
|
|
const embeddedPostgresSupport = testDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
describeEmbeddedPostgres("reportRunFailure", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let companyId!: string;
|
|
let agentId!: string;
|
|
let inheritedEnv: NodeJS.ProcessEnv;
|
|
|
|
beforeAll(async () => {
|
|
if (testDatabaseUrl) {
|
|
await applyPendingMigrations(testDatabaseUrl);
|
|
db = createDb(testDatabaseUrl);
|
|
} else {
|
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-failure-report-");
|
|
db = createDb(tempDb.connectionString);
|
|
}
|
|
}, 20_000);
|
|
|
|
beforeEach(() => {
|
|
// Unknown host values intentionally count as secrets, including short values
|
|
// such as "1". Keep these fixtures independent of the developer/CI environment
|
|
// and add secret values explicitly in the tests that exercise redaction.
|
|
inheritedEnv = process.env;
|
|
process.env = Object.fromEntries(
|
|
["PATH", "HOME", "USER", "USERNAME", "LOGNAME", "USERPROFILE", "TMPDIR", "TEMP", "TMP"]
|
|
.flatMap((key) => inheritedEnv[key] === undefined ? [] : [[key, inheritedEnv[key]]]),
|
|
);
|
|
process.env.PAPERCLIP_SENTRY_RUN_FAILURES = "unexpected";
|
|
});
|
|
|
|
afterEach(async () => {
|
|
vi.clearAllMocks();
|
|
vi.unstubAllEnvs();
|
|
process.env = inheritedEnv;
|
|
await db.delete(grokBotRoutines);
|
|
await db.delete(companySecrets);
|
|
await db.delete(nativeRunFinalizations);
|
|
await db.delete(heartbeatRuns);
|
|
await db.delete(issues);
|
|
await db.delete(agents);
|
|
await db.delete(companies);
|
|
await db.delete(mcpOauthClients);
|
|
await db.delete(authUsers);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
if (testDatabaseUrl) await closeRegisteredClients(testDatabaseUrl);
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function seedCompanyAndAgent(agentOverrides: Partial<typeof agents.$inferInsert> = {}) {
|
|
companyId = randomUUID();
|
|
agentId = randomUUID();
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Paperclip",
|
|
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
|
requireBoardApprovalForNewAgents: false,
|
|
defaultResponsibleUserId: "responsible-user",
|
|
});
|
|
await db.insert(agents).values({
|
|
id: agentId,
|
|
companyId,
|
|
name: "CodexCoder",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {},
|
|
permissions: {},
|
|
...agentOverrides,
|
|
});
|
|
return { companyId, agentId };
|
|
}
|
|
|
|
function buildRun(
|
|
overrides: Partial<typeof heartbeatRuns.$inferSelect> = {},
|
|
): typeof heartbeatRuns.$inferSelect {
|
|
return {
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
status: "failed",
|
|
error: "the provider process exited with code 1",
|
|
errorCode: "process_lost",
|
|
nativeIssueId: randomUUID(),
|
|
contextSnapshot: null,
|
|
...overrides,
|
|
} as unknown as typeof heartbeatRuns.$inferSelect;
|
|
}
|
|
|
|
it.each([undefined, "", "off", "true", "invalid"])("does no reporting work without an explicit valid mode: %s", async mode => {
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", mode);
|
|
if (mode === undefined) delete process.env.PAPERCLIP_SENTRY_RUN_FAILURES;
|
|
const select = vi.fn();
|
|
await reportRunFailure({ select } as unknown as Db, buildRun());
|
|
expect(select).not.toHaveBeenCalled();
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["quota", "git", "secret", "selection", "workspace", "hermes"])("reports known %s failures in all mode", async kind => {
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
await seedCompanyAndAgent(kind === "hermes" ? { adapterType: "hermes_gateway" } : {});
|
|
const fixtures = {
|
|
quota: providerQuotaRun,
|
|
git: gitConnectionRun,
|
|
secret: missingSecretRun,
|
|
selection: aiSelectionRun,
|
|
workspace: localPathConfigurationRun,
|
|
hermes: hermesConnectionRun,
|
|
};
|
|
const fixture = fixtures[kind as keyof typeof fixtures]();
|
|
const phase = ["git", "secret", "selection", "workspace"].includes(kind) ? "setup" : "execute";
|
|
const before = structuredClone(fixture);
|
|
await reportRunFailure(db, fixture, { phase });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
companyId, agentId, runId: fixture.id, runStatus: "failed", errorCode: fixture.errorCode,
|
|
}));
|
|
expect(fixture).toEqual(before);
|
|
});
|
|
|
|
it("reports company-scoped runs that have no task", async () => {
|
|
await seedCompanyAndAgent();
|
|
const fixture = buildRun({ nativeIssueId: null, contextSnapshot: null });
|
|
await reportRunFailure(db, fixture);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
companyId, agentId, taskId: null, runId: fixture.id,
|
|
}));
|
|
});
|
|
|
|
it.each(["succeeded", "queued", "running", "interrupted", "cancelled"])("all mode excludes %s runs and expected operator stops", async status => {
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
const select = vi.fn();
|
|
await reportRunFailure({ select } as unknown as Db, buildRun({ status, startedAt: new Date(), resultJson: {
|
|
cancellation: { source: "operator", expected: true, initiator: { type: "user" }, reason: "Stopped", recordedAt: new Date().toISOString() },
|
|
} }));
|
|
expect(select).not.toHaveBeenCalled();
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["fresh", "resume", "provider_started"] as const)("keeps workspace base-ref observations reportable: %s", async mode => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ errorCode: "configuration_incomplete", executionStage: "preparing", exitCode: null, signal: null,
|
|
runtimeMode: mode === "resume" ? "native" : "legacy", runnerProfileJson: mode === "resume" ? { nativeExecutionInput: {} } : null,
|
|
resultJson: { configurationIncomplete: { reason: "workspace_base_ref_unresolved", baseRefDiagnostic: {
|
|
schemaVersion: 1, remoteLookup: "resolved", authLookup: "resolved", fetch: "failed", fetchExitCode: 128,
|
|
fetchFailureKind: "remote_ref_not_found", refResolution: "failed", refExitCode: 128,
|
|
} }, executionRecovery: { kind: "bootstrap", providerWorkStarted: mode === "provider_started" } },
|
|
});
|
|
const before = structuredClone(run);
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ diagnostics: expect.objectContaining({
|
|
execution: expect.objectContaining({ workspaceBaseRefFetch: "failed", workspaceBaseRefFetchFailureKind: "remote_ref_not_found" }),
|
|
}) }));
|
|
expect(run).toEqual(before);
|
|
});
|
|
|
|
const gitConnection = { schemaVersion: 1, provider: "git", operation: "clone", reason: "authentication_failed" };
|
|
const hermesConnection = { schemaVersion: 1, provider: "hermes_gateway", operation: "create_run", reason: "connection_refused" };
|
|
|
|
function gitConnectionRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "setup_failed", executionStage: "preparing", exitCode: null, signal: null,
|
|
resultJson: { connectionFailure: gitConnection,
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
function hermesConnectionRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "hermes_gateway_connect_failed", executionStage: "executing", exitCode: 1, signal: null,
|
|
resultJson: { connectionFailure: hermesConnection }, ...overrides,
|
|
});
|
|
}
|
|
|
|
it("keeps a proven Git clone failure and its persisted task error unchanged", async () => {
|
|
await seedCompanyAndAgent();
|
|
const input = gitConnectionRun();
|
|
const [run] = await db.insert(heartbeatRuns).values(input).returning();
|
|
const before = structuredClone(run);
|
|
await reportRunFailure(db, run!, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(run).toEqual(before);
|
|
expect(await db.select().from(heartbeatRuns)).toEqual([before]);
|
|
});
|
|
|
|
it("keeps an all-external Git materialization aggregate local even across different external causes", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = gitConnectionRun({ errorCode: "workspace_validation_failed" });
|
|
run.resultJson!.workspaceValidation = { reason: "git_worktree_base_materialization_failed", materializationFailures: [
|
|
{ connectionFailure: gitConnection },
|
|
{ connectionFailure: { ...gitConnection, reason: "repository_unavailable" } },
|
|
] };
|
|
const before = structuredClone(run);
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(run).toEqual(before);
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { executionStage: "executing" }, { executionStage: null },
|
|
{ errorCode: "adapter_failed" }, { exitCode: 1 }, { exitCode: 0 }, { signal: "SIGTERM" },
|
|
{ resultJson: {} },
|
|
{ resultJson: { connectionFailure: { ...gitConnection, schemaVersion: 2 }, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } },
|
|
{ resultJson: { connectionFailure: hermesConnection, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } },
|
|
])("reports a Git marker without its matching setup outcome: %j", async (overrides) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, gitConnectionRun(overrides), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([undefined, "execute"] as const)("reports Git failures outside setup: %s", async (phase) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, gitConnectionRun(), { phase });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([undefined, { kind: "bootstrap" }, { kind: "bootstrap", providerWorkStarted: true },
|
|
{ kind: "runtime", providerWorkStarted: false }])("reports Git failures without unstarted bootstrap proof: %j", async (recovery) => {
|
|
await seedCompanyAndAgent();
|
|
const run = gitConnectionRun();
|
|
run.resultJson!.executionRecovery = recovery;
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([
|
|
[], [null], [{ connectionFailure: gitConnection }, {}],
|
|
[{ connectionFailure: gitConnection }, { connectionFailure: hermesConnection }],
|
|
[{ connectionFailure: gitConnection }, { connectionFailure: { ...gitConnection, reason: "unknown" } }],
|
|
[{ connectionFailure: { ...gitConnection, reason: "repository_unavailable" } }],
|
|
].map((failures) => ({ failures })))("reports mixed, malformed, or inconsistent Git materialization evidence: %j", async ({ failures }) => {
|
|
await seedCompanyAndAgent();
|
|
const run = gitConnectionRun({ errorCode: "workspace_validation_failed" });
|
|
run.resultJson!.workspaceValidation = { reason: "git_worktree_base_materialization_failed", materializationFailures: failures };
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([
|
|
{ reason: "git_worktree_base_materialization_failed", materializationFailures: [{ connectionFailure: gitConnection }] },
|
|
{ reason: "git_worktree_not_reusable" },
|
|
])("does not use setup_failed to bypass separate workspace validation evidence: %j", async (validation) => {
|
|
await seedCompanyAndAgent();
|
|
const run = gitConnectionRun();
|
|
run.resultJson!.workspaceValidation = validation;
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([
|
|
["configuration", "endpoint_missing", "hermes_gateway_api_base_url_missing"],
|
|
["configuration", "endpoint_invalid", "hermes_gateway_api_base_url_invalid"],
|
|
["configuration", "insecure_transport", "hermes_gateway_plain_http_remote_denied"],
|
|
["configuration", "credentials_missing", "hermes_gateway_api_key_missing"],
|
|
["create_run", "authentication_failed", "hermes_gateway_auth_failed"],
|
|
["create_run", "endpoint_not_found", "hermes_gateway_runs_unsupported"],
|
|
["create_run", "rate_limited", "hermes_gateway_rate_limited"],
|
|
["create_run", "remote_unavailable", "hermes_gateway_upstream_error"],
|
|
...["connection_refused", "dns_failure", "network_unreachable", "connection_reset", "connection_timeout", "tls_failure"]
|
|
.flatMap((reason) => ["hermes_gateway_connect_failed", "hermes_gateway_protocol_error"].map((code) => ["create_run", reason, code])),
|
|
])("keeps the matching Hermes %s/%s task error local", async (operation, reason, errorCode) => {
|
|
await seedCompanyAndAgent({ adapterType: "hermes_gateway" });
|
|
const run = hermesConnectionRun({ errorCode, resultJson: { connectionFailure: { ...hermesConnection, operation, reason } } });
|
|
const before = structuredClone(run);
|
|
await reportRunFailure(db, run, { phase: "execute" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(run).toEqual(before);
|
|
});
|
|
|
|
it.each([undefined, { kind: "bootstrap", providerWorkStarted: true }, { kind: "runtime", providerWorkStarted: true }])(
|
|
"does not infer remote nonacceptance from a classified Hermes connection failure: %j", async (recovery) => {
|
|
await seedCompanyAndAgent({ adapterType: "hermes_gateway" });
|
|
const input = hermesConnectionRun();
|
|
input.resultJson!.executionRecovery = recovery;
|
|
const [run] = await db.insert(heartbeatRuns).values(input).returning();
|
|
const before = structuredClone(run);
|
|
await reportRunFailure(db, run!);
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(await db.select().from(heartbeatRuns)).toEqual([before]);
|
|
},
|
|
);
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { status: "cancelled", startedAt: new Date(0) },
|
|
{ exitCode: 0 }, { exitCode: null }, { signal: "SIGTERM" },
|
|
{ errorCode: "hermes_gateway_timeout" }, { errorCode: "hermes_gateway_run_failed" },
|
|
{ errorCode: "hermes_gateway_auth_failed" },
|
|
{ resultJson: {} },
|
|
{ resultJson: { connectionFailure: { ...hermesConnection, reason: "unknown" } } },
|
|
{ resultJson: { connectionFailure: { ...hermesConnection, privateUrl: "https://private.example.test" } } },
|
|
{ resultJson: { connectionFailure: gitConnection } },
|
|
{ resultJson: { connectionFailure: { ...hermesConnection, operation: "configuration", reason: "credentials_missing" } } },
|
|
])("reports an unproven or mismatched Hermes outcome: %j", async (overrides) => {
|
|
await seedCompanyAndAgent({ adapterType: "hermes_gateway" });
|
|
const run = hermesConnectionRun(overrides);
|
|
const before = structuredClone(run);
|
|
await reportRunFailure(db, run, { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
expect(run).toEqual(before);
|
|
});
|
|
|
|
it("keeps setup errors reportable even when they carry a matching Hermes marker", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "hermes_gateway" });
|
|
await reportRunFailure(db, hermesConnectionRun(), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each(["codex_local", "hermes_local"])("requires the actual database adapter, not a Hermes marker on %s", async (adapterType) => {
|
|
await seedCompanyAndAgent({ adapterType });
|
|
const run = hermesConnectionRun({ contextSnapshot: { adapterType: "hermes_gateway" } });
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ agentAdapter: adapterType }));
|
|
});
|
|
|
|
it.each(["missing", "other_company"])("does not suppress Hermes when its database agent is %s", async (kind) => {
|
|
await seedCompanyAndAgent({ adapterType: "hermes_gateway" });
|
|
const run = hermesConnectionRun(kind === "missing" ? { agentId: randomUUID() } : { companyId: randomUUID() });
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ agentAdapter: "unknown" }));
|
|
});
|
|
|
|
function providerQuotaRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
error: "ACP agent reported a terminal limit failure.\nYou've hit your session limit · resets 2am (UTC)",
|
|
errorCode: "provider_quota", executionStage: "dispatching", exitCode: 1, signal: null,
|
|
resultJson: { errorFamily: "provider_quota", stopReason: "adapter_failed" }, ...overrides,
|
|
});
|
|
}
|
|
|
|
it.each(["claude_local", "codex_local"])("keeps a classified %s provider quota wait local", async (adapterType) => {
|
|
await seedCompanyAndAgent({ adapterType });
|
|
const [run] = await db.insert(heartbeatRuns).values(providerQuotaRun()).returning();
|
|
const before = structuredClone(run);
|
|
for (const phase of [undefined, "execute"] as const) await reportRunFailure(db, run!, { phase });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(await db.select().from(heartbeatRuns)).toEqual([before]);
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { status: "cancelled", startedAt: new Date(0) }, { signal: "SIGTERM" },
|
|
{ errorCode: "adapter_failed" }, { errorCode: "claude_transient_upstream" },
|
|
{ resultJson: {} }, { resultJson: null }, { resultJson: { errorFamily: "transient_upstream" } },
|
|
])("reports a provider quota outcome without matching adapter evidence: %j", async (overrides) => {
|
|
await seedCompanyAndAgent({ adapterType: "claude_local" });
|
|
await reportRunFailure(db, providerQuotaRun(overrides), { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("keeps a provider quota code reportable during setup", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "claude_local" });
|
|
await reportRunFailure(db, providerQuotaRun(), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
function codexModelCapacityRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
error: "ACP agent reported a terminal service failure.\nSelected model is at capacity. Please try a different model.",
|
|
errorCode: "codex_transient_upstream", executionStage: "dispatching", exitCode: 1, signal: null,
|
|
resultJson: {
|
|
errorFamily: "transient_upstream", stopReason: "adapter_failed",
|
|
terminalSessionFailure: { category: "service", title: "Selected model is at capacity. Please try a different model." },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it("keeps a classified Codex ACP model capacity retry local", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "codex_local" });
|
|
for (const phase of [undefined, "execute"] as const) await reportRunFailure(db, codexModelCapacityRun(), { phase });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { signal: "SIGTERM" }, { errorCode: "acpx_turn_failed" },
|
|
{ resultJson: { errorFamily: "transient_upstream" } },
|
|
{ resultJson: { errorFamily: "provider_quota", terminalSessionFailure: { category: "service" } } },
|
|
{ resultJson: { errorFamily: "transient_upstream", terminalSessionFailure: { category: "connection" } } },
|
|
])("reports a Codex transient outcome without typed capacity evidence: %j", async (overrides) => {
|
|
await seedCompanyAndAgent({ adapterType: "codex_local" });
|
|
await reportRunFailure(db, codexModelCapacityRun(overrides), { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("reports a Codex ACP model capacity failure during setup or in all mode", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "codex_local" });
|
|
await reportRunFailure(db, codexModelCapacityRun(), { phase: "setup" });
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
await reportRunFailure(db, codexModelCapacityRun(), { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
/** The persisted shape after the native finalizer commits a capacity decision. */
|
|
async function nativeCapacityFailure(input: {
|
|
run?: Partial<typeof heartbeatRuns.$inferSelect>;
|
|
resultJson?: Record<string, unknown>;
|
|
retry?: Partial<typeof heartbeatRuns.$inferInsert> | null;
|
|
} = {}) {
|
|
await seedCompanyAndAgent({ adapterType: "paperclip_runner" });
|
|
const run = buildRun({
|
|
error: "Selected model is at capacity. Please try a different model.",
|
|
errorCode: "native_provider_overloaded", runtimeMode: "native", nativePhase: "committed",
|
|
executionStage: "preparing", exitCode: null, signal: null,
|
|
resultJson: {
|
|
nativeProviderFailure: { provider: "codex", category: "server_overloaded" },
|
|
finalizationPhase: "committed", finalizationReasonCode: "native_provider_overloaded",
|
|
...input.resultJson,
|
|
},
|
|
...input.run,
|
|
});
|
|
const [stored] = await db.insert(heartbeatRuns).values(run).returning();
|
|
if (input.retry !== null) {
|
|
await db.insert(heartbeatRuns).values({
|
|
companyId, agentId, invocationSource: "automation", triggerDetail: "system", status: "scheduled_retry",
|
|
retryOfRunId: run.id, scheduledRetryReason: "native_provider_overloaded", scheduledRetryAttempt: 1,
|
|
scheduledRetryAt: new Date(Date.now() + 60_000), errorCode: "native_provider_overloaded",
|
|
contextSnapshot: { issueId: run.nativeIssueId, retryOfRunId: run.id }, ...input.retry,
|
|
});
|
|
}
|
|
return stored!;
|
|
}
|
|
|
|
it("keeps a native model capacity failure local once its automatic retry is scheduled", async () => {
|
|
const run = await nativeCapacityFailure();
|
|
await reportRunFailure(db, run);
|
|
await reportRunFailure(db, run, { phase: "execute" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
["no finalization fields", { finalizationPhase: undefined, finalizationReasonCode: undefined }],
|
|
["an earlier finalization phase", { finalizationPhase: "retryable_failure", finalizationReasonCode: undefined }],
|
|
["only the adapter's resultJson", { nativeProviderFailure: undefined, finalizationPhase: undefined, finalizationReasonCode: undefined }],
|
|
] as const)("keeps a scheduled native capacity retry local when the run row has %s", async (_label, resultJson) => {
|
|
// The live heartbeat's failed-status write replaces the finalizer's
|
|
// resultJson with the adapter's copy. The successor created by the
|
|
// committed decision still proves the scheduled retry.
|
|
const run = await nativeCapacityFailure({ resultJson });
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
["the retries are exhausted", { resultJson: { finalizationReasonCode: "native_provider_overloaded_exhausted" }, retry: null }],
|
|
["the task holds the retry", { resultJson: { finalizationReasonCode: "native_provider_overloaded_held" }, retry: null }],
|
|
["no retry run exists", { retry: null }],
|
|
["the successor belongs to another retry lane", { retry: { scheduledRetryReason: "native_safe_replacement" } }],
|
|
["the successor belongs to another task", { retry: { contextSnapshot: { issueId: randomUUID() } } }],
|
|
["the run has no native task", { run: { nativeIssueId: null } }],
|
|
["finalization has no fields and no retry run exists", { resultJson: { finalizationPhase: undefined, finalizationReasonCode: undefined }, retry: null }],
|
|
["the diagnostic is missing and no retry run exists", { resultJson: { nativeProviderFailure: undefined }, retry: null }],
|
|
["the run is legacy", { run: { runtimeMode: "legacy" } }],
|
|
["the run has a signal", { run: { signal: "SIGTERM" } }],
|
|
["the run timed out", { run: { status: "timed_out" } }],
|
|
] as const)("reports a native capacity failure when %s", async (_label, input) => {
|
|
const run = await nativeCapacityFailure(input as Parameters<typeof nativeCapacityFailure>[0]);
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("reports a native capacity retry during setup, in all mode, or when its retry cannot be read", async () => {
|
|
const run = await nativeCapacityFailure();
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
const select = db.select.bind(db);
|
|
// The agent lookup succeeds; the retry read fails.
|
|
const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select)
|
|
.mockImplementation(() => { throw new Error("fixture retry read failed"); });
|
|
try {
|
|
await reportRunFailure(db, run);
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3);
|
|
});
|
|
|
|
function providerAccessRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
error: "ACP agent reported a terminal access failure.\nInvalid API key · Fix external API key",
|
|
errorCode: "claude_auth_required", executionStage: "dispatching", exitCode: 1, signal: null,
|
|
resultJson: {
|
|
stopReason: "adapter_failed",
|
|
terminalSessionFailure: { category: "access", title: "Invalid API key · Fix external API key" },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it.each(["claude_auth_required", "acpx_auth_required"])("keeps a typed ACP provider access failure (%s) local", async (errorCode) => {
|
|
await seedCompanyAndAgent({ adapterType: "claude_local" });
|
|
for (const phase of [undefined, "execute"] as const) await reportRunFailure(db, providerAccessRun({ errorCode }), { phase });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { signal: "SIGTERM" }, { errorCode: "acpx_turn_failed" }, { errorCode: "adapter_failed" },
|
|
{ resultJson: {} }, { resultJson: null }, { resultJson: { terminalSessionFailure: { category: "service" } } },
|
|
{ resultJson: { terminalSessionFailure: "access" } },
|
|
])("reports a provider auth outcome without typed ACP access evidence: %j", async (overrides) => {
|
|
await seedCompanyAndAgent({ adapterType: "claude_local" });
|
|
await reportRunFailure(db, providerAccessRun(overrides), { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("reports a typed ACP provider access failure during setup or in all mode", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "claude_local" });
|
|
await reportRunFailure(db, providerAccessRun(), { phase: "setup" });
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
await reportRunFailure(db, providerAccessRun(), { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
function restartInterruptedSetupRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
error: "Process lost -- server may have restarted; retrying once",
|
|
runtimeMode: "legacy", executionStage: "preparing", processPid: null, processGroupId: null,
|
|
resultJson: {
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
processLossDiagnostic: {
|
|
pidRecorded: false, groupRecorded: false, localCheck: "no_identifiers",
|
|
retryEligible: true, runPredatesObserver: true,
|
|
},
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it("keeps a retried restart interruption of legacy setup local", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "paperclip_runner" });
|
|
await reportRunFailure(db, restartInterruptedSetupRun());
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("reports a retried restart interruption of legacy setup in all mode", async () => {
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
await seedCompanyAndAgent({ adapterType: "paperclip_runner" });
|
|
await reportRunFailure(db, restartInterruptedSetupRun());
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([
|
|
{ executionStage: "dispatching" }, { runtimeMode: "native" }, { processPid: 4242 },
|
|
{ errorCode: "adapter_failed" },
|
|
{ resultJson: { processLossDiagnostic: { localCheck: "no_identifiers", retryEligible: true, runPredatesObserver: true } } },
|
|
{ resultJson: {
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
processLossDiagnostic: { localCheck: "no_identifiers", retryEligible: false, runPredatesObserver: true },
|
|
} },
|
|
{ resultJson: {
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
processLossDiagnostic: { localCheck: "no_identifiers", retryEligible: true, runPredatesObserver: false },
|
|
} },
|
|
])("reports process loss without restart-before-dispatch evidence: %j", async (overrides) => {
|
|
await seedCompanyAndAgent({ adapterType: "paperclip_runner" });
|
|
await reportRunFailure(db, restartInterruptedSetupRun(overrides));
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
async function expiredGrokOffer(input: {
|
|
wakes: Array<{ status: "rejected" | "accepted" | "acknowledged" | "unknown"; httpStatus: number | null }>;
|
|
acceptedAt?: Date | null;
|
|
originalFailureCode?: string;
|
|
finalization?: boolean;
|
|
offers?: number;
|
|
}) {
|
|
await seedCompanyAndAgent({ adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot" } });
|
|
const userId = randomUUID();
|
|
await db.insert(authUsers).values({ id: userId, name: "Operator", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(mcpOauthClients).values({ id: userId, name: "Grok", redirectUris: ["https://client.example/callback"] });
|
|
const [grant] = await db.insert(mcpOauthGrants).values({ companyId, userId, clientId: userId, resource: "https://paperclip.example/mcp/runner",
|
|
purpose: "agent", agentId, scopes: ["paperclip:agent"] }).returning();
|
|
const [binding] = await db.insert(dotAgentBindings).values({ companyId, agentId, operatorId: userId, grantId: grant!.id,
|
|
providerKind: "grok_bot", status: "ready" }).returning();
|
|
const [secret] = await db.insert(companySecrets).values({ companyId, key: `grok-${binding!.id}`, name: "Grok routine" }).returning();
|
|
const [routine] = await db.insert(grokBotRoutines).values({ companyId, agentId, bindingId: binding!.id, bindingGeneration: 1,
|
|
botName: "Grok", secretId: secret!.id, secretVersion: 1 }).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Grok task", assigneeAgentId: agentId, status: "in_progress" }).returning();
|
|
const run = buildRun({ errorCode: "native_provider_terminal_failed", error: "Dot assignment acceptance or execution authority expired.",
|
|
nativeIssueId: issue!.id, runtimeMode: "native", executionStage: "preparing", nativePhase: "terminal_failure", exitCode: null, signal: null });
|
|
await db.insert(heartbeatRuns).values(run);
|
|
if (input.finalization !== false) {
|
|
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId, issueId: run.nativeIssueId!, phase: "terminal_failure",
|
|
failureCode: "native_provider_terminal_failed", failureDetail: { originalFailureCode: input.originalFailureCode ?? "dot_assignment_expired" } });
|
|
}
|
|
const offers = await db.insert(dotRunnerAssignments).values(Array.from({ length: input.offers ?? 1 }, () => ({ companyId, agentId,
|
|
bindingId: binding!.id, bindingGeneration: 1, runId: run.id, normalizedSessionId: randomUUID(), turnId: randomUUID(), controllerGeneration: 1,
|
|
catalogDigest: "fixture", projection: {}, status: "fenced" as const, acceptBy: new Date(Date.now() - 1000),
|
|
expiresAt: new Date(Date.now() + 3_600_000), acceptedAt: input.acceptedAt ?? null }))).returning();
|
|
const assignment = offers[0];
|
|
for (const wake of input.wakes) {
|
|
await db.insert(grokBotDispatches).values({ companyId, routineId: routine!.id, bindingId: binding!.id, bindingGeneration: 1,
|
|
grantId: grant!.id, routineRevision: 1, assignmentId: assignment!.id, sourceEventId: randomUUID(), kind: "mailbox_updated",
|
|
payload: {}, status: wake.status, httpStatus: wake.httpStatus, expiresAt: new Date(Date.now() + 600_000) });
|
|
}
|
|
return run;
|
|
}
|
|
|
|
it("keeps a Grok offer local when the endpoint rejected every work wake with HTTP 4xx", async () => {
|
|
const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: 400 }] });
|
|
await reportRunFailure(db, run, { phase: "execute" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("reports a rejected Grok wake in all mode", async () => {
|
|
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
|
const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }] });
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
companyId, agentId, runId: run.id, errorCode: "native_provider_terminal_failed",
|
|
}));
|
|
});
|
|
|
|
it.each([
|
|
["no wake was sent", { wakes: [] }],
|
|
["a wake was rejected locally without HTTP evidence", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: null }] }],
|
|
["the endpoint accepted a wake", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "acknowledged", httpStatus: 200 }] }],
|
|
["delivery is uncertain", { wakes: [{ status: "unknown", httpStatus: 503 }] }],
|
|
["the bot accepted the work", { wakes: [{ status: "rejected", httpStatus: 400 }], acceptedAt: new Date() }],
|
|
["the run failed for another provider reason", { wakes: [{ status: "rejected", httpStatus: 400 }], originalFailureCode: "provider_failed" }],
|
|
["the run has no finalization", { wakes: [{ status: "rejected", httpStatus: 400 }], finalization: false }],
|
|
["the run has more than one offer", { wakes: [{ status: "rejected", httpStatus: 400 }], offers: 2 }],
|
|
] as const)("keeps an expired Grok offer reportable when %s", async (_label, input) => {
|
|
const run = await expiredGrokOffer({ ...input, wakes: [...input.wakes] });
|
|
await reportRunFailure(db, run, { phase: "execute" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("keeps an expired Grok offer reportable when its receipts cannot be read", async () => {
|
|
const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }] });
|
|
const select = db.select.bind(db);
|
|
// The agent lookup succeeds; the receipt reads fail.
|
|
const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select)
|
|
.mockImplementation(() => { throw new Error("fixture receipt read failed"); });
|
|
try {
|
|
await reportRunFailure(db, run, { phase: "execute" });
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
function aiSelectionRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "configuration_incomplete",
|
|
executionStage: "preparing",
|
|
resultJson: {
|
|
configurationIncomplete: {
|
|
reason: "ai_connection_unavailable",
|
|
selectionFailure: "ai_connection_default_missing",
|
|
},
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it.each([
|
|
"ai_connection_responsible_user_missing", "ai_connection_default_missing",
|
|
"ai_connection_missing", "ai_connection_incompatible", "ai_connection_unavailable",
|
|
"ai_connection_credential_not_shared",
|
|
])("keeps the proven AI selection blocker local: %s", async (selectionFailure) => {
|
|
await seedCompanyAndAgent();
|
|
const run = aiSelectionRun();
|
|
(run.resultJson!.configurationIncomplete as Record<string, unknown>).selectionFailure = selectionFailure;
|
|
const saved = structuredClone(run);
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(run).toEqual(saved);
|
|
});
|
|
|
|
it.each([undefined, null, {}, [], "", "database_error", "ai_connection_busy"])(
|
|
"reports AI connection wrappers without a known selection cause: %j", async (selectionFailure) => {
|
|
await seedCompanyAndAgent();
|
|
const run = aiSelectionRun();
|
|
(run.resultJson!.configurationIncomplete as Record<string, unknown>).selectionFailure = selectionFailure;
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
|
|
it.each(["resume", "provider_started", "outside_setup", "unmarked"] as const)(
|
|
"reports credential sharing denials without fresh pre-provider proof: %s", async (cause) => {
|
|
await seedCompanyAndAgent();
|
|
const run = aiSelectionRun();
|
|
(run.resultJson!.configurationIncomplete as Record<string, unknown>).selectionFailure = "ai_connection_credential_not_shared";
|
|
run.error = "This credential is not shared with the responsible user";
|
|
if (cause === "resume") {
|
|
run.runtimeMode = "native";
|
|
run.runnerProfileJson = { nativeExecutionInput: {} };
|
|
}
|
|
if (cause === "provider_started") run.resultJson!.executionRecovery = { kind: "bootstrap", providerWorkStarted: true };
|
|
if (cause === "unmarked") delete (run.resultJson!.configurationIncomplete as Record<string, unknown>).selectionFailure;
|
|
await reportRunFailure(db, run, { phase: cause === "outside_setup" ? "execute" : "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
|
|
it.each([
|
|
{ runtimeMode: "native", runnerProfileJson: { nativeExecutionInput: {} } },
|
|
{ status: "timed_out" }, { errorCode: "setup_failed" }, { executionStage: "executing" },
|
|
{ executionStage: null }, { exitCode: 1 }, { exitCode: 0 }, { signal: "SIGTERM" },
|
|
{ resultJson: { configurationIncomplete: { reason: "ai_connection_unavailable", selectionFailure: "ai_connection_default_missing" } } },
|
|
{ resultJson: { configurationIncomplete: { reason: "ai_connection_unavailable", selectionFailure: "ai_connection_default_missing" }, executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } },
|
|
{ resultJson: { configurationIncomplete: { reason: "ai_connection_unavailable", selectionFailure: "ai_connection_default_missing" }, executionRecovery: { kind: "resume", providerWorkStarted: false } } },
|
|
{ resultJson: { configurationIncomplete: { reason: "ai_connection_unavailable", selectionFailure: "ai_connection_default_missing" }, executionRecovery: { kind: "bootstrap" } } },
|
|
{ resultJson: { configurationIncomplete: { reason: "controller_environment_unsupported", selectionFailure: "ai_connection_default_missing" }, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } },
|
|
])("reports AI configuration without pre-provider selection proof: %j", async (overrides) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, aiSelectionRun(overrides), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([undefined, "execute"] as const)("reports AI selection failures outside setup: %s", async (phase) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, aiSelectionRun(), { phase });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
function missingSecretRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "configuration_incomplete",
|
|
executionStage: "preparing",
|
|
resultJson: {
|
|
configurationIncomplete: {
|
|
reason: "secret_binding_missing",
|
|
missingBindings: [{ bindingType: "user_secret_ref", errorCode: "user_secret_missing" }],
|
|
},
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it.each([
|
|
{ bindingType: "secret_ref" },
|
|
{ bindingType: "secret_ref", errorCode: "binding_missing" },
|
|
...["binding_missing", "responsible_user_missing", "user_secret_missing", "secret_inactive", "user_secret_definition_inactive"]
|
|
.map((errorCode) => ({ bindingType: "user_secret_ref", errorCode })),
|
|
])("keeps the known pre-dispatch secret blocker local: %j", async (binding) => {
|
|
await seedCompanyAndAgent();
|
|
const run = missingSecretRun();
|
|
(run.resultJson!.configurationIncomplete as Record<string, unknown>).missingBindings = [binding];
|
|
const saved = structuredClone(run);
|
|
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(run).toEqual(saved);
|
|
});
|
|
|
|
it.each([
|
|
{},
|
|
{ reason: "secret_binding_missing", missingBindings: [] },
|
|
{ reason: "secret_binding_missing", missingBindings: [null] },
|
|
{ reason: "secret_binding_missing", missingBindings: [{ errorCode: "user_secret_missing" }] },
|
|
{ reason: "secret_binding_missing", missingBindings: [{ bindingType: "user_secret_ref", errorCode: "provider_error" }] },
|
|
{ reason: "secret_binding_missing", missingBindings: [{ bindingType: "user_secret_ref", errorCode: "user_secret_definition_missing" }] },
|
|
{ reason: "secret_binding_missing", missingBindings: [{ bindingType: "user_secret_ref", errorCode: "new_unknown_reason" }] },
|
|
{ reason: "secret_binding_missing", missingBindings: [
|
|
{ bindingType: "user_secret_ref", errorCode: "user_secret_missing" },
|
|
{ bindingType: "secret_ref", errorCode: "provider_error" },
|
|
] },
|
|
{ reason: "ai_connection_unavailable" },
|
|
])("reports ambiguous, unknown, and provider failures: %j", async (configurationIncomplete) => {
|
|
await seedCompanyAndAgent();
|
|
const run = missingSecretRun();
|
|
run.resultJson!.configurationIncomplete = configurationIncomplete;
|
|
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" },
|
|
{ errorCode: "setup_failed" },
|
|
{ errorCode: "workspace_validation_failed" },
|
|
{ errorCode: "adapter_failed" },
|
|
{ executionStage: "executing" },
|
|
{ executionStage: null },
|
|
{ exitCode: 1 },
|
|
{ exitCode: 0 },
|
|
{ signal: "SIGTERM" },
|
|
])("reports missing-secret text without a proven pre-dispatch outcome: %j", async (overrides) => {
|
|
await seedCompanyAndAgent();
|
|
|
|
await reportRunFailure(db, missingSecretRun(overrides), { phase: "setup" });
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([undefined, { kind: "bootstrap" }, { kind: "bootstrap", providerWorkStarted: true }])(
|
|
"reports missing-secret failures without explicit bootstrap proof: %j", async (executionRecovery) => {
|
|
await seedCompanyAndAgent();
|
|
const run = missingSecretRun();
|
|
run.resultJson!.executionRecovery = executionRecovery;
|
|
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
|
|
it.each([undefined, "execute"] as const)("reports configuration failures outside setup: %s", async (phase) => {
|
|
await seedCompanyAndAgent();
|
|
|
|
await reportRunFailure(db, missingSecretRun(), { phase });
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("captures once for the status failed", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "failed" });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
function restrictedWorkspaceSourceRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "configuration_incomplete",
|
|
executionStage: "preparing",
|
|
resultJson: {
|
|
configurationIncomplete: { reason: "workspace_source_restricted", issueId: randomUUID() },
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it("keeps a restricted shared workspace source actionable on the task", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, restrictedWorkspaceSourceRun(), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { errorCode: "setup_failed" }, { executionStage: "executing" }, { exitCode: 1 }, { signal: "SIGTERM" },
|
|
{ resultJson: { configurationIncomplete: { reason: "workspace_source_restricted" } } },
|
|
{ resultJson: { configurationIncomplete: { reason: "workspace_source_restricted" }, executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } },
|
|
])("reports restricted-source markers without an exact pre-dispatch outcome: %j", async (overrides) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, restrictedWorkspaceSourceRun(overrides), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([undefined, "execute"] as const)("reports restricted-source markers outside the setup report: %s", async (phase) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, restrictedWorkspaceSourceRun(), { phase });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
function aiConnectionRequiredRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "configuration_incomplete",
|
|
executionStage: "preparing",
|
|
resultJson: {
|
|
configurationIncomplete: { reason: "ai_connection_required", actionUrl: "/agents/fixture/runtime" },
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it("keeps a plugin Runner-task run without a managed AI connection actionable on the task", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, aiConnectionRequiredRun(), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([
|
|
{ executionStage: "executing" },
|
|
{ resultJson: { configurationIncomplete: { reason: "ai_connection_required" }, executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } },
|
|
])("reports AI-connection-required markers without an exact pre-dispatch outcome: %j", async (overrides) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, aiConnectionRequiredRun(overrides), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
function localPathConfigurationRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
|
return buildRun({
|
|
errorCode: "workspace_validation_failed",
|
|
executionStage: "preparing",
|
|
resultJson: {
|
|
workspaceValidation: {
|
|
reason: "git_worktree_base_not_git_checkout",
|
|
configurationReason: "local_path_requires_git_checkout",
|
|
resolvedWorkspaceSource: "project_primary",
|
|
workspaceStrategyType: "git_worktree",
|
|
requestedExecutionWorkspaceMode: "isolated_workspace",
|
|
},
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
|
},
|
|
...overrides,
|
|
});
|
|
}
|
|
|
|
it.each(["isolated_workspace", "operator_branch"])("keeps proven local-path policy conflicts actionable locally: %s", async (mode) => {
|
|
await seedCompanyAndAgent();
|
|
const run = localPathConfigurationRun();
|
|
(run.resultJson!.workspaceValidation as Record<string, unknown>).requestedExecutionWorkspaceMode = mode;
|
|
const before = structuredClone(run);
|
|
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
expect(run).toEqual(before);
|
|
});
|
|
|
|
it.each([
|
|
{ configurationReason: undefined },
|
|
{ configurationReason: "unknown_reason" },
|
|
{ reason: "git_worktree_base_materialization_failed" },
|
|
{ reason: "git_worktree_not_reusable" },
|
|
{ reason: "inherited_workspace_reuse_unavailable" },
|
|
{ resolvedWorkspaceSource: "agent_home" },
|
|
{ workspaceStrategyType: "project_primary" },
|
|
{ requestedExecutionWorkspaceMode: "agent_default" },
|
|
])("reports unproven and other workspace failures: %j", async (validation) => {
|
|
await seedCompanyAndAgent();
|
|
const run = localPathConfigurationRun();
|
|
Object.assign(run.resultJson!.workspaceValidation as object, validation);
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([
|
|
{ status: "timed_out" }, { errorCode: "setup_failed" }, { executionStage: "executing" },
|
|
{ exitCode: 1 }, { signal: "SIGTERM" }, { resultJson: {} },
|
|
{ resultJson: { workspaceValidation: { reason: "git_worktree_base_not_git_checkout", configurationReason: "local_path_requires_git_checkout" } } },
|
|
])("reports workspace failures without an exact pre-dispatch outcome: %j", async (overrides) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, localPathConfigurationRun(overrides), { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([undefined, "execute"] as const)("reports local-path markers outside the setup report: %s", async (phase) => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, localPathConfigurationRun(), { phase });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each([undefined, { kind: "bootstrap" }, { kind: "bootstrap", providerWorkStarted: true }])(
|
|
"reports local-path markers without affirmative unstarted-provider evidence: %j", async (executionRecovery) => {
|
|
await seedCompanyAndAgent();
|
|
const run = localPathConfigurationRun();
|
|
run.resultJson!.executionRecovery = executionRecovery;
|
|
await reportRunFailure(db, run, { phase: "setup" });
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
|
|
it("captures once for the status timed_out", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "timed_out" });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ runStatus: "timed_out" }),
|
|
);
|
|
});
|
|
|
|
it("forwards stored exit evidence even when the adapter error is generic", async () => {
|
|
await seedCompanyAndAgent();
|
|
for (const processExit of [
|
|
{ exitCode: 1, signal: null },
|
|
{ exitCode: null, signal: "SIGTERM" },
|
|
{ exitCode: null, signal: null },
|
|
]) {
|
|
mockCaptureRunFailure.mockClear();
|
|
await reportRunFailure(db, buildRun({
|
|
error: "Adapter failed",
|
|
errorCode: "adapter_failed",
|
|
...processExit,
|
|
stdoutExcerpt: "private-output",
|
|
stderrExcerpt: "private-error-output",
|
|
resultJson: { private: "adapter-result" },
|
|
}));
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(expect.objectContaining({
|
|
errorMessage: "Adapter failed",
|
|
errorCode: "adapter_failed",
|
|
...processExit,
|
|
}));
|
|
const captured = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(captured).not.toHaveProperty("stdoutExcerpt");
|
|
expect(captured).not.toHaveProperty("stderrExcerpt");
|
|
expect(captured).not.toHaveProperty("resultJson");
|
|
}
|
|
});
|
|
|
|
it("reports selected provider diagnostics and the original cause without copying arbitrary payloads", async () => {
|
|
await seedCompanyAndAgent();
|
|
const cause = Object.assign(new Error("provider unreachable"), {
|
|
code: "ECONNRESET", requestId: "request-123", status: 503,
|
|
response: { body: "private-response" },
|
|
});
|
|
const error = new Error("adapter threw", { cause });
|
|
const run = buildRun({
|
|
runtimeMode: "legacy",
|
|
executionStage: "execute",
|
|
error: "adapter threw",
|
|
stderrExcerpt: "private-stderr",
|
|
stdoutExcerpt: "private-stdout",
|
|
contextSnapshot: { prompt: "private-prompt" },
|
|
resultJson: {
|
|
terminalSessionFailure: { category: "service", title: "Provider unavailable", details: "request-123 failed", raw: "private-provider-raw" },
|
|
timeoutFired: false, summary: "private-summary", env: { SECRET: "private-env" },
|
|
},
|
|
});
|
|
await reportRunFailure(db, run, { error, phase: "execute", adapterErrorMeta: { phase: "turn", retryable: true, response: "private-adapter-response" } });
|
|
const captured = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(captured.diagnostics).toMatchObject({
|
|
execution: { runtimeMode: "legacy", executionStage: "execute", failurePhase: "execute", timeoutFired: false },
|
|
adapter: { phase: "turn", retryable: true },
|
|
provider: { category: "service", title: "Provider unavailable", details: "request-123 failed" },
|
|
exceptions: [{ message: "adapter threw" }, { message: "provider unreachable", code: "ECONNRESET", status: 503, requestId: "request-123" }],
|
|
});
|
|
expect(captured.diagnostics.exceptions[0].stack).toContain("run-failure-report.test.ts");
|
|
// Match the sensitive fixture values, not a legitimate checkout name in the stack.
|
|
for (const value of ["private-response", "private-stderr", "private-stdout", "private-prompt", "private-provider-raw", "private-summary", "private-env", "private-adapter-response"]) {
|
|
expect(JSON.stringify(captured)).not.toContain(value);
|
|
}
|
|
expect(error.cause).toBe(cause);
|
|
});
|
|
|
|
it.each([false, true])("redacts registered run secrets and fails closed when resolution fails: %s", async (fails) => {
|
|
await seedCompanyAndAgent();
|
|
const secret = "opaque-registered-value";
|
|
const run = buildRun({
|
|
error: `connection failed: ${secret}`,
|
|
contextSnapshot: { paperclipSecretRedactions: [{ fingerprintSha256: "fixture", material: { encrypted: "fixture" } }] },
|
|
resultJson: { terminalSessionFailure: { category: "service", details: `upstream rejected ${secret}` } },
|
|
});
|
|
await db.insert(heartbeatRuns).values(run);
|
|
if (fails) mockResolveSecret.mockRejectedValueOnce(new Error("fixture resolution failed"));
|
|
else mockResolveSecret.mockResolvedValueOnce(secret);
|
|
|
|
await expect(reportRunFailure(db, run, {
|
|
error: new Error(`failed ${secret}`, { cause: new Error(`cause ${secret}`) }),
|
|
adapterErrorMeta: { causeMessage: `adapter ${secret}` },
|
|
})).resolves.toBeUndefined();
|
|
expect(mockResolveSecret).toHaveBeenCalledTimes(1);
|
|
if (fails) {
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
} else {
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
const captured = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(JSON.stringify(captured)).not.toContain(secret);
|
|
expect(captured.diagnostics.provider.details).toContain(REDACTED_EVENT_VALUE);
|
|
expect(captured.diagnostics.exceptions[1].message).toContain(REDACTED_EVENT_VALUE);
|
|
}
|
|
});
|
|
|
|
it("redacts runtime and host environment secret values even without a persisted registry", async () => {
|
|
await seedCompanyAndAgent();
|
|
const runtimeSecret = "opaque-runtime-value";
|
|
const hostSecret = "opaque-host-value";
|
|
vi.stubEnv("FIXTURE_CUSTOM_VALUE", hostSecret);
|
|
const text = `connection failed: ${runtimeSecret} ${hostSecret}`;
|
|
const run = buildRun({ error: text, contextSnapshot: null,
|
|
resultJson: { terminalSessionFailure: { details: text } },
|
|
});
|
|
await reportRunFailure(db, run, {
|
|
error: new Error(text, { cause: new Error(text) }),
|
|
adapterErrorMeta: { causeMessage: text, stackPreview: text },
|
|
secretValues: [runtimeSecret],
|
|
});
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
const captured = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(JSON.stringify(captured)).not.toContain(runtimeSecret);
|
|
expect(JSON.stringify(captured)).not.toContain(hostSecret);
|
|
expect(captured.diagnostics.exceptions).toHaveLength(2);
|
|
expect(captured).not.toHaveProperty("secretValues");
|
|
});
|
|
|
|
it("still redacts short values from unknown host settings", async () => {
|
|
await seedCompanyAndAgent();
|
|
vi.stubEnv("FIXTURE_CUSTOM_VALUE", "1");
|
|
|
|
await reportRunFailure(db, buildRun());
|
|
|
|
expect(mockCaptureRunFailure.mock.calls[0][0].errorMessage)
|
|
.toBe(`the provider process exited with code ${REDACTED_EVENT_VALUE}`);
|
|
});
|
|
|
|
it("waits for the terminal write when a native run is failed pending a same-run retry", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, buildRun({ runtimeMode: "native", nativePhase: "retryable_failure" }));
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
|
|
await reportRunFailure(db, buildRun({ runtimeMode: "native", nativePhase: "terminal_failure" }));
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("treats leaving a pending native retry as a new terminal failure", () => {
|
|
const pending = { status: "failed", runtimeMode: "native", nativePhase: "retryable_failure" } as const;
|
|
expect(isNewTerminalRunFailure({ ...pending, status: "running", nativePhase: "observed" }, pending)).toBe(true);
|
|
expect(isNewTerminalRunFailure(pending, { ...pending, nativePhase: "terminal_failure" })).toBe(true);
|
|
expect(isNewTerminalRunFailure(pending, { ...pending, nativePhase: "committed" })).toBe(true);
|
|
expect(isNewTerminalRunFailure(pending, pending)).toBe(false);
|
|
expect(isNewTerminalRunFailure({ ...pending, nativePhase: null }, { ...pending, nativePhase: "terminal_failure" })).toBe(false);
|
|
expect(isNewTerminalRunFailure({ ...pending, runtimeMode: "legacy" }, { ...pending, runtimeMode: "legacy", nativePhase: null }))
|
|
.toBe(false);
|
|
});
|
|
|
|
it("captures nothing for succeeded, cancelled, and interrupted", async () => {
|
|
await seedCompanyAndAgent();
|
|
for (const status of ["succeeded", "cancelled", "interrupted"] as const) {
|
|
const run = buildRun({ status });
|
|
await reportRunFailure(db, run);
|
|
}
|
|
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
it.each(["provider", "unknown"])("reports an unexpected started cancellation from %s", async source => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "cancelled", startedAt: new Date(0), finishedAt: new Date(1000),
|
|
resultJson: { cancellation: { source, expected: false, initiator: { type: "provider", id: "private-actor" },
|
|
reason: "private-reason", recordedAt: new Date(1000).toISOString() } } });
|
|
await reportRunFailure(db, run);
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
expect(mockCaptureRunFailure.mock.calls[0][0]).toMatchObject({ runStatus: "cancelled",
|
|
diagnostics: { execution: { cancellationSource: source, cancellationExpected: false } } });
|
|
expect(JSON.stringify(mockCaptureRunFailure.mock.calls)).not.toMatch(/private-actor|private-reason/);
|
|
});
|
|
it("labels a host controller lease loss with its closed cause", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, buildRun({ status: "cancelled", startedAt: new Date(0), finishedAt: new Date(1000),
|
|
resultJson: { cancellation: controllerLeaseLossCancellation("renewal_deadline") } }));
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
expect(mockCaptureRunFailure.mock.calls[0][0]).toMatchObject({ diagnostics: { execution: {
|
|
cancellationSource: "controller_lease", cancellationExpected: false, controllerLeaseLossCause: "renewal_deadline" } } });
|
|
expect(JSON.stringify(mockCaptureRunFailure.mock.calls)).not.toMatch(/Legacy controller lease lost/);
|
|
});
|
|
it("reports bounded lease-loss timings for a setup failure", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, buildRun({ status: "failed", errorCode: "setup_failed", startedAt: new Date(0),
|
|
finishedAt: new Date(1000), resultJson: { controllerLeaseLoss: { cause: "renewal_deadline",
|
|
sinceRenewalMs: 61_000.4, renewalPendingMs: 59_000, deadlineLateMs: 1e12, renewalCount: -1, note: "private-note" } } }));
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
const execution = mockCaptureRunFailure.mock.calls[0][0].diagnostics.execution;
|
|
expect(execution).toMatchObject({ controllerLeaseLossCause: "renewal_deadline", controllerLeaseSinceRenewalMs: 61_000,
|
|
controllerLeaseRenewalPendingMs: 59_000, controllerLeaseDeadlineLateMs: 86_400_000 });
|
|
expect(execution).not.toHaveProperty("controllerLeaseRenewalCount");
|
|
expect(JSON.stringify(mockCaptureRunFailure.mock.calls)).not.toMatch(/private-note/);
|
|
});
|
|
it("drops lease-loss timings without a closed cause", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, buildRun({ status: "failed", errorCode: "setup_failed", startedAt: new Date(0),
|
|
finishedAt: new Date(1000), resultJson: { controllerLeaseLoss: { cause: "forged", sinceRenewalMs: 1 } } }));
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
|
const execution = mockCaptureRunFailure.mock.calls[0][0].diagnostics.execution;
|
|
expect(execution).not.toHaveProperty("controllerLeaseLossCause");
|
|
expect(execution).not.toHaveProperty("controllerLeaseSinceRenewalMs");
|
|
});
|
|
it("does not report an operator's Stop as a failure", async () => {
|
|
await seedCompanyAndAgent();
|
|
await reportRunFailure(db, buildRun({ status: "cancelled", startedAt: new Date(0), resultJson: {
|
|
cancellation: { source: "operator", expected: true, initiator: { type: "user", id: "board" },
|
|
reason: "Stop", recordedAt: new Date().toISOString() },
|
|
} }));
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("sends agents.adapterType from the loaded agent row", async () => {
|
|
await seedCompanyAndAgent({ adapterType: "claude_managed" });
|
|
const run = buildRun({ status: "failed" });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ agentAdapter: "claude_managed" }),
|
|
);
|
|
});
|
|
|
|
it("sends the adapter value unknown when the agent row is absent", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "failed", agentId: randomUUID() });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ agentAdapter: "unknown" }),
|
|
);
|
|
});
|
|
|
|
it("calls redactCurrentUserText on the error message before it composes the credential redactor", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "failed", error: "raw message" });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockRedactCurrentUserText).toHaveBeenCalledWith("raw message");
|
|
});
|
|
|
|
it("removes an Authorization: Bearer credential from the error message", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({
|
|
status: "failed",
|
|
error: "the adapter request failed: Authorization: Bearer live-secret-token-value",
|
|
});
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).not.toContain("live-secret-token-value");
|
|
expect(errorMessage).toContain(REDACTED_EVENT_VALUE);
|
|
});
|
|
|
|
it("removes an API-key form from the error message", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({
|
|
status: "failed",
|
|
error: `adapter payload: {"apiKey":"json-secret-value"}`,
|
|
});
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).not.toContain("json-secret-value");
|
|
expect(errorMessage).toContain(REDACTED_EVENT_VALUE);
|
|
});
|
|
|
|
it("removes a JSON Web Token form from the error message", async () => {
|
|
await seedCompanyAndAgent();
|
|
const jwt =
|
|
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c";
|
|
const run = buildRun({ status: "failed", error: `session token: ${jwt}` });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).not.toContain(jwt);
|
|
expect(errorMessage).toContain(REDACTED_EVENT_VALUE);
|
|
});
|
|
|
|
it("removes a password value from the error message", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({
|
|
status: "failed",
|
|
error: `login failed: password="hunter2-super-secret"`,
|
|
});
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).not.toContain("hunter2-super-secret");
|
|
expect(errorMessage).toContain(REDACTED_EVENT_VALUE);
|
|
});
|
|
|
|
it("removes a database connection string from the error message", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({
|
|
status: "failed",
|
|
error: `connect failed: connectionString: "postgres://appuser:s3cr3t-pass@db.internal:5432/paperclip"`,
|
|
});
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).not.toContain("s3cr3t-pass");
|
|
expect(errorMessage).toContain(REDACTED_EVENT_VALUE);
|
|
});
|
|
|
|
it("still masks the current user's home path in the error message", async () => {
|
|
await seedCompanyAndAgent();
|
|
const homeDir = os.homedir();
|
|
const rawError = `read failed at ${homeDir}/workspace/report.log`;
|
|
const run = buildRun({ status: "failed", error: rawError });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
// Environment-value redaction can fully mask the username before the
|
|
// current-user redactor applies its partial mask. Both remove the home path.
|
|
expect(errorMessage).toContain("workspace/report.log");
|
|
expect(errorMessage).not.toContain(homeDir);
|
|
});
|
|
|
|
const MAX_ERROR_MESSAGE_LENGTH = 4096;
|
|
|
|
it("truncates an error message that is longer than the bound", async () => {
|
|
await seedCompanyAndAgent();
|
|
const longError = "x".repeat(MAX_ERROR_MESSAGE_LENGTH + 500);
|
|
const run = buildRun({ status: "failed", error: longError });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).toHaveLength(MAX_ERROR_MESSAGE_LENGTH);
|
|
expect(errorMessage).toBe("x".repeat(MAX_ERROR_MESSAGE_LENGTH - 12) + "\n[truncated]");
|
|
});
|
|
|
|
it("preserves a short error message with known public host settings", async () => {
|
|
vi.stubEnv("PAPERCLIP_DB_BACKUP_ENABLED", "false");
|
|
vi.stubEnv("PAPERCLIP_DB_BACKUP_RETENTION_DAYS", "1");
|
|
vi.stubEnv("GITHUB_RUN_ATTEMPT", "1");
|
|
await seedCompanyAndAgent();
|
|
const shortError = "the provider process exited with code 1";
|
|
const run = buildRun({ status: "failed", error: shortError });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorMessage } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorMessage).toBe(shortError);
|
|
});
|
|
|
|
const MAX_ERROR_CODE_LENGTH = 200;
|
|
|
|
it("sends a normal error code such as adapter_failed to Sentry unchanged", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "failed", errorCode: "adapter_failed" });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ errorCode: "adapter_failed" }),
|
|
);
|
|
});
|
|
|
|
it("redacts an error code that holds a credential form", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({
|
|
status: "failed",
|
|
errorCode: "Authorization: Bearer live-secret-token-value",
|
|
});
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorCode } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorCode).not.toContain("live-secret-token-value");
|
|
expect(errorCode).toContain(REDACTED_EVENT_VALUE);
|
|
});
|
|
|
|
it("truncates an error code that is longer than 200 characters", async () => {
|
|
await seedCompanyAndAgent();
|
|
const longErrorCode = "y".repeat(MAX_ERROR_CODE_LENGTH + 50);
|
|
const run = buildRun({ status: "failed", errorCode: longErrorCode });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
const { errorCode } = mockCaptureRunFailure.mock.calls[0][0];
|
|
expect(errorCode).toHaveLength(MAX_ERROR_CODE_LENGTH);
|
|
expect(errorCode).toBe("y".repeat(MAX_ERROR_CODE_LENGTH - 12) + "\n[truncated]");
|
|
});
|
|
|
|
it("sends errorCode null unchanged when the run holds no error code", async () => {
|
|
await seedCompanyAndAgent();
|
|
const run = buildRun({ status: "failed", errorCode: null });
|
|
|
|
await reportRunFailure(db, run);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ errorCode: null }),
|
|
);
|
|
});
|
|
|
|
it("reads the task id from nativeIssueId, falling back to contextSnapshot.issueId", async () => {
|
|
await seedCompanyAndAgent();
|
|
const nativeIssueId = randomUUID();
|
|
const runWithNativeIssueId = buildRun({
|
|
status: "failed",
|
|
nativeIssueId,
|
|
contextSnapshot: { issueId: randomUUID() },
|
|
});
|
|
|
|
await reportRunFailure(db, runWithNativeIssueId);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ taskId: nativeIssueId }),
|
|
);
|
|
|
|
mockCaptureRunFailure.mockClear();
|
|
const contextIssueId = randomUUID();
|
|
const runWithContextIssueId = buildRun({
|
|
status: "failed",
|
|
nativeIssueId: null,
|
|
contextSnapshot: { issueId: contextIssueId },
|
|
});
|
|
|
|
await reportRunFailure(db, runWithContextIssueId);
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledWith(
|
|
expect.objectContaining({ taskId: contextIssueId }),
|
|
);
|
|
});
|
|
|
|
it("does not throw when the database read fails", async () => {
|
|
const throwingDb = {
|
|
select: () => {
|
|
throw new Error("connection reset");
|
|
},
|
|
} as unknown as Db;
|
|
const run = buildRun({ status: "failed", agentId: randomUUID() });
|
|
|
|
await expect(reportRunFailure(throwingDb, run)).resolves.toBeUndefined();
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
describe("waitForPendingRunFailureReports", () => {
|
|
function deferredAgentRows() {
|
|
let resolve!: (rows: Array<{ adapterType: string }>) => void;
|
|
const promise = new Promise<Array<{ adapterType: string }>>((res) => {
|
|
resolve = res;
|
|
});
|
|
return { promise, resolve };
|
|
}
|
|
|
|
function fakeDb(agentRowsPromise: Promise<Array<{ adapterType: string }>>): Db {
|
|
return {
|
|
select: () => ({
|
|
from: () => ({
|
|
where: () => agentRowsPromise,
|
|
}),
|
|
}),
|
|
} as unknown as Db;
|
|
}
|
|
|
|
it("resolves at once when no report is in flight", async () => {
|
|
await expect(waitForPendingRunFailureReports()).resolves.toBeUndefined();
|
|
});
|
|
|
|
it("waits for an unawaited report's database read and Sentry capture before it resolves", async () => {
|
|
const agentRows = deferredAgentRows();
|
|
const run = buildRun({ status: "failed", agentId: randomUUID() });
|
|
|
|
// Do not await the report — this models the fire-and-forget
|
|
// `void reportRunFailure(db, run)` call every caller uses.
|
|
void reportRunFailure(fakeDb(agentRows.promise), run);
|
|
|
|
const drain = waitForPendingRunFailureReports(1_000);
|
|
await new Promise((resolve) => setTimeout(resolve, 10));
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
|
|
agentRows.resolve([{ adapterType: "claude_managed" }]);
|
|
await drain;
|
|
|
|
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("gives up after the bound and does not throw when a report never settles", async () => {
|
|
const run = buildRun({ status: "failed", agentId: randomUUID() });
|
|
void reportRunFailure(fakeDb(new Promise(() => undefined)), run);
|
|
|
|
await expect(waitForPendingRunFailureReports(20)).resolves.toBeUndefined();
|
|
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
|
});
|
|
});
|
|
});
|