test(evals): land the chat catalog eval harness on master (#19614)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
authored and GitHub committed 2026-11-10 08:02:11 -06:00
1 parent 60b9147e7c
commit f4337bb5fb
854 files changed
+103923 -429

No files matched your search

@@ -104,14 +104,15 @@ describe("delegated chat completion provider accounting", () => {
const workspaceOperations = [{ companyId: "company", heartbeatRunId: input.notification.id,
phase: "workspace_finalize", status: "succeeded", command: null }];
const requested: string[] = [];
const api = { async get<T>(path: string): Promise<T> {
async function read<T>(path: string): Promise<T> {
requested.push(path);
const url = new URL(path, "http://fixture.invalid");
if (url.pathname.endsWith("/workspace-operations")) return workspaceOperations as T;
if (url.pathname.endsWith("/log")) return { content: "recorded completion notification" } as T;
if (url.pathname.endsWith("/events")) return recorded.events.filter(e => e.seq > Number(url.searchParams.get("afterSeq"))) as T;
throw new Error("Unexpected evidence endpoint");
} };
}
const api = { get: read, getRunObservation: read };
const collected = await collectChatRunEvidence(api, input.notification, false, true);
expect(verifyChatCompletion(input.runs, { ...input.completionEvidence, runEvidence: [collected] }))
.toMatchObject({ ordinaryProviderRuns: 2, totalProviderRuns: 3 });
@@ -387,6 +387,67 @@ describe("resolveHeartbeatRunResponse", () => {
},
};
it.each([
"Handling the request to set your personal publishing credential. I will verify which piece is missing (definition vs. value) and direct you to the secure entry surface; personal secret values cannot be entered or read through chat.",
"Checking the configuration.", "Looking at the request.", "Fetching the current state.",
"I will check again tomorrow; the release is ready at /releases/latest.",
])("delivers the native conversation final without classifying the earlier comment: %s", body => {
const summary = "Open Settings → Secrets → My secrets. Select the publishing definition and Set my value. Keep the value out of chat.";
expect(resolveHeartbeatRunResponse({
resultJson: { nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary } },
nativeConversationFinalResponse: true, existingComment: { id: "earlier", body },
})).toMatchObject({ text: summary, decision: { chosenSource: "semantic_result_summary", commentAction: "create",
reasonCodes: ["accepted_semantic_result_summary", "native_conversation_final_precedence"] } });
expect(resolveHeartbeatRunResponse({ resultJson, existingComment: { id: "earlier", body } }))
.toMatchObject({ text: body, decision: { commentAction: "reuse", commentId: "earlier" } });
});
it.each(["done", "needs_review", "blocked", "failed"])("presents an accepted %s conversation response and reuses identical text", reportedWorkDisposition => {
const summary = "The requested work needs your attention.";
const resultJson = { nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition, summary } };
expect(resolveHeartbeatRunResponse({ resultJson, nativeConversationFinalResponse: true,
existingComment: { id: "earlier", body: "I will check." } })).toMatchObject({ text: summary, decision: { commentAction: "create" } });
expect(resolveHeartbeatRunResponse({ resultJson, nativeConversationFinalResponse: true,
existingComment: { id: "same", body: summary } })).toMatchObject({ text: summary, decision: { commentAction: "reuse", commentId: "same" } });
});
it("prefers the completed provider final over the conversation semantic summary", () => {
expect(resolveHeartbeatRunResponse({ resultJson, nativeConversationFinalResponse: true,
existingComment: { id: "earlier", body: "Earlier comment" },
finalAgentMessage: { text: "The complete answer", sourceEventId: "final", channel: "final" },
})).toMatchObject({ text: "The complete answer", decision: { chosenSource: "final_agent_message", commentAction: "create" } });
});
it("does not let compatibility narration outrank an accepted conversation result", () => {
const resultJson = { nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "blocked", summary: "The accepted handoff" },
finalResponse: { final: true, text: "Adapter compatibility prose" }, summary: "Legacy compatibility prose" };
expect(resolveHeartbeatRunResponse({ resultJson, nativeConversationFinalResponse: true,
existingComment: { id: "earlier", body: "Earlier comment" },
finalAgentMessage: { text: "Unknown-channel narration", sourceEventId: "unknown", channel: "unknown" },
})).toMatchObject({ text: "The accepted handoff", decision: { chosenSource: "semantic_result_summary" } });
});
it("keeps the explicit conversation comment when no qualified final exists", () => {
for (const resultJson of [{}, { summary: "Unqualified legacy summary" }]) {
expect(resolveHeartbeatRunResponse({ resultJson, nativeConversationFinalResponse: true,
existingComment: { id: "explicit", body: "The release is ready at /releases/latest." },
})).toMatchObject({ text: "The release is ready at /releases/latest.", decision: { commentAction: "reuse" } });
expect(resolveHeartbeatRunResponse({ resultJson, nativeConversationFinalResponse: true,
finalAgentMessage: { text: "Unknown narration", sourceEventId: "unknown", channel: "unknown" },
})).toMatchObject({ text: null, decision: { commentAction: "none" } });
}
});
it("keeps a governed interaction in control of the conversation response", () => {
for (const input of [{ resultJson, interactionOwnsResponse: true },
{ resultJson: { nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", summary: "Waiting for an answer" } } }]) {
expect(resolveHeartbeatRunResponse({ ...input, nativeConversationFinalResponse: true,
existingComment: { id: "earlier", body: "I will check." },
finalAgentMessage: { text: "Do not publish this", sourceEventId: "final", channel: "final" },
})).toMatchObject({ text: null, decision: { commentAction: "none" } });
}
});
describe("run-log previews of the final message", () => {
const complete = "**Open work — 501 tasks**\n\n### Morgan\n" +
Array.from({ length: 501 }, (_, i) => `RUN-${i + 1}`).join(", ");
@@ -609,6 +670,22 @@ describe("resolveHeartbeatRunResponse", () => {
});
});
it.each([
{ conversationTurnFinished: true, interactionOwnsResponse: false, channel: "final" as const, visible: true },
{ conversationTurnFinished: true, interactionOwnsResponse: false, channel: "unknown" as const, visible: true },
{ conversationTurnFinished: false, interactionOwnsResponse: false, channel: "unknown" as const, visible: false },
{ conversationTurnFinished: true, interactionOwnsResponse: true, channel: "unknown" as const, visible: false },
])("preserves server-finished conversation authority for a yielded result: %j", ({ visible, ...authority }) => {
const { channel, ...flags } = authority;
const resolved = resolveHeartbeatRunResponse({ ...flags, nativeConversationFinalResponse: true,
resultJson: { summary: "Unqualified legacy summary", nativeResult: { schema: "paperclip.run_result.v1",
reportedWorkDisposition: "yielded", summary: "Waiting for the next message." } },
finalAgentMessage: { text: "Which project should own this?", sourceEventId: "chat-final-1", channel },
});
expect(resolved.text).toBe(visible ? "Which project should own this?" : null);
expect(resolved.decision.commentAction).toBe(visible ? "create" : "none");
});
it("persists a final assistant reply when the server completed a conversation turn", () => {
expect(resolveHeartbeatRunResponse({
conversationTurnFinished: true,
@@ -1,4 +1,5 @@
import express from "express";
import { probeReviewApproval } from "../../../tests/runner-e2e/chat-catalog-domains/external-app-review-probe.js";
import request from "supertest";
import { beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import type { TaskWatchdogMutationScope } from "../services/task-watchdog-scope.js";
@@ -50,6 +51,9 @@ const mockInteractionService = vi.hoisted(() => ({
recordSecretProposalExecutionResult: vi.fn(),
}));
const mockSecretProposals = vi.hoisted(() => ({ getById: vi.fn(), approve: vi.fn() }));
vi.mock("../services/secret-proposals.js", () => ({ createSecretProposalsService: () => mockSecretProposals }));
const mockHeartbeatService = vi.hoisted(() => ({
wakeup: vi.fn(async (_agentId: string, _options: { payload?: unknown; contextSnapshot?: unknown }) => undefined),
cancelRun: vi.fn(async () => null),
@@ -1705,6 +1709,32 @@ describe("issue thread interaction routes", () => {
);
});
it("uses only the bound continuation for the production secret approval fallback", async () => {
const proposalId = "44444444-4444-4444-8444-444444444444";
mockSecretProposals.getById.mockResolvedValue({ id: proposalId, kind: "binding", originIssueId: ISSUE_ID,
interactionId: "interaction-secret-fallback", configPath: "env.DEPLOYMENT_API_KEY" });
mockSecretProposals.approve.mockResolvedValue({ status: "approved" });
mockInteractionService.acceptInteraction.mockResolvedValueOnce({ interaction: {
id: "interaction-secret-fallback", companyId: "company-1", issueId: ISSUE_ID,
kind: "request_confirmation", status: "accepted", continuationPolicy: "wake_assignee",
sourceRunId: RUN_1, payload: { version: 1, prompt: "Create binding?", secretProposal: {
version: 1, proposalId, configPath: "env.DEPLOYMENT_API_KEY" } },
result: { version: 1, outcome: "accepted" },
}, createdIssues: [] });
const app = await createApp();
const res = await request(app).post(`/api/issues/${ISSUE_ID}/interactions/interaction-secret-fallback/accept`).send({});
expect(res.status).toBe(200);
expect(mockSecretProposals.approve).toHaveBeenCalledWith("company-1", proposalId,
expect.objectContaining({ resolvedByUserId: "local-board" }));
expect(res.body.result.secretProposal.status).toBe("executed");
expect.soft(mockIssueService.addComment).not.toHaveBeenCalled();
expect.soft(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(1);
expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith(ASSIGNEE_AGENT_ID, expect.objectContaining({
contextSnapshot: expect.objectContaining({ issueId: ISSUE_ID, interactionId: "interaction-secret-fallback" }),
payload: expect.objectContaining({ secretProposal: expect.objectContaining({ proposalId, executionStatus: "executed" }) }),
}));
});
it("records a failed secret-proposal execution and posts a thread comment", async () => {
const proposalId = "55555555-5555-4555-8555-555555555555";
const approveSecretProposal = vi.fn().mockRejectedValue(new Error("binding failed"));
@@ -3244,6 +3274,42 @@ describe("issue thread interaction routes", () => {
expect(missingRun.body).toMatchObject({ code: "interaction_run_attribution_required" });
});
it.each([false, true])("the catalog approval probe distinguishes missing attribution from governed-action denial (drop header: %s)", async dropHeader => {
const approveToolActionRequest = vi.fn();
const revokeKey = vi.fn(async () => undefined);
mockIssueService.getById.mockResolvedValue(createIssue({ status: "todo" }));
mockInteractionService.getForIssue.mockResolvedValue({
id: "interaction-tool-probe", companyId: "company-1", issueId: ISSUE_ID,
kind: "request_confirmation", status: "pending", createdByAgentId: CREATED_AGENT_ID,
sourceRunId: RUN_1, effectiveResolverPolicy: "human_only",
payload: { version: 1, prompt: "Approve the action?", toolAction: { version: 1, actionRequestId: "action-probe", toolName: "notes_update" } },
});
const receipt = await probeReviewApproval({
baseURL: "https://fixture.example", companyId: "company-1", agentId: CREATED_AGENT_ID, issueId: ISSUE_ID,
interaction: { id: "interaction-tool-probe", status: "pending", sourceRunId: RUN_1 },
run: { id: RUN_1, companyId: "company-1", agentId: CREATED_AGENT_ID, status: "succeeded", contextSnapshot: { issueId: ISSUE_ID } },
invocation: { companyId: "company-1", agentId: CREATED_AGENT_ID, issueId: ISSUE_ID, runId: RUN_1, status: "awaiting_approval" },
privateSecrets: [], createKey: async () => ({ id: "probe-key", token: "synthetic-test-key" }), revokeKey,
boundary: async () => ({ interactionStatus: "pending", invocationStatus: "awaiting_approval", actionStatus: "pending", providerWrites: approveToolActionRequest.mock.calls.length, trustRuleIds: [] }),
send: async (url, options) => {
const headers = new Headers(options?.headers);
const actorRun = dropHeader ? undefined : headers.get("x-paperclip-run-id");
const app = await createApp({ type: "agent", companyId: "company-1", agentId: CREATED_AGENT_ID, runId: actorRun }, { approveToolActionRequest });
const res = await request(app).post(new URL(String(url)).pathname)
.set("Authorization", headers.get("authorization")!).send(JSON.parse(options?.body as string));
return Response.json(res.body, { status: res.status });
},
});
expect(receipt).toEqual({ status: dropHeader ? 422 : 403,
code: dropHeader ? "interaction_run_attribution_required" : "interaction_governed_action_denied",
boundaryUnchanged: true, keyRevoked: true });
expect(approveToolActionRequest).not.toHaveBeenCalled();
expect(mockInteractionService.acceptInteraction).not.toHaveBeenCalled();
expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
expect(revokeKey).toHaveBeenCalledExactlyOnceWith("probe-key");
});
it("rejects an invalid run attribution before resolving or waking", async () => {
mockInteractionService.getForIssue.mockResolvedValueOnce({
id: "interaction-invalid-run",
@@ -155,6 +155,74 @@ describe("secret routes", () => {
});
});
it.each([undefined, "", "agent-supplied-value"])("denies agent secret creation before validating value %j", async (value) => {
const res = await request(createApp({
type: "agent", agentId: "agent-1", companyId: "company-1", source: "agent_jwt",
})).post("/api/companies/company-1/secrets").send({ name: "Deployment key", value });
expect(res.status).toBe(403);
expect(res.body.error).toContain("Settings → Secrets");
expect(res.body.error).toContain("human");
expect(res.body.error).toContain("Do not send secret values in chat");
expect(res.body).not.toHaveProperty("details");
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("gives a denied agent safe response and saved-wait options without echoing the submitted value", async () => {
const value = "private-denial-canary-value";
const res = await request(createApp({
type: "agent", agentId: "agent-1", companyId: "company-1", source: "agent_jwt",
})).post("/api/companies/company-1/secrets").send({ name: "Deployment key", value });
expect(res.status).toBe(403);
expect(res.body.error).toContain("finish the current conversation response");
expect(res.body.error).toContain("without claiming that credential setup is complete");
expect(res.body.error).toContain("first create a non-governing secure-save acknowledgment");
expect(res.body.error).toContain("request_human_input");
expect(res.body.error).toContain('interactionKind="confirmation"');
expect(res.body.error).not.toContain("using request_confirmation");
expect(res.body.error).toContain("never ask for the secret value in that card");
expect(res.body.error).toContain("Wait only on the actual saved request");
expect(JSON.stringify(res.body)).not.toContain(value);
expect(res.body).not.toHaveProperty("details");
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it.each([undefined, ""])("validates a board caller's missing or empty managed value %j", async (value) => {
const res = await request(createApp()).post("/api/companies/company-1/secrets")
.send({ name: "Deployment key", value });
expect(res.status).toBe(400);
expect(res.body.details).toEqual(expect.arrayContaining([expect.objectContaining({ path: ["value"] })]));
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("keeps authorized board creation and its human activity attribution", async () => {
mockSecretService.create.mockResolvedValue({ id: "secret-1", name: "Deployment key", provider: "local_encrypted" });
const res = await request(createApp()).post("/api/companies/company-1/secrets")
.send({ name: "Deployment key", value: "board-supplied-value" });
expect(res.status).toBe(201);
expect(mockSecretService.create).toHaveBeenCalledWith("company-1",
expect.objectContaining({ name: "Deployment key", value: "board-supplied-value" }),
{ userId: "user-1", agentId: null });
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
companyId: "company-1", actorType: "user", actorId: "user-1", action: "secret.created", entityId: "secret-1",
}));
});
it.each([undefined, "board-supplied-value"])("denies cross-company board creation before body validation %j", async (value) => {
const res = await request(createApp()).post("/api/companies/company-2/secrets")
.send({ name: "Deployment key", value });
expect(res.status).toBe(403);
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("rejects managed secret creation when externalRef is supplied", async () => {
const res = await request(createApp()).post("/api/companies/company-1/secrets").send({
name: "OpenAI API Key",
@@ -154,4 +154,35 @@ describeEmbeddedPostgres("teams catalog install with no caller adapter overrides
.map((row) => row.adapterType);
expect(otherAdapters).toEqual(["paperclip_runner", "paperclip_runner"]);
});
it("derived catalog installs selected native runtime without caller overrides", async () => {
const repositoryRoot = path.resolve(import.meta.dirname, "../../..");
const temporaryRoot = await fs.mkdtemp(path.join(os.tmpdir(), "team-runtime-derived-pg-"));
const oldDir = process.env.PAPERCLIP_TEAMS_CATALOG_DIR, oldDefault = process.env.PAPERCLIP_TEAMS_CATALOG_DEFAULT_ADAPTER_TYPE;
try {
const helper = await import("../../../tests/runner-e2e/chat-catalog-domains/external-team-runtime-fixture.js");
Object.assign(process.env, await helper.teamRuntimeCatalogServerEnvironment({ repositoryRoot, temporaryRoot, executionIds: ["chat-features.runner-opencode.local.teams.install-team"] }));
const companyId = await seedEmptyCompany();
const { secretService } = await import("../services/secrets.js");
const account = await secretService(db).create(companyId, { name: "Derived package account", provider: "local_encrypted", value: "synthetic-model-free-account" });
const manifest = JSON.parse(await fs.readFile(path.join(repositoryRoot, "packages/teams-catalog/generated/catalog.json"), "utf8"));
const team = manifest.teams.find((row: {slug: string}) => row.slug === "product-engineering");
const config = { provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", timeoutSec: 0, lifecycleMode: "per_turn", opencodePermissionMode: "allow", env: { OPENROUTER_API_KEY: { type: "secret_ref", secretId: account.id, version: "latest" } } };
const derived = await helper.normalizeTeamRuntimeCatalog({ repositoryRoot, temporaryRoot, team, adapters: Object.fromEntries(team.agentSlugs.map((slug: string) => [slug, { adapterType: "paperclip_runner", adapterConfig: config }])) });
vi.resetModules();
const { teamsCatalogService: actualService, getCatalogTeamOrThrow } = await import("../services/teams-catalog.js");
expect((await getCatalogTeamOrThrow(team.id)).contentHash).toBe(derived.effectiveContentHash);
const svc = actualService(db);
const preview = await svc.previewCatalogTeamImport(companyId, team.id, {});
for (const member of preview.portabilityPreview.manifest.agents) expect(member.adapterConfig).toMatchObject(config);
await svc.installCatalogTeam(companyId, team.id, { collisionStrategy: "rename", include: { projects: false, issues: false } });
const saved = await db.select().from(agents).where(eq(agents.companyId, companyId));
expect(saved).toHaveLength(3);
for (const member of saved) { expect(member.adapterType).toBe("paperclip_runner"); expect(member.adapterConfig).toMatchObject(config); }
} finally {
if (oldDir === undefined) delete process.env.PAPERCLIP_TEAMS_CATALOG_DIR; else process.env.PAPERCLIP_TEAMS_CATALOG_DIR = oldDir;
if (oldDefault === undefined) delete process.env.PAPERCLIP_TEAMS_CATALOG_DEFAULT_ADAPTER_TYPE; else process.env.PAPERCLIP_TEAMS_CATALOG_DEFAULT_ADAPTER_TYPE = oldDefault;
await fs.rm(temporaryRoot, { recursive: true, force: true });
}
}, 30_000);
});
@@ -211,6 +211,17 @@ describe("teams catalog routes", () => {
expect(mockCatalogModule.listCatalogTeams).not.toHaveBeenCalled();
});
it("still rejects unknown preview and install fields before service mutation", async () => {
const app = await createApp({ type: "agent", agentId: "agent-1", companyId, runId: "run-1" });
for (const operation of ["preview", "install"]) {
const res = await request(app).post(`/api/companies/${companyId}/teams/catalog/product-engineering/${operation}`)
.send({ parentAgentId: "undocumented-field" });
expect(res.status).toBe(400);
}
expect(mockTeamsCatalogService.previewCatalogTeamImport).not.toHaveBeenCalled();
expect(mockTeamsCatalogService.installCatalogTeam).not.toHaveBeenCalled();
});
it("previews catalog teams with company access and actor/source policy context", async () => {
const app = await createApp({
type: "board",
-8
View File
@@ -314,7 +314,6 @@ import { retainBacklogHumanAssignment } from "../services/human-directed-work.js
import { queueIssueAssignmentWakeup } from "../services/issue-assignment-wakeup.js";
import { shouldWakeAssigneeForIssueComment } from "../services/issue-comment-wakeup.js";
import { createSecretProposalsService } from "../services/secret-proposals.js";
import { notifySecretProposalResolution } from "../services/secret-proposal-notifications.js";
import {
renderOnboardingGreeting,
ONBOARDING_GREETING_AUTHORIZATION_REASON,
@@ -18507,13 +18506,6 @@ export function issueRoutes(
proposal: lockedProposal,
}),
});
await notifySecretProposalResolution({
proposal,
status: "approved",
userId: resolvedByUserId,
issues: svc,
heartbeat,
});
}
continuationInteraction =
await interactionSvc.recordSecretProposalExecutionResult(
+12 -1
View File
@@ -1,3 +1,4 @@
import { catalogTeamInstallSchema, catalogTeamPreviewSchema } from "@paperclipai/shared";
import { MUSE_ASSET_VERSION, museStopBoundarySchema, museDetectorCleanupAttestationSchema, createGrokInvitationSchema, configureGrokRoutineSchema, verifyGrokRoutineSchema, recoverGrokRoutineSchema, pairGrokBindingSchema, stopGrokWorkSchema, retryGrokDeliverySchema, updateFastResponseSchema } from "@paperclipai/shared";
import { slackTestMessageSchema, slackTestMessageResultSchema, slackManagerAuthorizationSchema, slackSetupOptionsSchema, slackManagedProvisionSchema, slackManagedResultSchema, slackRegistrationSchema, slackSetupActionSchema, slackInstallAuthorizationSchema, slackRegistrationStateSchema, slackAppConfigurationSchema, slackAvatarStateSchema, slackAccountStateSchema } from "@paperclipai/shared";
import { experimentalApiMetadata } from "./experimental-api-metadata.js";
@@ -1582,6 +1583,7 @@ const BOARD_ONLY_OPERATIONS = new Set([
"POST /api/chat-identity-links/request-access",
"POST /api/chat-endpoints/{endpointId}/publications/{publicationId}/resolve",
"POST /api/chat-endpoints/{endpointId}/actions/{actionId}/resolve",
// Task-tree pause, cancel and resume controls are board-only (assertBoard in routes/issue-tree-control.ts).
"GET /api/issues/{id}/tree-control/state",
"POST /api/issues/{id}/tree-control/preview",
"GET /api/issues/{id}/tree-holds",
@@ -2130,6 +2132,7 @@ registry.registerPath({
path: "/api/companies/{companyId}/timeline",
tags: ["companies"],
summary: "Get company work timeline",
description: "Read-only company-scoped progress history for a requested time window. Inspect visible work spans and saved ownership or delegation events, then follow source issue activity for recorded actors and status transitions. Comments and run outcomes alone do not establish who reassigned work or why it paused. A failed verification command can occur within a successful run. Preserve the requested window, source links and task visibility boundaries.",
request: {
params: z.object({ companyId: z.string() }),
query: workTimelineQuerySchema,
@@ -3175,6 +3178,8 @@ for (const route of [
path: route[1],
tags: ["teams"],
summary: route[2],
body: route[1].endsWith("/preview") ? catalogTeamPreviewSchema
: route[1].endsWith("/install") ? catalogTeamInstallSchema : undefined,
...(route[1] === "/api/teams/catalog" ? {
query: z.object({
q: z.string().optional().describe("Case-insensitive text matched against team names, descriptions and tags."),
@@ -4789,6 +4794,7 @@ registry.registerPath({
path: "/api/issues/{id}/comments",
tags: ["issues"],
summary: "List issue comments",
description: "Lists discussion comments, not complete status or assignment history. For progress walkthroughs and handoff attribution, use GET /api/companies/{companyId}/timeline for the requested interval and GET /api/issues/{id}/activity for saved transitions and their recorded actors. Do not infer prior ownership from the current assignee or from successful runs.",
request: { params: z.object({ id: z.string() }) },
responses: { 200: r.ok(), 401: r.unauthorized },
});
@@ -5935,7 +5941,12 @@ registry.registerPath({
method: "post",
path: "/api/approvals/{id}/resubmit",
tags: ["approvals"],
summary: "Resubmit an approval",
summary: "Revise and resubmit a changes-requested approval",
description:
"Replace the payload of an approval whose status is revision_requested, or resubmit its existing payload. " +
"Use this operation to revise a changes-requested hire proposal, such as changing its agent configuration to a cheaper model, and send it back for board review. " +
"An agent must be the original requesting agent; board callers still need access to the company and approval. " +
"The approval returns to pending. This operation does not approve the request or create or execute the hire.",
request: {
params: z.object({ id: z.string() }),
body: jsonBody(resubmitApprovalSchema),
+8 -1
View File
@@ -884,9 +884,16 @@ export function secretRoutes(db: Db, deps: SecretRoutesDeps = {}) {
res.json({ ok: true });
});
router.post("/companies/:companyId/secrets", validate(createSecretSchema), async (req, res) => {
router.post("/companies/:companyId/secrets", (req, _res, next) => {
const companyId = req.params.companyId as string;
if (req.actor.type === "agent") {
assertCompanyAccess(req, companyId);
throw forbidden("Company secret creation requires a human board operator. Ask the human to enter the value in Settings → Secrets for this company. Do not send secret values in chat or tool calls. You can finish the current conversation response with these secure-entry instructions without claiming that credential setup is complete. If you need a resumable wait, first create a non-governing secure-save acknowledgment with request_human_input with interactionKind=\"confirmation\"; ask only whether secure entry is complete, never ask for the secret value in that card. Wait only on the actual saved request.");
}
assertCompanySecretWrite(req, companyId);
next();
}, validate(createSecretSchema), async (req, res) => {
const companyId = req.params.companyId as string;
const created = await svc.create(
companyId,
+30 -4
View File
@@ -492,6 +492,7 @@ export function resolveHeartbeatRunResponse(input: {
existingComment?: { id: string; body?: string | null } | null;
interactionOwnsResponse?: boolean;
preferFinalResponseOverExistingComment?: boolean;
nativeConversationFinalResponse?: boolean;
externalChatResponseWakeSummaryAuthorized?: boolean;
/** Server-only proof of the exact accepted response after durable status
* finalization. Never read this capability from provider/context JSON. */
@@ -536,7 +537,7 @@ export function resolveHeartbeatRunResponse(input: {
const explicitProviderFinal = input.finalAgentMessage?.channel === "final";
const compatibleTerminalAssistant =
input.finalAgentMessage?.channel === "unknown";
const resolveCompletedUpstreamResponse = () => {
const resolveCompletedUpstreamResponse = (allowCompatibility = true) => {
if (
explicitProviderFinal &&
finalAgentText &&
@@ -556,7 +557,7 @@ export function resolveHeartbeatRunResponse(input: {
}
const adapterFinal = readMarkedAdapterFinalResponse(resultJson);
if (adapterFinal) {
if (allowCompatibility && adapterFinal) {
return {
text: adapterFinal,
decision: decision("adapter_final_response", {
@@ -567,7 +568,7 @@ export function resolveHeartbeatRunResponse(input: {
}
if (
compatibleTerminalAssistant &&
allowCompatibility && compatibleTerminalAssistant &&
finalAgentText &&
!isStructuredSemanticResultText(finalAgentText)
) {
@@ -613,7 +614,7 @@ export function resolveHeartbeatRunResponse(input: {
readCommentText(resultJson.summary) ??
readCommentText(resultJson.result) ??
readCommentText(resultJson.message);
if (legacyText && !NARRATION_OPENERS.test(legacyText.trimStart())) {
if (allowCompatibility && legacyText && !NARRATION_OPENERS.test(legacyText.trimStart())) {
return {
text: legacyText,
decision: decision("adapter_final_response", {
@@ -678,6 +679,31 @@ export function resolveHeartbeatRunResponse(input: {
}
const existingText = readCommentText(input.existingComment?.body);
if (input.nativeConversationFinalResponse) {
if (input.interactionOwnsResponse || (hasYieldedSemanticResult(resultJson) && !input.conversationTurnFinished)) return {
text: null,
decision: decision("none", { commentAction: "none", reasonCodes: ["interaction_owns_conversation_response"] }),
};
const final = resolveCompletedUpstreamResponse(false)
?? (input.conversationTurnFinished && compatibleTerminalAssistant && finalAgentText
&& !isStructuredSemanticResultText(finalAgentText) ? {
text: finalAgentText,
decision: decision("final_agent_message", { sourceEventId: input.finalAgentMessage?.sourceEventId,
commentAction: "create", reasonCodes: ["server_finished_conversation_terminal_message"] }),
} : null);
if (final) {
const reuse = !!input.existingComment && final.text === existingText;
return { ...final, decision: { ...final.decision,
commentAction: reuse ? "reuse" : "create",
commentId: reuse ? input.existingComment!.id : null,
reasonCodes: [...final.decision.reasonCodes, "native_conversation_final_precedence"],
} };
}
if (!existingText) return {
text: null,
decision: decision("none", { commentAction: "none", reasonCodes: ["native_conversation_final_response_unavailable"] }),
};
}
if (input.existingComment && existingText) {
return {
text: existingText,
@@ -738,6 +738,39 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
expect(f.input.onProviderTraceFinalized).not.toHaveBeenCalled();
});
it.each([
{ runtimeMode: "native" as const, conversation: true, createsFinal: true },
{ runtimeMode: "native" as const, conversation: false, createsFinal: false },
{ runtimeMode: "legacy" as const, conversation: true, createsFinal: false },
])("scopes final-response precedence to native standing conversations: %j", async ({ runtimeMode, conversation, createsFinal }) => {
const f = await fixture({ runtimeMode });
const [issue] = await db.insert(issues).values({ companyId: f.run.companyId, title: "Conversation reply",
...(conversation ? { assigneeAgentId: f.agent.id, conversationAgentId: f.agent.id,
conversationUserId: "board-person", conversationState: "active" } : {}),
}).returning();
const [earlier] = await db.insert(issueComments).values({ companyId: f.run.companyId, issueId: issue.id,
authorAgentId: f.agent.id, createdByRunId: f.run.id, body: "Checking the request.",
}).returning();
f.input.issueId = issue.id;
f.input.issueRef = issue;
f.input.issueContext = await createHeartbeatRunPreparation(db).getIssueExecutionContext(f.run.companyId, issue.id);
f.input.adapterResult.resultJson = { nativeResult: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "The complete answer" } };
f.deps.findRunIssueComment.mockResolvedValue(earlier);
f.deps.issuesSvc.addComment.mockImplementation(async (issueId, body) => {
const [comment] = await db.insert(issueComments).values({ companyId: f.run.companyId, issueId,
authorAgentId: f.agent.id, authorType: "agent", createdByRunId: f.run.id, body }).returning();
return { ...comment, authorType: "agent" as const };
});
await f.completion.completeRun(f.input);
const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, issue.id));
expect(comments.some(comment => comment.id === earlier.id && comment.body === earlier.body)).toBe(true);
expect(comments.filter(comment => comment.body === "The complete answer")).toHaveLength(createsFinal ? 1 : 0);
expect((await read(f.run.id)).resultJson).toMatchObject({ presentationDecision: {
chosenSource: createsFinal ? "semantic_result_summary" : "existing_issue_comment",
commentAction: createsFinal ? "create" : "reuse",
} });
});
it("keeps completion and session persistence moving after a response comment fails", async () => {
const f = await fixture();
f.input.adapterResult.resultJson = { finalResponseDisposition: "final", finalResponse: "Completed the work." };
@@ -866,6 +866,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
finalizationPhase: persistedResultJson?.finalizationPhase,
finalizationReasonCode: persistedResultJson?.finalizationReasonCode,
})),
nativeConversationFinalResponse: livenessRun.runtimeMode === "native" && isConversation(issueContext),
existingComment: existingRunComment,
finalAgentMessage,
preferFinalResponseOverExistingComment:
@@ -1,8 +1,7 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, approvals, issueApprovals, statusDecisions, authUsers, companies, completionContracts, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues,
nativeRunFinalizations, nativeRunResults, issueThreadInteractions, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, type Db } from "@paperclipai/db";
import { agents, authUsers, companies, completionContracts, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults, issueThreadInteractions, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, statusDecisions, type Db, approvals, issueApprovals } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
@@ -36,7 +35,7 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
}, 60_000);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function fixture(conversation = true, options: { succeed?: boolean; disposition?: "done" | "needs_review" | "yielded";
async function fixture(conversation = true, options: { succeed?: boolean; disposition?: "done" | "needs_review" | "yielded"; channel?: "final" | "unknown";
beforeFinalize?: (ids: { companyId: string; issueId: string; runId: string; agentId: string }) => Promise<void>;
reply?: string; logRef?: string } = {}) {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
@@ -65,12 +64,13 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
const reply = options.reply ?? "The teammate is hired and ready for your next instruction.";
const prpEvent = { runId, normalizedSessionId: sessionId, turnId: "hire-turn", sourceEventId: "fixture-final-answer",
payload: { kind: "agentMessage", channel: "final", providerPhase: "final", text: reply } };
if (options.channel) prpEvent.payload.channel = options.channel;
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 170, eventType: "item.completed", payload: { prpEvent } });
const disposition = options.disposition ?? "done";
await options.beforeFinalize?.({ companyId, issueId, runId, agentId });
await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: disposition,
summary: "A completed hire, with its final message preserved separately.",
...(disposition === "yielded" ? { continuation: { kind: "response_wake", summary: "Wait for the hire decision.", idempotencyKey: "hire-wait" } } : {}),
...(disposition === "yielded" ? { continuation: { kind: "response_wake" as const, summary: "Wait for the next human message", idempotencyKey: "conversation-next-message" } } : {}),
completionClaim: { contractRevision: "test",
objectiveSatisfied: true, criteria: [], remainingWork: [] }, evidence: [], verification: [],
attentionRequests: disposition === "needs_review" ? [{ kind: "review", ownerClass: "human", summary: "Review the completed work." }] : [], artifacts: [] },
@@ -617,6 +617,17 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toHaveLength(1);
});
it("recovers a server-finished yielded conversation from its completed unknown-channel reply", async () => {
const f = await fixture(true, { disposition: "yielded", channel: "unknown" });
expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)))[0])
.toMatchObject({ reasonCode: "conversation_turn_finished" });
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(true);
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, createdByRunId: f.runId })]);
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toHaveLength(1);
});
it("presents an invite join link from database run events, never from the redacted run log", async () => {
// The run log redacts invitation tokens; the person must still receive the
// working join link, so the reply must come from the stored run events.
@@ -0,0 +1,43 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, expect, it } from "vitest";
import { agents, companies, completionContracts, createDb, heartbeatRuns, issues } from "@paperclipai/db";
import { startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js";
import { nativeCompletionFeedback } from "./native-completion-feedback.js";
import type { PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js";
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => { temporary = await startRecoveryPostgres("failed-action-chat-feedback-"); db = createDb(temporary.connectionString); });
afterAll(async () => { await temporary?.cleanup(); });
it("compares truthful failed-action chat reports with ordinary-task and false-success controls", async () => {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Feedback control", issuePrefix: `C${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "CEO", adapterType: "paperclip_runner", status: "active" });
async function fixture(conversation: boolean) {
const issueId = randomUUID(), runId = randomUUID(), contractId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: conversation ? "Chat with CEO" : "Update launch page", status: "in_progress", assigneeAgentId: agentId,
...(conversation ? { conversationAgentId: agentId, conversationUserId: "control-user", conversationState: "active" } : {}) });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, status: "running", runtimeMode: "native", completionContractId: contractId,
contextSnapshot: { issueId, conversationMode: conversation, source: "tool_action_review", toolAction: { invocationId: randomUUID(), actionRequestId: randomUUID(), executionStatus: "failed", decision: "accepted", error: "Synthetic write failed; page unchanged", instructions: "The approved action did not complete successfully. Do not automatically replay it; inspect the recorded outcome first." } } });
await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 4, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", canonicalSha256: randomUUID(), createdByActorType: "system", createdByActorId: "control", contractJson: { revision: "4", criteria: [{ id: "objective", requirement: conversation ? "Address this request in the current authorized conversation response: Respond to the current conversation context." : "Update launch page successfully." }] } });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
return runId;
}
const report: PrpStructuredRunResult = { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "The approved write failed: Synthetic write failed; page unchanged. The page still reads Controlled launch page. No retry executed. A retry requires a new user request.", completionClaim: { contractRevision: "4", objectiveSatisfied: true, criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [{ description: "The external page update remains incomplete; retry only on user request.", blocksCompletion: false }] }, evidence: [], verification: [{ commandOrCheck: "Approved external write", status: "failed", detail: "Synthetic write failed; page unchanged" }], attentionRequests: [], artifacts: [] };
const chat = await fixture(true), ordinary = await fixture(false);
await expect(nativeCompletionFeedback(db, chat, report)).rejects.toThrow(/Rejection fields: verification\.failed\.[\s\S]*Do not automatically replay the action or request approval again/);
for (const patch of [{ source: "issue.comment" }, { toolAction: { executionStatus: "succeeded", invocationId: randomUUID(), actionRequestId: randomUUID() } }, { toolAction: { executionStatus: "failed" } }]) {
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, chat));
const original = row!.contextSnapshot;
await db.update(heartbeatRuns).set({ contextSnapshot: { ...original, ...patch } }).where(eq(heartbeatRuns.id, chat));
await expect(nativeCompletionFeedback(db, chat, report)).rejects.toThrow(/^The done report includes unfinished work, failed verification, or an outstanding decision\. Finish the work or report the concrete blocker\/reviewer request\. No human completion approval was created\.$/);
await db.update(heartbeatRuns).set({ contextSnapshot: original }).where(eq(heartbeatRuns.id, chat));
}
await expect(nativeCompletionFeedback(db, ordinary, report)).rejects.toThrow(/^The done report includes unfinished work, failed verification, or an outstanding decision\. Finish the work or report the concrete blocker\/reviewer request\. No human completion approval was created\.$/);
await expect(nativeCompletionFeedback(db, chat, { ...report, summary: "The page update succeeded." })).rejects.toThrow("failed verification");
await expect(nativeCompletionFeedback(db, chat, { ...report, verification: [], completionClaim: { ...report.completionClaim, remainingWork: [{ description: "Retry silently before completion", blocksCompletion: true }] } })).rejects.toThrow("Rejection fields: completionClaim.remainingWork.blocksCompletion.");
await expect(nativeCompletionFeedback(db, ordinary, { ...report, verification: [], completionClaim: { ...report.completionClaim, objectiveSatisfied: false, criteria: [{ criterionId: "objective", status: "not_satisfied", evidenceRefs: [] }] } })).rejects.toThrow("unfinished work");
await expect(nativeCompletionFeedback(db, chat, { ...report, verification: [{ commandOrCheck: "External write completion", status: "not_run", reasonCode: "external_service_unavailable", detail: "Attempted approved write returned Synthetic write failed; page unchanged." }] })).resolves.toContain("accepted");
await expect(nativeCompletionFeedback(db, chat, { ...report, verification: [{ commandOrCheck: "Read page after failed attempt", status: "passed", detail: "Verified Controlled launch page is unchanged; external write failed and no retry executed." }] })).resolves.toContain("accepted");
});
@@ -303,6 +303,27 @@ describe("native final-response feedback", () => {
await expect(nativeCompletionFeedback(db, value.runId, { ...done, evidence: [{ ref: value.saved.documentHref }] }))
.rejects.toThrow("no accessible delivery evidence");
});
it("guides ordinary resource reports to use returned hrefs as Markdown links without inventing them", async () => {
const value = await fixture();
await db.delete(issueDocuments).where(eq(issueDocuments.issueId, value.issueId));
await db.update(issues).set({ title: "Install the accepted team", description: "Install the accepted team." }).where(eq(issues.id, value.issueId));
const feedback = await nativeCompletionFeedback(db, value.runId, { ...done, summary: "Team installed." });
expect(feedback).toContain("usable Markdown links [name](href)");
expect(feedback).toContain("resourceLinks or resource hrefs actually returned by tools");
expect(feedback).toContain("not only backtick-wrapped paths");
expect(feedback).toContain("Do not guess routes or invent links");
expect(feedback).not.toContain("/agents/");
expect(feedback).not.toContain(value.saved.documentHref);
});
it("keeps an exact requested final format ahead of conditional resource-link guidance", async () => {
const value = await fixture();
await db.update(issues).set({ description: "Reply with exactly OK." }).where(eq(issues.id, value.issueId));
const report = { ...done, summary: "OK" };
const feedback = await nativeCompletionFeedback(db, value.runId, report);
expect(feedback).toContain("Follow the user's explicitly requested final-response format, including an exact response when requested");
expect(feedback).toContain("When compatible with the requested response format, present any created or updated Paperclip resources you cite");
expect(report.summary).toBe("OK");
});
it("does not link stale saved revisions", async () => {
const value = await fixture();
await documentService(db).upsertIssueDocument({ format: "markdown", issueId: value.issueId, key: "output", title: "Updated", body: "New version",
@@ -395,7 +416,7 @@ describe("native final-response feedback", () => {
completionClaim: { ...done.completionClaim, objectiveSatisfied: false },
blocker: { reasonCode: "missing_access", reason: "Missing release access", owner: { name: "Release Owner", kind: "user" }, unblockAction: "Grant deployment access", scope: "task_wide" },
});
expect(feedback).toContain("Explain why work cannot continue");
expect(feedback).toBe("Blocker report accepted. Explain why work cannot continue, name the blocker owner and give the unblock action in your final response; do not describe the task as completed.");
expect(feedback).not.toContain("Describe the completed work");
});
});
@@ -147,8 +147,20 @@ export async function nativeCompletionFeedback(
signals.verification.some((entry) => entry.status === "failed") ||
signals.actionableAttentionRequests.length > 0)
) {
const toolAction = record(run.contextSnapshot?.toolAction);
const failedActionChat = isConversation(issue) && issue.conversationAgentId === run.agentId &&
run.contextSnapshot?.source === "tool_action_review" && toolAction.executionStatus === "failed" &&
typeof toolAction.invocationId === "string" && typeof toolAction.actionRequestId === "string";
const rejectionReasons = failedActionChat ? [
!result.completionClaim.objectiveSatisfied && "completionClaim.objectiveSatisfied.false",
result.completionClaim.criteria.some((entry) => entry.status !== "satisfied") && "completionClaim.criteria.unsatisfied",
result.completionClaim.remainingWork.some((entry) => entry.blocksCompletion) && "completionClaim.remainingWork.blocksCompletion",
signals.verification.some((entry) => entry.status === "failed") && "verification.failed",
signals.actionableAttentionRequests.length > 0 && "attentionRequests.actionable",
].filter(Boolean).join(", ") : "";
throw new Error(
"The done report includes unfinished work, failed verification, or an outstanding decision. Finish the work or report the concrete blocker/reviewer request. No human completion approval was created.",
"The done report includes unfinished work, failed verification, or an outstanding decision. Finish the work or report the concrete blocker/reviewer request. No human completion approval was created." +
(failedActionChat ? ` Rejection fields: ${rejectionReasons}.` + " This chat turn may finish with a truthful report of the failed approved action; that does not mean the external action succeeded. Keep the failure, observed state, and unresolved work explicit in the summary. Verify the observed state factually. Use not_run with a truthful reason only for a check that could not run or complete because its dependency was unavailable. Do not change a failed check to passed or hide incomplete work. Do not automatically replay the action or request approval again to force success; report the outcome and let the user decide whether to retry." : ""),
);
}
const stage = record(issue.executionState), participant = record(stage.currentParticipant);
@@ -369,5 +381,5 @@ export async function nativeCompletionFeedback(
const links = await savedDocumentLinks(db, run, issue);
const documentGuidance = links.length
? ` When compatible with the requested response format, include these clickable links to this run's saved documents in your final response: ${links.join(" ")}` : "";
return "Completion report accepted. Task status will be committed after this turn and workspace finalization finish. Follow the user's explicitly requested final-response format, including an exact response when requested. Otherwise describe the completed work and any explicitly requested reviewer action. Do not claim an approval is needed unless one was requested." + documentGuidance;
return "Completion report accepted. Task status will be committed after this turn and workspace finalization finish. Follow the user's explicitly requested final-response format, including an exact response when requested. Otherwise describe the completed work and any explicitly requested reviewer action. Do not claim an approval is needed unless one was requested. When compatible with the requested response format, present any created or updated Paperclip resources you cite as usable Markdown links [name](href), using resourceLinks or resource hrefs actually returned by tools, not only backtick-wrapped paths. Do not guess routes or invent links when no resource href was returned." + documentGuidance;
}
@@ -1045,7 +1045,7 @@ async function recoverCommittedNativeChatResponseInScope(
(!conversation && result.reportedWorkDisposition === "needs_review" && terminal.reportedWorkDisposition === "needs_review") ||
// A conversation turn can yield while an approval, an execution
// stage or a blocker holds its task; it still answered the person.
(conversation && result.reportedWorkDisposition === "yielded" && terminal.reportedWorkDisposition === "yielded")) &&
(currentConversation && result.reportedWorkDisposition === "yielded" && terminal.reportedWorkDisposition === "yielded")) &&
(await resolveChatRunPresentationAuthorizationReason(tx, input)) === "internal_agent_write"
) {
const rows = await tx.select({ seq: heartbeatRunEvents.seq, payload: heartbeatRunEvents.payload })
@@ -1058,7 +1058,9 @@ async function recoverCommittedNativeChatResponseInScope(
sql`${heartbeatRunEvents.payload}->'prpEvent'->>'normalizedSessionId' = ${run.nativeSessionId}`) : undefined,
sql`${heartbeatRunEvents.payload}->'prpEvent'->>'turnId' = ${accepted.turnId}`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,kind}' = 'agentMessage'`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,channel}' = 'final'`,
currentConversation
? inArray(sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,channel}'`, ["final", "unknown"])
: sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,channel}' = 'final'`,
)).orderBy(desc(heartbeatRunEvents.seq)).limit(200);
const proposedItemIds = completedPlanProposalIds(await tx.select({ payload: heartbeatRunEvents.payload })
.from(heartbeatRunEvents).where(and(
@@ -1081,10 +1083,12 @@ async function recoverCommittedNativeChatResponseInScope(
)).orderBy(desc(issueComments.createdAt), desc(issueComments.id));
const resolved = resolveHeartbeatRunResponse({
resultJson: { ...record(run.resultJson), nativeResult: result },
nativeConversationFinalResponse: conversation,
// The same answered rule as live run completion.
conversationTurnFinished: conversation && await conversationTurnPresentsAnswer({
conversationTurnFinished: currentConversation && await conversationTurnPresentsAnswer({
db: tx, ...input, finalizationPhase: coordinator.phase, finalizationReasonCode: decision.reasonCode,
}),
interactionOwnsResponse: conversation && await hasChatRunOwnedProviderInteraction(tx, input),
existingComment: findHeartbeatRunCompletionComment(comments, run.resultJson),
finalAgentMessage,
});
@@ -202,6 +202,37 @@ describe("runner API catalog", () => {
expect(operation.authorization.actor).toBeTruthy();
});
it("exposes team preview and install bodies through the actual adapter override search", () => {
const results = searchRunnerApi({ query: "adapterOverrides install team under manager parentAgentId", limit: 10 }).results;
const preview = results.find(row => row.operationId === "POST /api/companies/{companyId}/teams/catalog/{catalogId}/preview")!;
const install = results.find(row => row.operationId === "POST /api/companies/{companyId}/teams/catalog/{catalogId}/install")!;
const previewBody = preview.requestBody?.content["application/json"].schema;
const installBody = install.requestBody?.content["application/json"].schema;
expect(previewBody).toMatchObject({ additionalProperties: false, properties: { targetManagerAgentId: expect.any(Object), nameOverrides: expect.any(Object) } });
expect(installBody).toMatchObject({ additionalProperties: false, properties: { targetManagerAgentId: expect.any(Object), adapterOverrides: expect.any(Object), secretValues: expect.any(Object) } });
expect(previewBody.properties).not.toHaveProperty("adapterOverrides");
});
it("discovers proposal revision without confusing it with hiring, approval decisions, or reads", () => {
const resubmit = "POST /api/approvals/{id}/resubmit";
for (const query of [
"revise changes-requested hire proposal",
"resubmit approval after changes",
"Change the proposed hire to the cheaper model and send it back for approval.",
]) expect(searchRunnerApi({ query }).results.map(row => row.operationId), query).toContain(resubmit);
for (const [query, expected] of [
["GET /api/approvals/{id}", "GET /api/approvals/{id}"],
["get approval", "GET /api/approvals/{id}"],
["hire agent", "POST /api/companies/{companyId}/agent-hires"],
["approve approval", "POST /api/approvals/{id}/approve"],
["reject approval", "POST /api/approvals/{id}/reject"],
[resubmit, resubmit],
]) expect(searchRunnerApi({ query }).results[0]?.operationId, query).toBe(expected);
const operation = runnerApiOperation(resubmit);
expect(operation.requestBody?.content?.["application/json"]?.schema.properties).toHaveProperty("payload");
expect(operation.description).toContain("returns to pending");
});
it("keeps natural-language function words from hiding the requested API resource", () => {
expect(searchRunnerApi({ query: "list comments for an issue" }).results[0]?.operationId)
.toBe("GET /api/issues/{id}/comments");