fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages

## Linked Issues or Issue Description

**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.

**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.

**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.

**Paperclip version or commit**
master at 9dfdeaad86.

## What Changed

- Added `GET /api/companies/{companyId}/observability/status`. Agents of
the company and all members can read it. It returns `configured`,
`enabled`, `provider`, `credentialsConfigured`, `updatedAt` and the
latest test status. It never returns the destination or a credential.
- Added the shared `AgentObservabilityStatus` type and
`agentObservabilityStatus()` mapper, and registered the route in the
OpenAPI catalog so that `search_api` finds it.
- The chat task markdown now fills the conversation's own company prefix
(from its identifier, for example `COM-1` gives `COM`) into the chat
directives in place of `/<prefix>/`.
- `search_api` results now get the same substitution for the caller's
company. The shared catalog is not changed.
- Pinned the changed shared source in
`tests/runner-e2e/source-sdk-public-inputs.json`.

## Verification

- `cd server && npx vitest run src/__tests__/agent-observability.test.ts
src/__tests__/agent-observability-status-guidance.test.ts
src/services/heartbeat/task-markdown.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`
- Red/green: with the product change removed, 10 of the new or updated
tests fail (the agent status read gets 403, search does not find the
route, and the chat markdown still contains `/<prefix>/`). With the
change, all pass.
- `npx tsc -p server/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.

## Risks

- Low risk. The new route is read-only and returns no destination and no
credential. The existing settings route keeps its board-only rule.
- The chat prompt text changes for chats with an identifier. The text is
constant for each company, so prompt caching is not affected inside a
conversation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-11 20:55:21 -06:00
1 parent 65ff80a4bb
commit 601babe322
12 files changed
+128 -15

No files matched your search

+17
View File
@@ -84,6 +84,23 @@ POST /api/companies/{companyId}/archive
Archives a company. Archived companies are hidden from default listings.
## Get Agent Trace Export Status
```
GET /api/companies/{companyId}/observability/status
```
Returns whether agent trace export is set up for the company. Every member of the company and every agent in it can read this. It never returns the destination or any credential. Reading or changing the full settings at `/api/companies/{companyId}/observability` stays with board users who can manage connections.
| Field | Type | Description |
|-------|------|-------------|
| `configured` | boolean | Whether a destination is saved |
| `enabled` | boolean | Whether export is on |
| `provider` | string \| null | Destination provider, for example `otlp` |
| `credentialsConfigured` | boolean | Whether credentials are saved for the destination |
| `updatedAt` | string \| null | When the settings last changed |
| `lastTest` | object \| null | Latest connection test: `status` and `finishedAt` |
## Company Fields
| Field | Type | Description |
@@ -88,6 +88,22 @@ export type AgentObservabilitySettings = {
updatedAt: string; lastTest: AgentObservabilityTestReceipt | null;
};
export type AgentObservabilitySettingsResponse = { canManage: boolean; settings: AgentObservabilitySettings | null };
/**
* Whether the company exports agent traces, without the destination or any
* credential. Readable by the company's agents and every member, so an agent
* can answer from saved settings instead of guessing from its own config.
*/
export type AgentObservabilityStatus = {
configured: boolean; enabled: boolean; provider: AgentObservabilityDestination["provider"] | null;
credentialsConfigured: boolean; updatedAt: string | null;
lastTest: { status: AgentObservabilityTestReceipt["status"]; finishedAt: string } | null;
};
export function agentObservabilityStatus(settings: AgentObservabilitySettings | null): AgentObservabilityStatus {
if (!settings) return { configured: false, enabled: false, provider: null, credentialsConfigured: false, updatedAt: null, lastTest: null };
return { configured: true, enabled: settings.enabled, provider: settings.destination.provider,
credentialsConfigured: settings.credentialsConfigured, updatedAt: settings.updatedAt,
lastTest: settings.lastTest ? { status: settings.lastTest.status, finishedAt: settings.lastTest.finishedAt } : null };
}
export type AgentObservabilityUnavailableReason =
| "configuration_invalid" | "credentials_unavailable" | "unsupported_harness" | "unsupported_home_mode";
@@ -1050,7 +1050,7 @@ describe("chat prompt policy", () => {
expect(AGENT_CHAT_DIRECTIVE).toContain("/<prefix>/agents/<urlKey or id>/api-keys for API keys");
expect(AGENT_CHAT_DIRECTIVE).toContain("The API keys page is not in the profile's navigation");
const prompt = buildPaperclipTaskMarkdown({ issue: { id: "chat", identifier: "RUN-1", title: "Chat", workMode: "ask", conversationAgentId: "agent" } });
expect(prompt).toContain("/<prefix>/agents/<urlKey or id>/api-keys");
expect(prompt).toContain("/RUN/agents/<urlKey or id>/api-keys");
});
it("says how to read the company link prefix from a task identifier so agent links are not guessed", () => {
@@ -0,0 +1,28 @@
import { expect, it } from "vitest";
import { runnerApiOperation, searchRunnerApi, withCompanyPagePaths } from "../services/native-runtime/runner-api-catalog.js";
const STATUS = "GET /api/companies/{companyId}/observability/status";
it.each([
"test collector configuration",
"agent traces otlp collector",
"observability export enabled",
])("search %j finds the agent-readable trace export status", (query) => {
expect(searchRunnerApi({ query, limit: 10 }).results.map(row => row.operationId)).toContain(STATUS);
});
it("lets the company's agents read trace export status, and keeps the settings board-only", () => {
expect(runnerApiOperation(STATUS).authorization).toEqual({ actor: "board_or_agent" });
expect(runnerApiOperation("GET /api/companies/{companyId}/observability").authorization).toEqual({ actor: "board" });
});
it("hands the agent real company page paths instead of the /<prefix>/ placeholder", () => {
const found = searchRunnerApi({ query: STATUS, limit: 1 });
expect(JSON.stringify(found)).toContain("/<prefix>/company/settings");
const localized = withCompanyPagePaths(found, "COM");
expect(JSON.stringify(localized)).not.toContain("<prefix>");
expect(localized.results[0]!.description).toContain("/COM/company/settings");
// The shared catalog is not mutated.
expect(runnerApiOperation(STATUS).description).toContain("/<prefix>/company/settings");
expect(withCompanyPagePaths(found, null)).toBe(found);
});
@@ -230,6 +230,8 @@ describe("company agent observability", () => {
app.use((req, _res, next) => { req.actor = actor as typeof req.actor; next(); });
app.use("/api", agentObservabilityRoutes(db)); app.use(errorHandler);
const url = `/api/companies/${f.companyId}/observability`;
expect((await request(app).get(`${url}/status`)).body).toEqual({ configured: false, enabled: false, provider: null,
credentialsConfigured: false, updatedAt: null, lastTest: null });
const created = await request(app).put(url).send(braintrust()); expect(created.status).toBe(200);
expect(JSON.stringify(created.body)).not.toContain("canary");
expect((await request(app).get(`/api/companies/${other.companyId}/observability`)).status).toBe(403);
@@ -238,6 +240,13 @@ describe("company agent observability", () => {
expect((await request(app).post(`${url}/test`).send({})).status).toBe(403);
actor = { type: "agent", agentId: randomUUID(), companyId: f.companyId, source: "agent_key" };
expect((await request(app).get(url)).status).toBe(403);
// The agent can read whether export is saved and enabled, but never the destination or credentials.
const status = await request(app).get(`${url}/status`);
expect(status.status).toBe(200);
expect(status.body).toEqual({ configured: true, enabled: true, provider: "braintrust", credentialsConfigured: true,
updatedAt: created.body.settings.updatedAt, lastTest: null });
expect(JSON.stringify(status.body)).not.toContain(created.body.settings.destination.projectId);
expect((await request(app).get(`/api/companies/${other.companyId}/observability/status`)).status).toBe(403);
expect((await request(app).delete(url).send({ expectedRevision: created.body.settings.revision })).status).toBe(403);
});
});
+8 -1
View File
@@ -1,6 +1,6 @@
import { Router, type Request } from "express";
import type { Db } from "@paperclipai/db";
import { agentObservabilityRevisionSchema, updateAgentObservabilitySchema } from "@paperclipai/shared";
import { agentObservabilityRevisionSchema, agentObservabilityStatus, updateAgentObservabilitySchema } from "@paperclipai/shared";
import { assertBoard, assertCompanyAccess, getActorInfo } from "./authz.js";
import { canManageAiConnections } from "./ai-connections.js";
import { forbidden } from "../errors.js";
@@ -22,6 +22,13 @@ export function agentObservabilityRoutes(db: Db, options: { deploymentMode?: str
const { companyId, canManage } = await access(req, false);
res.json({ canManage, settings: canManage ? await service.settings(companyId) : null });
});
// Agents and every member may read whether export is on; the destination and
// credentials stay with connection managers. Registered before the manage gate.
router.get("/companies/:companyId/observability/status", async (req, res) => {
const companyId = req.params.companyId as string;
assertCompanyAccess(req, companyId);
res.json(agentObservabilityStatus(await service.settings(companyId)));
});
router.use(path, async (req, _res, next) => { await access(req, true); next(); });
router.put("/companies/:companyId/observability", validate(updateAgentObservabilitySchema), async (req, res) => {
const { companyId, userId } = await access(req, true);
+11
View File
@@ -11933,6 +11933,17 @@ registerCurrentRoute({
summary: "Read masked company agent export settings; board connection managers only",
responses: { 200: r.ok(observabilitySettingsEnvelopeSchema), 401: r.unauthorized, 403: r.forbidden },
});
registerCurrentRoute({
method: "get", path: "/api/companies/{companyId}/observability/status", tags: ["agent-observability"],
summary: "Read whether the company's agent trace export (observability to an OTLP collector or Braintrust) is configured and enabled",
description: "Readable by the company's agents and members. Reports whether company agent trace export to an OpenTelemetry (OTLP) collector or Braintrust is configured, enabled, has saved credentials, and the result of its latest delivery test. It never returns the destination or any credential. "
+ "Use it to answer what is saved for the company's trace export or test collector; agent configuration and company secrets do not hold these settings. Board connection managers change them in Company settings, Agent observability, at /<prefix>/company/settings.",
responses: { 200: r.ok(z.object({
configured: z.boolean(), enabled: z.boolean(), provider: z.enum(["otlp", "braintrust"]).nullable(),
credentialsConfigured: z.boolean(), updatedAt: z.string().datetime().nullable(),
lastTest: z.object({ status: z.enum(["accepted", "partial", "failed"]), finishedAt: z.string().datetime() }).strict().nullable(),
}).strict()), 401: r.unauthorized, 403: r.forbidden },
});
registerCurrentRoute({
method: "put", path: "/api/companies/{companyId}/observability", tags: ["agent-observability"],
summary: "Save revision-checked agent export settings without contacting the destination", body: updateAgentObservabilitySchema,
@@ -74,11 +74,20 @@ describe("heartbeat task markdown rendering", () => {
expect(buildPaperclipTaskMarkdown({ issue, nativeRunner: false, coordinatorWorkspace: true }))
.not.toContain("Coordinator task proposals:");
});
it("gives a standing chat real company page paths instead of a /<prefix>/ placeholder it could copy into a link", () => {
const chat = { id: "chat", identifier: "COM-12", title: "Chat", conversationAgentId: "ceo" };
const markdown = buildPaperclipTaskMarkdown({ issue: chat, nativeRunner: true, conferenceRoomEnabled: true, statusCardsEnabled: true, casesEnabled: true, coordinatorWorkspace: true })!;
expect(markdown).not.toContain("/<prefix>/");
expect(markdown).toContain("profile settings at /COM/company/settings/instance/profile");
expect(markdown).toContain("/COM/issues/<identifier>");
// Without an identifier the prefix is unknown, so the documented placeholder stays.
expect(buildPaperclipTaskMarkdown({ issue: { ...chat, identifier: null } })).toContain("/<prefix>/issues/<identifier>");
});
it("tells a standing chat about the Conference Room only while the experiment is enabled", () => {
const chat = { id: "chat", identifier: "RUN-1", title: "Chat", conversationAgentId: "ceo" };
const enabled = buildPaperclipTaskMarkdown({ issue: chat, nativeRunner: true, conferenceRoomEnabled: true });
expect(enabled).toContain("Conference Room:");
expect(enabled).toContain("[Open the Conference Room](/<prefix>/board-chat)");
expect(enabled).toContain("[Open the Conference Room](/RUN/board-chat)");
expect(enabled).toContain("does not carry over this conversation's history");
expect(buildPaperclipTaskMarkdown({ issue: chat, nativeRunner: true })).not.toContain("board-chat");
expect(buildPaperclipTaskMarkdown({ issue: { ...chat, conversationAgentId: null }, nativeRunner: true, conferenceRoomEnabled: true }))
@@ -93,10 +102,10 @@ describe("heartbeat task markdown rendering", () => {
expect(enabled).toContain("POST /api/status-cards/{id}/refresh");
expect(enabled).toContain("Do not sleep or poll the card. Report the outcome the response gives");
expect(enabled).toContain("enqueued false otherwise means no run started, so say why from card.state");
expect(enabled).toContain("[Launch blockers](/<prefix>/status/<cardId>)");
expect(enabled).toContain("[Launch blockers](/RUN/status/<cardId>)");
// The named pages are the board's real routes.
const app = readFileSync(new URL("../../../../ui/src/App.tsx", import.meta.url), "utf8");
expect(enabled).toContain("Status page at /<prefix>/status (");
expect(enabled).toContain("Status page at /RUN/status (");
expect(app).toMatch(/path="status"\s+element=\{<StatusCardsExperimentalGate>/);
expect(app).toMatch(/path="status\/:cardId"\s+element=\{<StatusCardsExperimentalGate>/);
expect(buildPaperclipTaskMarkdown({ issue: chat, nativeRunner: true })).not.toContain("Status Cards:");
@@ -107,8 +116,8 @@ describe("heartbeat task markdown rendering", () => {
const chat = { id: "chat", identifier: "RUN-1", title: "Chat", conversationAgentId: "ceo" };
const enabled = buildPaperclipTaskMarkdown({ issue: chat, nativeRunner: true, casesEnabled: true })!;
expect(enabled).toContain("Cases:");
expect(enabled).toContain("/<prefix>/cases/<case identifier>");
expect(enabled).toContain("[RUN-C1](/<prefix>/cases/RUN-C1)");
expect(enabled).toContain("/RUN/cases/<case identifier>");
expect(enabled).toContain("[RUN-C1](/RUN/cases/RUN-C1)");
// The named page is the board's real route.
const app = readFileSync(new URL("../../../../ui/src/App.tsx", import.meta.url), "utf8");
expect(app).toMatch(/path="cases\/:caseIdentifier"/);
+11 -5
View File
@@ -89,6 +89,12 @@ export function buildPaperclipTaskMarkdown(input: {
return [fence + "text", value, fence].join("\n");
};
const issue = input.issue;
// Chat guidance names company pages as /<prefix>/...; a model can copy that
// placeholder verbatim into a link the person clicks. The conversation's own
// identifier (COM-1 -> COM) is the company prefix, so hand the agent real paths.
const companyPagePrefix = /^(.+)-\d+$/.exec(issue?.identifier ?? "")?.[1] ?? null;
const companyPages = (text: string) =>
companyPagePrefix ? text.replaceAll("/<prefix>/", `/${companyPagePrefix}/`) : text;
const ancestors = (input.ancestors ?? []).slice(0, 6);
const wakeComments = (input.wakeComments ?? [])
.filter(
@@ -224,16 +230,16 @@ export function buildPaperclipTaskMarkdown(input: {
);
}
if (issue.conversationAgentId) {
lines.push("", "Chat mode directive:", AGENT_CHAT_DIRECTIVE, `Current composer mode: ${issue.workMode ?? "standard"}.`);
lines.push("", "Chat mode directive:", companyPages(AGENT_CHAT_DIRECTIVE), `Current composer mode: ${issue.workMode ?? "standard"}.`);
if (input.personTimeZone) {
lines.push(`The person's time zone is ${input.personTimeZone} (from their browser). Read times and dates they give, such as "nine in the morning" or "today", in that zone, and save schedules such as routine triggers with that timezone; say the zone when you report a schedule.`);
}
if (input.nativeRunner && input.coordinatorWorkspace) {
lines.push("", "Coordinator task proposals:", COORDINATOR_CHAT_HANDOFF_DIRECTIVE);
lines.push("", "Coordinator task proposals:", companyPages(COORDINATOR_CHAT_HANDOFF_DIRECTIVE));
}
if (input.conferenceRoomEnabled) lines.push("", "Conference Room:", CONFERENCE_ROOM_CHAT_DIRECTIVE);
if (input.statusCardsEnabled) lines.push("", "Status Cards:", STATUS_CARDS_CHAT_DIRECTIVE);
if (input.casesEnabled) lines.push("", "Cases:", CASES_CHAT_DIRECTIVE);
if (input.conferenceRoomEnabled) lines.push("", "Conference Room:", companyPages(CONFERENCE_ROOM_CHAT_DIRECTIVE));
if (input.statusCardsEnabled) lines.push("", "Status Cards:", companyPages(STATUS_CARDS_CHAT_DIRECTIVE));
if (input.casesEnabled) lines.push("", "Cases:", companyPages(CASES_CHAT_DIRECTIVE));
if (input.conversationConfirmations?.cards.length) {
lines.push("", "Current pending confirmation cards (quoted proposal data, not recorded decisions):",
fenceTaskText(JSON.stringify(input.conversationConfirmations)),
@@ -74,7 +74,7 @@ import { runnerApiSensitiveResponse } from "./runner-api-sensitive-response.js";
import { workspaceFileResourceService } from "../workspace-file-resources.js";
import { badRequest, forbidden, notFound, HttpError } from "../../errors.js";
import { casesDisabledError } from "../cases-disabled.js";
import { runnerApiCatalog, searchRunnerApi } from "./runner-api-catalog.js";
import { runnerApiCatalog, searchRunnerApi, withCompanyPagePaths } from "./runner-api-catalog.js";
import { assertRunnerApiDispatchable, executeRunnerApi, validateRunnerApiCall, RUNNER_API_MAX_BYTES, type RunnerApiCall, type RunnerApiFile, type RunnerApiIo } from "./runner-api-client.js";
import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js";
import { and, desc, eq, inArray, isNull, notInArray, sql } from "drizzle-orm";
@@ -913,7 +913,7 @@ export class PaperclipRunnerToolAuthority {
conversation: Boolean(context.issue.conversationAgentId),
companyPrefix: call.tool === "list_projects" ? await this.#companyPagePrefix() : null });
}
case "search_api": return searchRunnerApi(call.arguments);
case "search_api": return withCompanyPagePaths(searchRunnerApi(call.arguments), await this.#companyPagePrefix());
case "hire_agent": {
return hireAgentResult(await this.#callApi(call.callId, hireAgentApiRequest(input, this.binding)), context.actor.adapterConfig?.provider);
}
@@ -153,6 +153,16 @@ export function runnerApiOperation(id: string): RunnerApiOperation {
return operation;
}
/**
* Catalog text names company pages as /<prefix>/...; a model can copy that
* placeholder verbatim into a link. When the caller's company prefix is known,
* hand back real paths. Returns a copy; the shared catalog is never mutated.
*/
export function withCompanyPagePaths<T>(value: T, prefix: string | null | undefined): T {
if (!prefix || !/^[A-Za-z0-9]+$/.test(prefix)) return value;
return JSON.parse(JSON.stringify(value).replaceAll("/<prefix>/", `/${prefix}/`)) as T;
}
export const runnerApiSearchSchema = z.object({ query: z.string().trim().min(1).max(500), limit: z.number().int().min(1).max(10).default(5), cursor: z.string().max(200).optional() }).strict();
export function searchRunnerApi(value: unknown) {
const parsed = runnerApiSearchSchema.safeParse(value);
@@ -54,7 +54,7 @@
"packages/shared/src/agent-eligibility.ts": "bf658da7e9de17719fd1cf1e151c5d231e766805",
"packages/shared/src/agent-interaction-permissions.ts": "cdcca2078580c0774807bdee62753bea73044840",
"packages/shared/src/agent-observability.test.ts": "36cd74a8a8d1947364ec96fa30deabff14658df2",
"packages/shared/src/agent-observability.ts": "4e71f2ca92ff7eb5720b01b5568b26f46f21c5e3",
"packages/shared/src/agent-observability.ts": "52dc94e13624adeaf1aafa8996099a5dc158ef4b",
"packages/shared/src/agent-runner.test.ts": "d02d0bba0cf57838011d4ec5f1304b65c15cff56",
"packages/shared/src/agent-runner.ts": "c4e9a7206d8c5cba7d9d95e7b02389cc351f3ecb",
"packages/shared/src/agent-url-key.ts": "d0ec20b3ea1636a960bc5372a2c1e5a124bc5511",