Files
PaperClipAI/server/src/__tests__/secrets-routes.test.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

1221 lines
44 KiB
TypeScript

import express from "express";
import request from "supertest";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { secretRoutes } from "../routes/secrets.js";
import { errorHandler } from "../middleware/error-handler.js";
import { HttpError, unprocessable } from "../errors.js";
const mockSecretService = vi.hoisted(() => ({
listProviders: vi.fn(),
checkProviders: vi.fn(),
listProviderConfigs: vi.fn(),
previewProviderConfigDiscovery: vi.fn(),
getProviderConfigById: vi.fn(),
createProviderConfig: vi.fn(),
updateProviderConfig: vi.fn(),
disableProviderConfig: vi.fn(),
removeProviderConfig: vi.fn(),
setDefaultProviderConfig: vi.fn(),
checkProviderConfigHealth: vi.fn(),
getById: vi.fn(),
list: vi.fn(),
getByKey: vi.fn(),
create: vi.fn(),
rotate: vi.fn(),
update: vi.fn(),
remove: vi.fn(),
listUserSecretDefinitions: vi.fn(),
createUserSecretDefinition: vi.fn(),
updateUserSecretDefinition: vi.fn(),
removeUserSecretDefinition: vi.fn(),
getUserSecretDefinitionCoverage: vi.fn(),
listCurrentUserSecretValues: vi.fn(),
createCurrentUserSecretValue: vi.fn(),
updateCurrentUserSecretValue: vi.fn(),
rotateCurrentUserSecretValue: vi.fn(),
removeCurrentUserSecretValue: vi.fn(),
previewRemoteImport: vi.fn(),
importRemoteSecrets: vi.fn(),
listBindingReferences: vi.fn(),
listAccessEvents: vi.fn(),
listAgentSecretAccess: vi.fn(),
resolveSecretValueForAgentAccess: vi.fn(),
}));
const mockLogActivity = vi.hoisted(() => vi.fn());
const mockAccessService = vi.hoisted(() => ({
decide: vi.fn(),
}));
vi.mock("../services/index.js", () => ({
accessService: () => mockAccessService,
secretService: () => mockSecretService,
logActivity: mockLogActivity,
}));
vi.mock("../services/access.js", () => ({
accessService: () => mockAccessService,
}));
function createApp(actor: Record<string, unknown> = {
type: "board",
userId: "user-1",
source: "session",
companyIds: ["company-1"],
memberships: [{ companyId: "company-1", status: "active", membershipRole: "admin" }],
}) {
const app = express();
app.use(express.json());
app.use((req, _res, next) => {
(req as any).actor = actor;
next();
});
app.use("/api", secretRoutes({} as any));
app.use(errorHandler);
return app;
}
describe("secret routes", () => {
beforeEach(() => {
for (const mock of Object.values(mockSecretService)) {
mock.mockReset();
}
mockLogActivity.mockReset();
mockAccessService.decide.mockReset();
mockAccessService.decide.mockResolvedValue({
allowed: true,
reason: "allow_standard_agent",
explanation: "Allowed by test policy",
});
});
it("returns an opaque secretRef in agent secret metadata without internal binding details", async () => {
const secretId = "11111111-1111-4111-8111-111111111111";
mockSecretService.listAgentSecretAccess.mockResolvedValue([{
secretId,
bindingId: "22222222-2222-4222-8222-222222222222",
configPath: "env.OPENAI_API_KEY",
key: "openai_api_key",
name: "OpenAI API key",
description: "Used for model access",
delivery: "env",
projectionClass: "unclassified",
latestVersion: 3,
versionSelector: "latest",
resolvedVersion: 3,
}]);
const res = await request(createApp({
type: "agent",
agentId: "33333333-3333-4333-8333-333333333333",
companyId: "44444444-4444-4444-8444-444444444444",
runId: "55555555-5555-4555-8555-555555555555",
source: "agent_jwt",
keyScope: { kind: "standard" },
})).get("/api/agents/me/secrets");
expect(res.status).toBe(200);
expect(res.body.secrets).toEqual([{
secretRef: secretId,
key: "openai_api_key",
name: "OpenAI API key",
description: "Used for model access",
delivery: "env",
projectionClass: "unclassified",
latestVersion: 3,
versionSelector: "latest",
resolvedVersion: 3,
}]);
expect(res.body.secrets[0]).not.toHaveProperty("secretId");
expect(res.body.secrets[0]).not.toHaveProperty("bindingId");
expect(res.body.secrets[0]).not.toHaveProperty("configPath");
});
it("returns provider health checks for board callers with company access", async () => {
mockSecretService.checkProviders.mockResolvedValue([
{
provider: "local_encrypted",
status: "ok",
message: "Local encrypted provider configured",
backupGuidance: ["Back up the key file together with database backups."],
},
]);
const res = await request(createApp()).get("/api/companies/company-1/secret-providers/health");
expect(res.status).toBe(200);
expect(res.body).toEqual({
providers: [
{
provider: "local_encrypted",
status: "ok",
message: "Local encrypted provider configured",
backupGuidance: ["Back up the key file together with database backups."],
},
],
});
});
it.each([undefined, "", "agent-supplied-value"])("denies agent secret creation before validating value %j", async (value) => {
const res = await request(createApp({
type: "agent", agentId: "agent-1", companyId: "company-1", source: "agent_jwt",
})).post("/api/companies/company-1/secrets").send({ name: "Deployment key", value });
expect(res.status).toBe(403);
expect(res.body.error).toContain("Settings → Secrets");
expect(res.body.error).toContain("human");
expect(res.body.error).toContain("Do not send secret values in chat");
expect(res.body).not.toHaveProperty("details");
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("gives a denied agent safe response and saved-wait options without echoing the submitted value", async () => {
const value = "private-denial-canary-value";
const res = await request(createApp({
type: "agent", agentId: "agent-1", companyId: "company-1", source: "agent_jwt",
})).post("/api/companies/company-1/secrets").send({ name: "Deployment key", value });
expect(res.status).toBe(403);
expect(res.body.error).toContain("finish the current conversation response");
expect(res.body.error).toContain("without claiming that credential setup is complete");
expect(res.body.error).toContain("first create a non-governing secure-save acknowledgment");
expect(res.body.error).toContain("request_human_input");
expect(res.body.error).toContain('interactionKind="confirmation"');
expect(res.body.error).not.toContain("using request_confirmation");
expect(res.body.error).toContain("never ask for the secret value in that card");
expect(res.body.error).toContain("Wait only on the actual saved request");
expect(JSON.stringify(res.body)).not.toContain(value);
expect(res.body).not.toHaveProperty("details");
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it.each([undefined, ""])("validates a board caller's missing or empty managed value %j", async (value) => {
const res = await request(createApp()).post("/api/companies/company-1/secrets")
.send({ name: "Deployment key", value });
expect(res.status).toBe(400);
expect(res.body.details).toEqual(expect.arrayContaining([expect.objectContaining({ path: ["value"] })]));
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("keeps authorized board creation and its human activity attribution", async () => {
mockSecretService.create.mockResolvedValue({ id: "secret-1", name: "Deployment key", provider: "local_encrypted" });
const res = await request(createApp()).post("/api/companies/company-1/secrets")
.send({ name: "Deployment key", value: "board-supplied-value" });
expect(res.status).toBe(201);
expect(mockSecretService.create).toHaveBeenCalledWith("company-1",
expect.objectContaining({ name: "Deployment key", value: "board-supplied-value" }),
{ userId: "user-1", agentId: null });
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
companyId: "company-1", actorType: "user", actorId: "user-1", action: "secret.created", entityId: "secret-1",
}));
});
it.each([undefined, "board-supplied-value"])("denies cross-company board creation before body validation %j", async (value) => {
const res = await request(createApp()).post("/api/companies/company-2/secrets")
.send({ name: "Deployment key", value });
expect(res.status).toBe(403);
expect(mockSecretService.create).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("rejects managed secret creation when externalRef is supplied", async () => {
const res = await request(createApp()).post("/api/companies/company-1/secrets").send({
name: "OpenAI API Key",
managedMode: "paperclip_managed",
value: "secret-value",
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:shared/other",
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/Managed secrets cannot set externalRef/);
expect(mockSecretService.create).not.toHaveBeenCalled();
});
it("returns sanitized AWS provider errors when managed secret creation fails", async () => {
mockSecretService.create.mockRejectedValue(
new HttpError(
403,
"AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.",
{
code: "access_denied",
provider: "aws_secrets_manager",
operation: "secret.create",
providerConfigId: "11111111-1111-4111-8111-111111111111",
region: "us-east-1",
credentialPath: "Paperclip server runtime/provider credential path",
requiredCapability: "secretsmanager:CreateSecret",
actionableMessage:
"AWS managed secret creation needs secretsmanager:CreateSecret in the selected region for this provider vault.",
safeAlternative:
"If the secret already exists in AWS, link it as an external reference instead of creating a Paperclip-managed value.",
},
),
);
const res = await request(createApp()).post("/api/companies/company-1/secrets").send({
name: "Vercel token",
key: "vercel_token",
provider: "aws_secrets_manager",
providerConfigId: "11111111-1111-4111-8111-111111111111",
managedMode: "paperclip_managed",
value: "vcp_test",
});
expect(res.status).toBe(403);
expect(res.body).toMatchObject({
code: "access_denied",
error: "AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.",
details: {
code: "access_denied",
provider: "aws_secrets_manager",
operation: "secret.create",
providerConfigId: "11111111-1111-4111-8111-111111111111",
region: "us-east-1",
requiredCapability: "secretsmanager:CreateSecret",
},
});
expect(JSON.stringify(res.body)).not.toContain("arn:aws");
expect(JSON.stringify(res.body)).not.toContain("123456789012");
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("restricts user secret definition management to company admins", async () => {
const res = await request(createApp({
type: "board",
userId: "user-1",
source: "session",
companyIds: ["company-1"],
memberships: [{ companyId: "company-1", status: "active", membershipRole: "member" }],
})).post("/api/companies/company-1/user-secret-definitions").send({
key: "github_token",
name: "GitHub token",
provider: "local_encrypted",
});
expect(res.status).toBe(403);
expect(mockSecretService.createUserSecretDefinition).not.toHaveBeenCalled();
});
it("records implicit user-secret definition admins as system actors instead of board pseudo-users", async () => {
mockSecretService.createUserSecretDefinition.mockResolvedValue({
id: "definition-1",
companyId: "company-1",
key: "github_token",
name: "GitHub token",
provider: "local_encrypted",
status: "active",
});
const res = await request(createApp({
type: "board",
source: "local_implicit",
isInstanceAdmin: true,
companyIds: ["company-1"],
memberships: [],
})).post("/api/companies/company-1/user-secret-definitions").send({
key: "github_token",
name: "GitHub token",
provider: "local_encrypted",
});
expect(res.status).toBe(201);
expect(mockSecretService.createUserSecretDefinition).toHaveBeenCalledWith(
"company-1",
expect.objectContaining({ key: "github_token" }),
{ userId: null, agentId: null },
);
expect(mockLogActivity).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({
actorType: "system",
actorId: "local_implicit",
action: "user_secret_definition.created",
}),
);
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("\"board\"");
});
it("logs patched user-secret definition deletion as deletion activity", async () => {
mockSecretService.updateUserSecretDefinition.mockResolvedValue({
id: "definition-1",
companyId: "company-1",
key: "github_token__deleted__definition-1",
name: "GitHub token",
provider: "local_encrypted",
status: "deleted",
});
const res = await request(createApp())
.patch("/api/companies/company-1/user-secret-definitions/definition-1")
.send({ status: "deleted" });
expect(res.status).toBe(200);
expect(mockSecretService.updateUserSecretDefinition).toHaveBeenCalledWith(
"company-1",
"definition-1",
expect.objectContaining({ status: "deleted" }),
{ userId: "user-1", agentId: null },
);
expect(mockLogActivity).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({
action: "user_secret_definition.deleted",
entityType: "user_secret_definition",
entityId: "definition-1",
}),
);
});
it("does not turn agent discovery of personal entry into human identity authority", async () => {
const app = createApp({ type: "agent", agentId: "agent-1", companyId: "company-1" });
expect((await request(app).get("/api/companies/company-1/user-secret-definitions")).status).toBe(403);
expect((await request(app).get("/api/companies/company-1/me/user-secrets")).status).toBe(403);
expect((await request(app).post("/api/companies/company-1/me/user-secrets").send({
definitionKey: "github_token", value: "synthetic-value",
})).status).toBe(403);
expect(mockSecretService.listUserSecretDefinitions).not.toHaveBeenCalled();
expect(mockSecretService.createCurrentUserSecretValue).not.toHaveBeenCalled();
});
it("creates current-user secret values for the authenticated user only", async () => {
mockSecretService.createCurrentUserSecretValue.mockResolvedValue({
id: "secret-1",
companyId: "company-1",
scope: "user",
ownerUserId: "user-1",
userSecretDefinitionId: "definition-1",
provider: "local_encrypted",
latestVersion: 1,
});
const res = await request(createApp()).post("/api/companies/company-1/me/user-secrets").send({
definitionKey: "github_token",
value: "secret-value",
});
expect(res.status).toBe(201);
expect(mockSecretService.createCurrentUserSecretValue).toHaveBeenCalledWith(
"company-1",
"user-1",
{
definitionKey: "github_token",
definitionId: undefined,
value: "secret-value",
externalRef: undefined,
providerVersionRef: undefined,
providerConfigId: undefined,
},
{ userId: "user-1", agentId: null },
);
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("secret-value");
});
it("rejects current-user secret values without a concrete user identity", async () => {
const res = await request(createApp({
type: "board",
source: "local_implicit",
companyIds: ["company-1"],
memberships: [{ companyId: "company-1", status: "active", membershipRole: "admin" }],
})).post("/api/companies/company-1/me/user-secrets").send({
definitionKey: "github_token",
value: "secret-value",
});
expect(res.status).toBe(401);
expect(res.body).toMatchObject({ error: "User identity required for user-specific secrets" });
expect(mockSecretService.createCurrentUserSecretValue).not.toHaveBeenCalled();
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("rejects empty current-user secret rotation payloads", async () => {
const res = await request(createApp())
.post("/api/companies/company-1/me/user-secrets/secret-1/rotate")
.send({});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/requires value, externalRef/);
expect(mockSecretService.rotateCurrentUserSecretValue).not.toHaveBeenCalled();
});
it("hides user-scoped secrets from legacy company secret mutation routes", async () => {
mockSecretService.getById.mockResolvedValue({
id: "secret-1",
companyId: "company-1",
scope: "user",
ownerUserId: "user-2",
status: "active",
});
const res = await request(createApp()).post("/api/secrets/secret-1/rotate").send({
value: "new-secret-value",
});
expect(res.status).toBe(404);
expect(mockSecretService.rotate).not.toHaveBeenCalled();
});
it("rejects provider vault routes for non-board actors", async () => {
const res = await request(createApp({
type: "agent",
agentId: "agent-1",
companyId: "company-1",
})).get("/api/companies/company-1/secret-provider-configs");
expect(res.status).toBe(403);
expect(mockSecretService.listProviderConfigs).not.toHaveBeenCalled();
});
it("rejects provider vault cross-company access before calling the service", async () => {
const res = await request(createApp({
type: "board",
userId: "user-1",
source: "session",
companyIds: ["company-2"],
memberships: [{ companyId: "company-2", status: "active", membershipRole: "admin" }],
})).get("/api/companies/company-1/secret-provider-configs");
expect(res.status).toBe(403);
expect(mockSecretService.listProviderConfigs).not.toHaveBeenCalled();
});
it("rejects provider vault discovery preview for non-board actors", async () => {
const res = await request(createApp({
type: "agent",
agentId: "agent-1",
companyId: "company-1",
}))
.post("/api/companies/company-1/secret-provider-configs/discovery/preview")
.send({
provider: "aws_secrets_manager",
config: { region: "us-east-1" },
});
expect(res.status).toBe(403);
expect(mockSecretService.previewProviderConfigDiscovery).not.toHaveBeenCalled();
});
it("rejects sensitive provider vault config fields", async () => {
const res = await request(createApp()).post("/api/companies/company-1/secret-provider-configs").send({
provider: "aws_secrets_manager",
displayName: "AWS prod",
config: {
region: "us-east-1",
accessKeyId: "AKIA...",
},
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/sensitive field/i);
expect(mockSecretService.createProviderConfig).not.toHaveBeenCalled();
});
it("rejects sensitive provider vault discovery draft config fields", async () => {
const res = await request(createApp())
.post("/api/companies/company-1/secret-provider-configs/discovery/preview")
.send({
provider: "aws_secrets_manager",
config: {
region: "us-east-1",
secretAccessKey: "secret",
},
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/sensitive field/i);
expect(mockSecretService.previewProviderConfigDiscovery).not.toHaveBeenCalled();
});
it("previews provider vault discovery and logs only aggregate metadata", async () => {
mockSecretService.previewProviderConfigDiscovery.mockResolvedValue({
provider: "aws_secrets_manager",
nextToken: null,
sampledSecretCount: 2,
skippedForeignPaperclipSampleCount: 0,
candidates: [
{
provider: "aws_secrets_manager",
displayName: "AWS production",
config: {
region: "us-east-1",
namespace: "prod-use1",
secretNamePrefix: "paperclip",
environmentTag: "production",
ownerTag: "platform",
kmsKeyId: null,
},
sampleCount: 2,
samples: [
{ name: "paperclip/prod-use1/company-1/openai", hasKmsKey: false, tagKeys: ["environment"] },
],
signals: {
namespace: "prod-use1",
secretNamePrefix: "paperclip",
environmentTag: "production",
ownerTag: "platform",
kmsKeyId: null,
hasKmsKey: false,
sampleCount: 2,
paperclipManagedSampleCount: 0,
skippedForeignPaperclipSampleCount: 0,
},
warnings: [],
},
],
warnings: [],
});
const res = await request(createApp())
.post("/api/companies/company-1/secret-provider-configs/discovery/preview")
.send({
provider: "aws_secrets_manager",
config: { region: "us-east-1" },
query: "paperclip",
pageSize: 25,
});
expect(res.status).toBe(200);
expect(mockSecretService.previewProviderConfigDiscovery).toHaveBeenCalledWith("company-1", {
provider: "aws_secrets_manager",
config: { region: "us-east-1" },
query: "paperclip",
nextToken: undefined,
pageSize: 25,
});
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
action: "secret_provider_config.discovery_previewed",
entityType: "secret_provider_config_discovery",
entityId: "company-1",
details: {
provider: "aws_secrets_manager",
candidateCount: 1,
sampledSecretCount: 2,
warningCount: 0,
},
}));
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("paperclip/prod-use1/company-1/openai");
});
it("returns actionable sanitized provider vault discovery errors", async () => {
mockSecretService.previewProviderConfigDiscovery.mockRejectedValue(
new HttpError(
403,
"AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.",
{
code: "access_denied",
provider: "aws_secrets_manager",
operation: "secret_provider_config.discovery.preview",
providerConfigId: "discovery-preview",
providerVaultContext: "draft_config",
region: "us-east-1",
credentialPath: "Paperclip server runtime/provider credential path",
requiredCapability: "secretsmanager:ListSecrets",
actionableMessage:
"AWS discovery preview needs secretsmanager:ListSecrets in the selected region for the Paperclip server runtime/provider credential path.",
safeAlternative:
"If the operator already knows the exact AWS Secrets Manager ARN, paste/link that ARN instead of using discovery. Exact-resource DescribeSecret and runtime read permissions are still required.",
},
),
);
const res = await request(createApp())
.post("/api/companies/company-1/secret-provider-configs/discovery/preview")
.send({
provider: "aws_secrets_manager",
config: { region: "us-east-1" },
pageSize: 25,
});
expect(res.status).toBe(403);
expect(res.body).toMatchObject({
error: "AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.",
details: {
code: "access_denied",
provider: "aws_secrets_manager",
operation: "secret_provider_config.discovery.preview",
providerVaultContext: "draft_config",
region: "us-east-1",
requiredCapability: "secretsmanager:ListSecrets",
},
});
expect(res.body.details.actionableMessage).toContain("Paperclip server runtime/provider credential path");
expect(res.body.details.safeAlternative).toContain("paste/link that ARN");
expect(JSON.stringify(res.body)).not.toContain("arn:aws");
expect(JSON.stringify(res.body)).not.toContain("123456789012");
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("rejects ready status for coming-soon provider vaults", async () => {
const res = await request(createApp()).post("/api/companies/company-1/secret-provider-configs").send({
provider: "vault",
displayName: "Vault draft",
status: "ready",
config: {
address: "https://vault.example.com",
},
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/locked while coming soon/i);
expect(mockSecretService.createProviderConfig).not.toHaveBeenCalled();
});
it("rejects credential-bearing Vault provider vault addresses before persistence", async () => {
const res = await request(createApp()).post("/api/companies/company-1/secret-provider-configs").send({
provider: "vault",
displayName: "Vault draft",
config: {
address: "https://user:pass@vault.example.com",
},
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/origin-only HTTP\(S\) URL/i);
expect(mockSecretService.createProviderConfig).not.toHaveBeenCalled();
});
it.each([
"https://vault.example.com?token=hvs.x",
"https://vault.example.com#token=hvs.x",
])("rejects token-bearing Vault provider vault address %s before persistence", async (address) => {
const res = await request(createApp()).post("/api/companies/company-1/secret-provider-configs").send({
provider: "vault",
displayName: "Vault draft",
config: { address },
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/origin-only HTTP\(S\) URL/i);
expect(mockSecretService.createProviderConfig).not.toHaveBeenCalled();
});
it("rejects unsafe Vault provider vault address patches before persistence", async () => {
const res = await request(createApp()).patch("/api/secret-provider-configs/vault-1").send({
config: {
address: "https://vault.example.com#token=hvs.x",
},
});
expect(res.status).toBe(400);
expect(JSON.stringify(res.body)).toMatch(/origin-only HTTP\(S\) URL/i);
expect(mockSecretService.getProviderConfigById).not.toHaveBeenCalled();
expect(mockSecretService.updateProviderConfig).not.toHaveBeenCalled();
});
it("creates provider vaults and logs safe activity details", async () => {
const createdAt = new Date("2026-05-06T00:00:00.000Z");
mockSecretService.createProviderConfig.mockResolvedValue({
id: "11111111-1111-4111-8111-111111111111",
companyId: "company-1",
provider: "aws_secrets_manager",
displayName: "AWS prod",
status: "ready",
isDefault: true,
config: { region: "us-east-1" },
healthStatus: null,
healthCheckedAt: null,
healthMessage: null,
healthDetails: null,
disabledAt: null,
createdByAgentId: null,
createdByUserId: "user-1",
createdAt,
updatedAt: createdAt,
});
const res = await request(createApp()).post("/api/companies/company-1/secret-provider-configs").send({
provider: "aws_secrets_manager",
displayName: "AWS prod",
isDefault: true,
config: { region: "us-east-1" },
});
expect(res.status).toBe(201);
expect(mockSecretService.createProviderConfig).toHaveBeenCalledWith(
"company-1",
{
provider: "aws_secrets_manager",
displayName: "AWS prod",
status: undefined,
isDefault: true,
config: { region: "us-east-1" },
},
{ userId: "user-1", agentId: null },
);
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
action: "secret_provider_config.created",
details: {
provider: "aws_secrets_manager",
displayName: "AWS prod",
status: "ready",
isDefault: true,
},
}));
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("accessKey");
});
it("removes provider vault config locally without deleting remote provider data", async () => {
const createdAt = new Date("2026-05-06T00:00:00.000Z");
const providerConfig = {
id: "11111111-1111-4111-8111-111111111111",
companyId: "company-1",
provider: "aws_secrets_manager",
displayName: "AWS prod",
status: "ready",
isDefault: false,
config: { region: "us-east-1" },
healthStatus: null,
healthCheckedAt: null,
healthMessage: null,
healthDetails: null,
disabledAt: null,
createdByAgentId: null,
createdByUserId: "user-1",
createdAt,
updatedAt: createdAt,
};
mockSecretService.getProviderConfigById.mockResolvedValue(providerConfig);
mockSecretService.removeProviderConfig.mockResolvedValue(providerConfig);
const res = await request(createApp()).delete(
"/api/secret-provider-configs/11111111-1111-4111-8111-111111111111",
);
expect(res.status).toBe(200);
expect(mockSecretService.removeProviderConfig).toHaveBeenCalledWith(
"11111111-1111-4111-8111-111111111111",
);
expect(mockSecretService.disableProviderConfig).not.toHaveBeenCalled();
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
action: "secret_provider_config.removed",
details: {
provider: "aws_secrets_manager",
displayName: "AWS prod",
remoteDeleted: false,
},
}));
});
it("rejects remote import preview for non-board actors", async () => {
const res = await request(createApp({
type: "agent",
agentId: "agent-1",
companyId: "company-1",
})).post("/api/companies/company-1/secrets/remote-import/preview").send({
providerConfigId: "11111111-1111-4111-8111-111111111111",
});
expect(res.status).toBe(403);
expect(mockSecretService.previewRemoteImport).not.toHaveBeenCalled();
});
it("previews remote imports and logs only aggregate metadata", async () => {
mockSecretService.previewRemoteImport.mockResolvedValue({
providerConfigId: "11111111-1111-4111-8111-111111111111",
provider: "aws_secrets_manager",
nextToken: null,
candidates: [
{
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/openai",
remoteName: "prod/openai",
name: "openai",
key: "openai",
providerVersionRef: null,
providerMetadata: { description: "OpenAI API key" },
status: "ready",
importable: true,
conflicts: [],
},
],
});
const res = await request(createApp())
.post("/api/companies/company-1/secrets/remote-import/preview")
.send({
providerConfigId: "11111111-1111-4111-8111-111111111111",
query: "openai",
pageSize: 25,
});
expect(res.status).toBe(200);
expect(mockSecretService.previewRemoteImport).toHaveBeenCalledWith("company-1", {
providerConfigId: "11111111-1111-4111-8111-111111111111",
query: "openai",
nextToken: undefined,
pageSize: 25,
});
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
action: "secret.remote_import.previewed",
details: {
provider: "aws_secrets_manager",
candidateCount: 1,
readyCount: 1,
duplicateCount: 0,
conflictCount: 0,
},
}));
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("prod/openai");
});
it("returns sanitized remote import preview provider errors", async () => {
mockSecretService.previewRemoteImport.mockRejectedValue(
new HttpError(
403,
"AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.",
{ code: "access_denied" },
),
);
const res = await request(createApp())
.post("/api/companies/company-1/secrets/remote-import/preview")
.send({
providerConfigId: "11111111-1111-4111-8111-111111111111",
});
expect(res.status).toBe(403);
expect(res.body).toEqual({
code: "access_denied",
error: "AWS Secrets Manager denied the request. Check IAM permissions for this provider vault.",
details: { code: "access_denied" },
});
expect(JSON.stringify(res.body)).not.toContain("arn:aws");
expect(JSON.stringify(res.body)).not.toContain("123456789012");
expect(mockLogActivity).not.toHaveBeenCalled();
});
it("imports remote references and logs aggregate row counts", async () => {
mockSecretService.importRemoteSecrets.mockResolvedValue({
providerConfigId: "11111111-1111-4111-8111-111111111111",
provider: "aws_secrets_manager",
importedCount: 1,
skippedCount: 0,
errorCount: 0,
results: [
{
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/openai",
name: "OpenAI API key",
key: "openai-api-key",
status: "imported",
reason: null,
secretId: "22222222-2222-4222-8222-222222222222",
conflicts: [],
},
],
});
const res = await request(createApp())
.post("/api/companies/company-1/secrets/remote-import")
.send({
providerConfigId: "11111111-1111-4111-8111-111111111111",
secrets: [
{
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/openai",
name: "OpenAI API key",
key: "openai-api-key",
description: "Operator-entered Paperclip description",
},
],
});
expect(res.status).toBe(200);
expect(mockSecretService.importRemoteSecrets).toHaveBeenCalledWith(
"company-1",
{
providerConfigId: "11111111-1111-4111-8111-111111111111",
secrets: [
{
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/openai",
name: "OpenAI API key",
key: "openai-api-key",
description: "Operator-entered Paperclip description",
},
],
},
{ userId: "user-1", agentId: null },
);
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
action: "secret.remote_import.completed",
details: {
provider: "aws_secrets_manager",
importedCount: 1,
skippedCount: 0,
errorCount: 0,
},
}));
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("prod/openai");
});
it("surfaces update-route externalRef retarget rejection without logging raw refs", async () => {
mockSecretService.getById.mockResolvedValue({
id: "22222222-2222-4222-8222-222222222222",
companyId: "company-1",
name: "OpenAI API key",
key: "openai-api-key",
provider: "aws_secrets_manager",
managedMode: "external_reference",
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:shared/original",
});
mockSecretService.update.mockRejectedValue(
unprocessable("External reference secrets cannot be retargeted through generic update"),
);
const res = await request(createApp())
.patch("/api/secrets/22222222-2222-4222-8222-222222222222")
.send({
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:shared/repointed",
});
expect(res.status).toBe(422);
expect(mockSecretService.update).toHaveBeenCalledWith(
"22222222-2222-4222-8222-222222222222",
expect.objectContaining({
externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:shared/repointed",
}),
);
expect(mockLogActivity).not.toHaveBeenCalled();
expect(JSON.stringify(mockLogActivity.mock.calls)).not.toContain("shared/repointed");
});
it("returns 404 for cross-tenant GET /secrets/:id/usage without leaking existence", async () => {
mockSecretService.getById.mockResolvedValue({
id: "44444444-4444-4444-8444-444444444444",
companyId: "company-2",
name: "Other tenant secret",
key: "other-secret",
provider: "aws_secrets_manager",
managedMode: "paperclip_managed",
});
const crossTenantApp = createApp({
type: "board",
userId: "mallory",
source: "session",
companyIds: ["company-1"],
memberships: [{ companyId: "company-1", status: "active", membershipRole: "admin" }],
isInstanceAdmin: false,
});
const res = await request(crossTenantApp).get(
"/api/secrets/44444444-4444-4444-8444-444444444444/usage",
);
expect(res.status).toBe(404);
expect(res.body).toEqual({ error: "Secret not found" });
expect(mockSecretService.listBindingReferences).not.toHaveBeenCalled();
});
it("returns 404 for missing GET /secrets/:id/usage with identical response shape", async () => {
mockSecretService.getById.mockResolvedValue(null);
const res = await request(createApp()).get(
"/api/secrets/55555555-5555-4555-8555-555555555555/usage",
);
expect(res.status).toBe(404);
expect(res.body).toEqual({ error: "Secret not found" });
expect(mockSecretService.listBindingReferences).not.toHaveBeenCalled();
});
it("returns 404 for cross-tenant GET /secrets/:id/access-events without leaking existence", async () => {
mockSecretService.getById.mockResolvedValue({
id: "66666666-6666-4666-8666-666666666666",
companyId: "company-2",
name: "Other tenant secret",
key: "other-secret",
provider: "aws_secrets_manager",
managedMode: "paperclip_managed",
});
const crossTenantApp = createApp({
type: "board",
userId: "mallory",
source: "session",
companyIds: ["company-1"],
memberships: [{ companyId: "company-1", status: "active", membershipRole: "admin" }],
isInstanceAdmin: false,
});
const res = await request(crossTenantApp).get(
"/api/secrets/66666666-6666-4666-8666-666666666666/access-events",
);
expect(res.status).toBe(404);
expect(res.body).toEqual({ error: "Secret not found" });
expect(mockSecretService.listAccessEvents).not.toHaveBeenCalled();
});
it("returns 404 for missing GET /secrets/:id/access-events with identical response shape", async () => {
mockSecretService.getById.mockResolvedValue(null);
const res = await request(createApp()).get(
"/api/secrets/77777777-7777-4777-8777-777777777777/access-events",
);
expect(res.status).toBe(404);
expect(res.body).toEqual({ error: "Secret not found" });
expect(mockSecretService.listAccessEvents).not.toHaveBeenCalled();
});
it("returns usage bindings for in-tenant GET /secrets/:id/usage", async () => {
mockSecretService.getById.mockResolvedValue({
id: "88888888-8888-4888-8888-888888888888",
companyId: "company-1",
name: "OpenAI",
key: "openai",
provider: "aws_secrets_manager",
managedMode: "paperclip_managed",
});
mockSecretService.listBindingReferences.mockResolvedValue([]);
const res = await request(createApp()).get(
"/api/secrets/88888888-8888-4888-8888-888888888888/usage",
);
expect(res.status).toBe(200);
expect(res.body).toEqual({
secretId: "88888888-8888-4888-8888-888888888888",
bindings: [],
});
expect(mockSecretService.listBindingReferences).toHaveBeenCalledWith(
"company-1",
"88888888-8888-4888-8888-888888888888",
);
});
it("allows DELETE to retry cleanup for already soft-deleted secrets", async () => {
const secret = {
id: "33333333-3333-4333-8333-333333333333",
companyId: "company-1",
name: "OpenAI API Key__deleted__33333333-3333-4333-8333-333333333333",
key: "openai-api-key__deleted__33333333-3333-4333-8333-333333333333",
provider: "aws_secrets_manager",
managedMode: "paperclip_managed",
status: "deleted",
};
mockSecretService.getById.mockResolvedValue(secret);
mockSecretService.remove.mockResolvedValue(secret);
const res = await request(createApp()).delete(
"/api/secrets/33333333-3333-4333-8333-333333333333",
);
expect(res.status).toBe(200);
expect(res.body).toEqual({ ok: true });
expect(mockSecretService.remove).toHaveBeenCalledWith(
"33333333-3333-4333-8333-333333333333",
{ beforeDelete: expect.any(Function) },
);
expect(mockLogActivity).toHaveBeenCalledWith(
expect.anything(),
expect.objectContaining({
action: "secret.deleted",
companyId: "company-1",
entityId: secret.id,
}),
);
});
describe("GET /companies/:companyId/secrets/catalog", () => {
const fullSecrets = [
{
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
name: "MY_API_KEY",
key: "my_api_key",
status: "active",
companyId: "company-1",
provider: "local_encrypted",
providerMetadata: null,
referenceCount: 2,
},
{
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
name: "DB_PASSWORD",
key: "db_password",
status: "active",
companyId: "company-1",
provider: "local_encrypted",
providerMetadata: null,
referenceCount: 0,
},
];
it("returns id/name/key/status only for board callers", async () => {
mockSecretService.list.mockResolvedValue(fullSecrets);
const res = await request(createApp()).get("/api/companies/company-1/secrets/catalog");
expect(res.status).toBe(200);
expect(res.body).toEqual([
{ id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", name: "MY_API_KEY", key: "my_api_key", status: "active" },
{ id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb", name: "DB_PASSWORD", key: "db_password", status: "active" },
]);
expect(res.body[0]).not.toHaveProperty("provider");
expect(res.body[0]).not.toHaveProperty("referenceCount");
});
it("returns id/name/key/status only for agent callers in the same company", async () => {
mockSecretService.list.mockResolvedValue(fullSecrets);
const agentApp = createApp({
type: "agent",
agentId: "agent-1",
companyId: "company-1",
});
const res = await request(agentApp).get("/api/companies/company-1/secrets/catalog");
expect(res.status).toBe(200);
expect(res.body).toHaveLength(2);
expect(res.body[0]).toMatchObject({ id: expect.any(String), name: "MY_API_KEY", key: "my_api_key", status: "active" });
});
it.each(["skill_test", "task_bridge"])(
"rejects %s keys from the company secret catalog",
async (kind) => {
mockAccessService.decide.mockResolvedValue({
allowed: false,
reason: "deny_key_scope",
explanation: "Restricted keys cannot read the company secret catalog.",
});
const res = await request(createApp({
type: "agent",
agentId: "agent-1",
companyId: "company-1",
source: "agent_key",
keyScope: { kind },
})).get("/api/companies/company-1/secrets/catalog");
expect(res.status).toBe(403);
expect(mockSecretService.list).not.toHaveBeenCalled();
expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({
action: "secrets:read",
resource: { type: "company", companyId: "company-1" },
}));
},
);
it("rejects unauthenticated requests", async () => {
const res = await request(createApp({ type: "none" }))
.get("/api/companies/company-1/secrets/catalog");
// assertBoardOrAgent throws forbidden (403) for all non-agent/non-board actors;
// it does not call assertAuthenticated first, so type:"none" gets 403, not 401.
expect(res.status).toBe(403);
});
it("rejects agents from a different company", async () => {
const res = await request(createApp({
type: "agent",
agentId: "agent-1",
companyId: "company-2",
})).get("/api/companies/company-1/secrets/catalog");
expect(res.status).toBe(403);
});
});
});