mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for work
> - The Railway connection gives agents governed deployment tools, such as `list-deployments`
> - Railway Environment RBAC lets non-admin members see a restricted environment, but Railway denies reads of its services, deployments, and logs
> - The client maps this denial to `railway_api_authorization_required` and tells the user to reconnect or to use IDs from a consented workspace
> - That advice is wrong for this case: workspace consent is correct, and reconnecting does not change the result
> - This pull request identifies the restricted environment and reports the correct remedy
> - The benefit is that agents and operators get a clear, actionable error instead of an incorrect reconnect loop
## Linked Issues or Issue Description
(B) No public issue exists. Bug description:
**What happened?**
An agent called `paperclip-railway-list-deployments` for a service. In the same session, `list-projects`, `list-services`, and `list-environments` succeeded for the same project. The deployment-history call failed every time with `railway_api_authorization_required` and the message "Use IDs from a workspace selected during consent, or reconnect". The project-level reads prove that workspace consent is correct. The denied environment is restricted for the connected member, so the message sends the user to the wrong fix.
**Expected behavior**
When Railway denies the environment-scoped target check because the environment is restricted, the tool must report a distinct error. The error must say that a Railway project admin must grant access. It must not tell the user to reconnect.
**Steps to reproduce**
1. Connect Railway with a member (non-admin) account in a workspace that has a restricted environment.
2. Call `paperclip-railway-list-environments` for the project. The restricted environment is in the list.
3. Call `paperclip-railway-list-deployments` (or `service-status`) for a service in that environment.
4. Railway returns `Not Authorized`, and Paperclip reports a workspace consent error.
**Paperclip version or commit**
`master` at 8282d386da.
## What Changed
- `server/src/services/railway.ts`: When the target query fails with an authorization denial, the client reads the project's environments with `canAccess`. If the requested environment has `canAccess: false`, the client throws `railway_environment_restricted` (HTTP 403) with the admin-access remedy. In all other cases, the client throws the original error. If the access check fails, the client also throws the original error. The access check is a read query. No mutation runs after a denied target check.
- `server/src/services/railway.ts`: `list-environments` now returns `canAccess` for each environment, so agents can choose an accessible environment before they call a target tool.
- `server/src/__tests__/railway.test.ts`: Adds a regression test for the restricted, accessible, and unknown cases.
- `doc/connections/RAILWAY.md`: Adds a recovery entry for the new error code.
## Verification
- Failing before the fix: `cd server && npx vitest run src/__tests__/railway.test.ts`. The new test "reports a restricted environment instead of a workspace consent error when Railway denies the target" failed. It expected `code: "railway_environment_restricted"`, but it received `code: "railway_api_authorization_required"`. The other 28 tests passed.
- Passing after the fix: the same file passes (29/29).
- Related suites pass: `npx vitest run src/__tests__/railway.test.ts src/__tests__/railway-connection.test.ts src/__tests__/railway-ssh.test.ts src/__tests__/connector-tool-policy.test.ts` (4 files, 63 tests).
- Scoped `tsc --noEmit` with the server `tsconfig.json` for the changed files passes.
- Not run: the full server typecheck and `pnpm verify`. In the test sandbox, they used more than the 8 GB memory limit. CI must confirm them.
- Public reference for the provider behavior: https://docs.railway.com/enterprise/environment-rbac
## Risks
- Low risk. The change applies only after Railway has already denied the target check. It adds one read query in that case.
- If Railway removes or renames `canAccess`, the access check fails, and the client throws the original error, as before this change.
- The access check reads the first 100 environments of the project. If a project has more environments, the client throws the original error.
- `list-environments` output gets one more field.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
199 lines
15 KiB
TypeScript
199 lines
15 KiB
TypeScript
import { describe, expect, it, vi } from "vitest";
|
|
import { railwayCommandBudgetMs, createRailwayClient, discoverRailwayWorkspace, isRailwayConnection, isRailwayEndpoint, isRailwayToolBlocked, RAILWAY_API_URL, RAILWAY_MCP_URL, RAILWAY_QUERIES, RAILWAY_TOOLS, railwayRisk } from "../services/railway.js";
|
|
import { deploymentData, instanceId, target, targetData } from "./fixtures/railway/provider.js";
|
|
|
|
function fixture(responder?: (query: string) => Response | undefined) {
|
|
const request = vi.fn(async (_url: string, init: RequestInit) => {
|
|
const { query } = JSON.parse(String(init.body));
|
|
return responder?.(query) ?? Response.json({ data: query === RAILWAY_QUERIES.target ? targetData : query === RAILWAY_QUERIES.deployment ? deploymentData : { deploymentRestart: true } });
|
|
});
|
|
const runCommand = vi.fn(async () => ({ exitCode: 0, stdout: "ok", stderr: "" }));
|
|
const controller = new AbortController();
|
|
return { request, runCommand, controller, client: createRailwayClient({ authorization: "Bearer railway-fixture-secret", request, runCommand, signal: controller.signal }) };
|
|
}
|
|
|
|
describe("Railway governed operations", () => {
|
|
it("binds project listing and the access probe to an explicit workspace", async () => {
|
|
const f = fixture(() => Response.json({ data: { projects: { edges: [] } } }));
|
|
await expect(f.client.call("paperclip-railway-list-projects", {})).rejects.toMatchObject({ code: "railway_invalid_arguments" });
|
|
expect(f.request).not.toHaveBeenCalled();
|
|
await f.client.probe(target.projectId);
|
|
await f.client.call("paperclip-railway-list-projects", { workspaceId: target.projectId, first: 5 });
|
|
expect(f.request.mock.calls.map(([, init]) => JSON.parse(String(init.body)).variables)).toEqual([
|
|
{ workspaceId: target.projectId, first: 1 }, { workspaceId: target.projectId, first: 5 },
|
|
]);
|
|
expect(RAILWAY_QUERIES.projects).toContain("projects(workspaceId:$workspaceId,");
|
|
});
|
|
|
|
it.each(["structured", "sse"])("discovers workspace access from the hosted %s response without account profile scopes", async (format) => {
|
|
const data = { workspaces: [{ id: target.projectId }] };
|
|
const payload = { jsonrpc: "2.0", id: "paperclip-railway-workspace-probe", result: format === "structured" ? { structuredContent: data } : { content: [{ type: "text", text: JSON.stringify(data) }] } };
|
|
const request = vi.fn(async () => format === "structured" ? Response.json(payload) : new Response(`data: ${JSON.stringify(payload)}\n\n`, { headers: { "content-type": "text/event-stream" } }));
|
|
await expect(discoverRailwayWorkspace({ authorization: "Bearer fixture", request, signal: new AbortController().signal })).resolves.toBe(target.projectId);
|
|
expect(request).toHaveBeenCalledWith(RAILWAY_MCP_URL, expect.objectContaining({ redirect: "error", body: expect.stringContaining('"name":"list-workspaces"') }));
|
|
});
|
|
|
|
it("keeps operations unavailable for empty or failed workspace discovery without exposing provider output", async () => {
|
|
for (const result of [{ structuredContent: { workspaces: [] } }, { isError: true, content: [{ type: "text", text: "private provider details" }] }]) {
|
|
const request = vi.fn(async () => Response.json({ result }));
|
|
await expect(discoverRailwayWorkspace({ authorization: "Bearer fixture", request, signal: new AbortController().signal })).rejects.toThrow(/Railway/);
|
|
}
|
|
});
|
|
|
|
it("recognizes Railway's HTTP 200 authorization errors without echoing provider details", async () => {
|
|
const f = fixture(() => Response.json({ errors: [{ message: "Not Authorized", extensions: { code: "INTERNAL_SERVER_ERROR", private: "provider secret" } }], data: null }));
|
|
await expect(f.client.probe(target.projectId)).rejects.toMatchObject({ code: "railway_api_authorization_required", status: 403, message: expect.stringContaining("workspace selected during consent") });
|
|
expect(f.request).toHaveBeenCalledTimes(1);
|
|
});
|
|
it("reports a restricted environment instead of a workspace consent error when Railway denies the target", async () => {
|
|
const denied = () => Response.json({ errors: [{ message: "Not Authorized" }], data: null });
|
|
const other = "66666666-6666-4666-8666-666666666666";
|
|
const page = (nodes: { id: string; canAccess: boolean }[], endCursor?: string) => Response.json({ data: { project: { id: target.projectId, environments: { edges: nodes.map((node) => ({ node: { ...node, name: "env", projectId: target.projectId } })), pageInfo: { hasNextPage: !!endCursor, endCursor: endCursor ?? null } } } } });
|
|
const queries = (f: ReturnType<typeof fixture>) => f.request.mock.calls.map(([, init]) => JSON.parse(String(init.body)));
|
|
const { deploymentId: _, ...ids } = target;
|
|
|
|
const restricted = fixture((q) => q === RAILWAY_QUERIES.target ? denied() : undefined);
|
|
restricted.request.mockImplementation(async (_url, init) => {
|
|
const { query, variables } = JSON.parse(String(init.body));
|
|
if (query === RAILWAY_QUERIES.target) return denied();
|
|
return variables.after ? page([{ id: target.environmentId, canAccess: false }]) : page([{ id: other, canAccess: true }], "next");
|
|
});
|
|
await expect(restricted.client.call("paperclip-railway-list-deployments", ids)).rejects.toMatchObject({ code: "railway_environment_restricted", status: 403, message: expect.stringContaining("restricted") });
|
|
expect(queries(restricted).map((body) => [body.query, body.variables.after])).toEqual([[RAILWAY_QUERIES.target, undefined], [RAILWAY_QUERIES.environments, undefined], [RAILWAY_QUERIES.environments, "next"]]);
|
|
|
|
const http403 = fixture((q) => q === RAILWAY_QUERIES.target ? new Response(null, { status: 403 }) : page([{ id: target.environmentId, canAccess: false }]));
|
|
await expect(http403.client.call("paperclip-railway-service-status", ids)).rejects.toMatchObject({ code: "railway_environment_restricted" });
|
|
|
|
const accessible = fixture((q) => q === RAILWAY_QUERIES.target ? denied() : page([{ id: target.environmentId, canAccess: true }]));
|
|
await expect(accessible.client.call("paperclip-railway-service-status", ids)).rejects.toMatchObject({ code: "railway_api_authorization_required" });
|
|
const unknown = fixture((q) => q === RAILWAY_QUERIES.target || q === RAILWAY_QUERIES.environments ? denied() : undefined);
|
|
await expect(unknown.client.call("paperclip-railway-restart", target)).rejects.toMatchObject({ code: "railway_api_authorization_required" });
|
|
expect(queries(unknown).every((body) => !body.query.startsWith("mutation"))).toBe(true);
|
|
|
|
const revoked = fixture((q) => q === RAILWAY_QUERIES.target ? new Response(null, { status: 401 }) : page([{ id: target.environmentId, canAccess: false }]));
|
|
await expect(revoked.client.call("paperclip-railway-service-status", ids)).rejects.toMatchObject({ code: "railway_api_authorization_required", status: 401 });
|
|
expect(revoked.request).toHaveBeenCalledTimes(1);
|
|
|
|
const aborted = fixture();
|
|
aborted.request.mockImplementation(async (_url, init) => {
|
|
if (JSON.parse(String(init.body)).query === RAILWAY_QUERIES.target) return denied();
|
|
aborted.controller.abort(new Error("cancelled"));
|
|
throw new Error("network");
|
|
});
|
|
await expect(aborted.client.call("paperclip-railway-service-status", ids)).rejects.toThrow("cancelled");
|
|
});
|
|
|
|
it("returns each environment's access flag from list-environments", async () => {
|
|
const edges = [{ node: { id: target.environmentId, name: "production", projectId: target.projectId, canAccess: false } }, { node: { id: target.serviceId, name: "staging", projectId: target.projectId, canAccess: true } }];
|
|
const f = fixture(() => Response.json({ data: { project: { id: target.projectId, environments: { edges, pageInfo: { hasNextPage: false, endCursor: null } } } } }));
|
|
await expect(f.client.call("paperclip-railway-list-environments", { projectId: target.projectId })).resolves.toMatchObject({ project: { environments: { edges: [{ node: { canAccess: false } }, { node: { canAccess: true } }] } } });
|
|
expect(JSON.parse(String(f.request.mock.calls[0][1].body)).query).toBe(RAILWAY_QUERIES.environments);
|
|
});
|
|
|
|
it("requires exact provider identity and treats shell and remote agents as privileged", () => {
|
|
expect(isRailwayEndpoint("https://mcp.railway.com/")).toBe(true);
|
|
for (const url of ["https://mcp.railway.com/path", "https://mcp.railway.com?token=x", "https://mcp.railway.com.evil.test", "http://mcp.railway.com"]) expect(isRailwayEndpoint(url)).toBe(false);
|
|
expect(isRailwayConnection({ transport: "mcp_remote", authKind: "oauth", credentialSource: "paperclip_vault", config: { url: "https://mcp.railway.com", sourceTemplateKey: "railway", connectionMethodKey: "mcp-oauth" } })).toBe(true);
|
|
expect(railwayRisk("railwayAgent")).toBe("destructive");
|
|
expect(isRailwayToolBlocked("accept_deploy")).toBe(true);
|
|
expect(railwayRisk("paperclip-railway-run-command")).toBe("destructive");
|
|
expect(railwayRisk("unfamiliar-tool")).toBe("write");
|
|
expect(RAILWAY_TOOLS).toHaveLength(11);
|
|
expect(RAILWAY_TOOLS.map((tool) => tool.name)).not.toContain("paperclip-railway-deploy-revision");
|
|
});
|
|
|
|
it("gives container commands time for target checks without exceeding the gateway limit", () => {
|
|
expect(railwayCommandBudgetMs({})).toBe(40000);
|
|
expect(railwayCommandBudgetMs({ timeoutSeconds: 1 })).toBe(11000);
|
|
expect(railwayCommandBudgetMs({ timeoutSeconds: 60 })).toBe(60000);
|
|
expect(railwayCommandBudgetMs({ timeoutSeconds: 999 })).toBe(60000);
|
|
});
|
|
|
|
it("checks full deployment membership before a single fixed mutation", async () => {
|
|
const f = fixture();
|
|
await expect(f.client.call("paperclip-railway-restart", target)).resolves.toEqual({ deploymentRestart: true, targetDeploymentId: target.deploymentId });
|
|
expect(f.request.mock.calls.map(([, init]) => JSON.parse(String(init.body)).query)).toEqual([RAILWAY_QUERIES.target, RAILWAY_QUERIES.deployment, RAILWAY_QUERIES.restart]);
|
|
for (const [url, init] of f.request.mock.calls) {
|
|
expect(url).toBe(RAILWAY_API_URL);
|
|
expect(init.redirect).toBe("error");
|
|
expect(init.headers).toMatchObject({ Authorization: "Bearer railway-fixture-secret" });
|
|
}
|
|
expect(JSON.parse(String(f.request.mock.calls[2][1].body)).variables).toEqual({ deploymentId: target.deploymentId });
|
|
});
|
|
|
|
it.each(["project", "environment", "service", "deployment"])("denies a mismatched %s before mutation", async (field) => {
|
|
const f = fixture((q) => {
|
|
if (field === "deployment" && q === RAILWAY_QUERIES.deployment) return Response.json({ data: { deployment: { ...deploymentData.deployment, serviceId: instanceId } } });
|
|
if (field !== "deployment" && q === RAILWAY_QUERIES.target) return Response.json({ data: { ...targetData, [field]: { ...targetData[field as keyof typeof targetData], id: instanceId } } });
|
|
});
|
|
await expect(f.client.call("paperclip-railway-restart", target)).rejects.toMatchObject({ code: "railway_target_mismatch" });
|
|
expect(f.request.mock.calls.every(([, init]) => !JSON.parse(String(init.body)).query.startsWith("mutation"))).toBe(true);
|
|
});
|
|
|
|
it("bounds and redacts logs without selecting variables", async () => {
|
|
const f = fixture((q) => q === RAILWAY_QUERIES.runtimeLogs ? Response.json({ data: { deploymentLogs: Array.from({ length: 20 }, () => ({ message: `railway-fixture-secret ${"x".repeat(9000)}`, severity: "INFO" })) } }) : undefined);
|
|
const result = await f.client.call("paperclip-railway-read-logs", { ...target, limit: 10 });
|
|
expect(JSON.stringify(result)).not.toContain("railway-fixture-secret");
|
|
expect(Buffer.byteLength(JSON.stringify(result))).toBeLessThan(66000);
|
|
expect(result).toMatchObject({ truncated: true });
|
|
await expect(f.client.call("paperclip-railway-read-logs", { ...target, limit: 501 })).rejects.toMatchObject({ code: "railway_invalid_arguments" });
|
|
expect(Object.values(RAILWAY_QUERIES).join(" ")).not.toMatch(/variableCollection|variables\s*\{/);
|
|
});
|
|
|
|
it("reports when a single long log line was cut", async () => {
|
|
const f = fixture((q) => q === RAILWAY_QUERIES.runtimeLogs ? Response.json({ data: { deploymentLogs: [{ message: "x".repeat(20000) }] } }) : undefined);
|
|
await expect(f.client.call("paperclip-railway-read-logs", target)).resolves.toMatchObject({ truncated: true, limitReached: false });
|
|
});
|
|
|
|
it.each([401, 403, 429, 500])("sanitizes HTTP %s without retries", async (status) => {
|
|
const f = fixture(() => new Response("provider secret", { status }));
|
|
await expect(f.client.probe(target.projectId)).rejects.toThrow(/Railway/);
|
|
expect(f.request).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("does not expose GraphQL error details or retry an ambiguous mutation", async () => {
|
|
const f = fixture((q) => q === RAILWAY_QUERIES.restart ? Response.json({ errors: [{ message: "sensitive provider payload" }] }) : undefined);
|
|
await expect(f.client.call("paperclip-railway-restart", target)).rejects.toMatchObject({ code: "railway_api_error" });
|
|
expect(f.request).toHaveBeenCalledTimes(3);
|
|
});
|
|
|
|
it.each([
|
|
["restart", { deploymentRestart: false }],
|
|
["rollback", { deploymentRollback: false }],
|
|
["redeploy", { deploymentRedeploy: null }],
|
|
["redeploy", { deploymentRedeploy: { id: "not-a-deployment-id" } }],
|
|
])("does not report an unconfirmed %s as successful", async (operation, data) => {
|
|
const f = fixture((q) => q.startsWith("mutation") ? Response.json({ data }) : undefined);
|
|
await expect(f.client.call(`paperclip-railway-${operation}`, target)).rejects.toMatchObject({ code: "railway_operation_unconfirmed" });
|
|
expect(f.request).toHaveBeenCalledTimes(3);
|
|
});
|
|
|
|
it("rejects oversized responses, cancelled calls and GraphQL passthrough", async () => {
|
|
const f = fixture(() => new Response("x".repeat(1024 * 1024 + 1)));
|
|
await expect(f.client.probe(target.projectId)).rejects.toMatchObject({ code: "railway_output_limit" });
|
|
await expect(f.client.call("paperclip-railway-list-projects", { query: "mutation Evil" })).rejects.toMatchObject({ code: "railway_invalid_arguments" });
|
|
f.controller.abort();
|
|
await expect(f.client.probe(target.projectId)).rejects.toMatchObject({ name: "AbortError" });
|
|
expect(f.request).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each(["paperclip-railway-deploy-revision", "paperclip_railway_deploy_revision", "paperclipRailwayDeployRevision"])("blocks %s before any repository preflight or deployment mutation", async (name) => {
|
|
// Even a matching repository in the preflight can change before mutation.
|
|
// Without an atomic provider binding, no upstream request is safe to send.
|
|
const f = fixture();
|
|
const { deploymentId: _, ...ids } = target;
|
|
await expect(f.client.call(name, { ...ids, repository: "example/app", commitSha: "a".repeat(40) })).rejects.toMatchObject({ code: "railway_action_blocked", status: 403 });
|
|
expect(f.request).not.toHaveBeenCalled();
|
|
expect(isRailwayToolBlocked(name)).toBe(true);
|
|
expect(railwayRisk(name)).toBe("destructive");
|
|
});
|
|
|
|
it("allows only an instance in the exact running deployment to reach SSH", async () => {
|
|
const f = fixture();
|
|
await expect(f.client.call("paperclip-railway-run-command", { ...target, deploymentInstanceId: target.serviceId, command: "true" })).rejects.toMatchObject({ code: "railway_target_mismatch" });
|
|
expect(f.runCommand).not.toHaveBeenCalled();
|
|
await f.client.call("paperclip-railway-run-command", { ...target, deploymentInstanceId: instanceId, command: "true" });
|
|
expect(f.runCommand).toHaveBeenCalledWith({ deploymentInstanceId: instanceId, command: "true", timeoutSeconds: 30, signal: f.controller.signal });
|
|
});
|
|
});
|