Files
PaperClipAI/tests/runner-e2e/connection-reviews.ts
T
DottaandClaude Opus 5.5 7afb57c022 test(runner-e2e): repair restart-cell harness drift (#19538)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner Product E2E suite proves that agent work survives a
control-plane restart
> - A restart chaos campaign found three restart cells that fail in the
test harness, not in the product
> - A harness failure hides the real restart result, so the campaign
cannot classify these cells
> - This pull request repairs the harness drift in those cells and names
the one failure the harness cannot prevent
> - The benefit is that these cells now test restart behavior, or fail
with a clear cause

## Linked Issues or Issue Description

Refs #14811 (connector setup became one screen). Refs #19475
(completion-update restart fixture).

**What happened?**
1. `lifecycle-baseline.runner-codex.local.tool-review-restart` clicked a
"Save and continue" button. #14811 removed that button. The cell failed
during setup. After that fix, the cell also failed after the restart:
the expanded review card does not survive the post-restart reload, so
"Approve & run" was not visible.
2.
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`
failed with `page.goto: net::ERR_ABORTED` after the controller restart.
The old document's Vite client reloads when the replacement server
starts, and this aborts the test navigation. The code tolerated only
`TimeoutError`.
3.
`completion-updates.runner-acpx-claude.local.handoff-completion-restart`
waited for the source agent to run an instructed wait command. Claude
replied without the command (2/2 attempts), so the restart never
occurred. The cell then failed with a generic timeout.

**Expected behavior**
Each cell reaches its restart boundary and tests restart behavior. If
the boundary cannot be reached, the cell names the cause.

**Steps to reproduce**
Run `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0
--id <cell>` for each cell above on `master`.

## What Changed

- `tests/runner-e2e/connection-reviews.ts`: accept the one-screen "Check
link" step and the older "Save and continue" step. This uses the same
regex approach as `connection-flow.ts`.
- `tests/runner-e2e/runner.spec.ts` (tool review): click "Review
request" after the restart reload, on the document that acts on the
card.
- `tests/runner-e2e/chat-restart.ts`: add `gotoAfterServerRestart`. It
retries only `net::ERR_ABORTED`, within the caller's deadline and at
most 5 attempts. It still accepts `TimeoutError`. It rethrows all other
errors. The structured-question restart uses it. The existing sentinel
and durable-state assertions still prove the new document.
- `tests/runner-e2e/completion-updates.ts` and
`completion-update-flow.ts`: add `completionRestartGateSkipped` and
`completionRestartGateTimeoutDetail`. The gate poll stops at once with
`restart boundary not reached (agent skipped gate)` when no completion
turn is active and the turn that carries this task's Done facts
(`chatCompletionUpdates`, as in `completionDelivery`) succeeded or
published its reply before the gate reports `waiting`. A failed or
cancelled turn is not blamed on the agent, because delivery can retry
it; the timeout detail names it instead. The post-restart chat
navigation now uses `gotoAfterServerRestart`.
- A fixture-level gate is not possible without a product hook. The
native run finalizer publishes the completion reply inside its own
transaction, through the issue service closure. The isolated-server
entrypoint can patch only class prototypes and HTTP responses, and the
reply uses no tool call or HTTP request.
- `tests/runner-e2e/README.md`: document the skipped-gate failure.
- Docs: use `pnpm --filter @paperclipai/paperclip-runner run
build:runner-binaries` in `tests/runner-e2e/README.md`,
`tests/restart-chaos/README.md`, `tests/restart-chaos/main.ts` and
`doc/DEVELOPING.md`.
- No product code changes.

## Verification

- Unit: `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-restart.test.ts
tests/runner-e2e/completion-updates.test.ts
tests/runner-e2e/connection-reviews.test.ts` (36 passed).
- Typecheck: `node scripts/typecheck-test-projects.mjs tests/runner-e2e`
passed.
- Live cells, local, `--max-parallel 1 --max-automatic-retries 0`,
macOS, load average 35–65:
- `lifecycle-baseline.runner-codex.local.tool-review-restart`: PASS
(first run found the review-card reload drift, fixed in the second
commit).
-
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`:
PASS. A first attempt failed before the restart with `PRP command
session.open timed out` at load ~36 (environment).
-
`completion-updates.runner-acpx-claude.local.handoff-completion-restart`:
FAIL as designed, twice (before and after the review fix), with `restart
boundary not reached (agent skipped gate): completion turn … ended
succeeded without the reference gate reporting waiting`. Claude skipped
the wait both times, so this cell still does not exercise a restart.

## Risks

- Low risk. Test harness and docs only.
- The navigation retry could hide a real abort. It is limited to
`net::ERR_ABORTED`, 5 attempts and the deadline, and the test still
asserts that a new document loaded.
- The ACPX Claude completion restart cell still does not exercise a
restart when the agent skips the gate. It now reports this cause by
name.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, browser test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 12:59:54 -06:00

116 lines
4.3 KiB
TypeScript

import { randomBytes } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import type { RunnerApi } from "./api.js";
import { startReviewProvider } from "../fixtures/connection-review-provider.js";
/** The button after the URL step: Check link today, Save and continue before #14811. */
export const CONNECTOR_SETUP_NEXT_STEP = /^(Save and continue|Check link)$/i;
const CHECK_LINK = /^Check link$/i;
/** Uses the same production connection and permission screens as the fixture browser suite. */
export async function setupConnectionReview(input: {
page: Page;
api: RunnerApi;
prefix: string;
companyId: string;
agentId: string;
marker: string;
authenticated?: boolean;
}) {
const credential = input.authenticated
? randomBytes(32).toString("hex")
: undefined;
const provider = await startReviewProvider(input.marker, credential);
try {
const { page, api } = input;
let connection: { id: string };
if (credential) {
// Setup through the public API avoids putting this fixture credential in
// browser traces or the agent's environment. Approval remains a UI action.
const connected = await api.postSensitive<any>(
`/api/companies/${input.companyId}/tools/apps/connect`,
{
link: provider.url,
name: "Studio Page Service",
authMode: "bearer",
credentialValues: { "credentials.authorization": credential },
grantKind: "organization",
},
);
const ids = [
...connected.actions.readOnly,
...connected.actions.canMakeChanges,
].map((a: any) => a.catalogEntryId);
await api.post(
`/api/companies/${input.companyId}/tools/apps/${connected.connectionId}/finish`,
{
enabledCatalogEntryIds: ids,
askFirstCatalogEntryIds: ids,
access: { agentIds: [input.agentId] },
},
);
connection = { id: connected.connectionId };
} else {
await page.goto(`/${input.prefix}/apps`, { waitUntil: "domcontentloaded" });
const connector = page
.getByRole("list", { name: "Connector list" })
.getByRole("listitem")
.filter({ hasText: "Connect your own tool" });
await connector
.getByRole("button", { name: "Connect", exact: true })
.click();
await connector
.getByRole("button", { name: "Connect your own MCP server" })
.click();
await page
.getByPlaceholder("https://example.com/actions")
.fill(provider.url);
await page.getByRole("button", { name: "Continue", exact: true }).click();
// #14811 made connector setup one screen: Continue now leads straight to
// Check link. Accept the older intermediate "Save and continue" step too,
// as connection-flow.ts does, so either layout reaches the link check.
const next = page
.getByRole("button", { name: CONNECTOR_SETUP_NEXT_STEP })
.first();
await expect(next).toBeVisible();
if (!CHECK_LINK.test((await next.textContent()) ?? "")) await next.click();
await page.getByRole("button", { name: CHECK_LINK }).click();
await expect(
page.getByRole("heading", { name: /is ready/i }),
).toBeVisible();
const {
connections: [createdConnection],
} = await api.get<{ connections: Array<{ id: string }> }>(
`/api/companies/${input.companyId}/tools/connections`,
);
connection = createdConnection!;
}
const installed = await api.request.put(
`/api/tool-connections/${connection.id}/installs`,
{
data: { installs: [{ targetType: "agent", targetId: input.agentId }] },
},
);
expect(installed.ok()).toBe(true);
await page.goto(`/${input.prefix}/apps/${connection.id}/permissions`, {
waitUntil: "domcontentloaded",
});
const askFirst = page.getByRole("radio", {
name: "List fixture pages: Ask first",
});
await expect(askFirst).toBeVisible({ timeout: 30_000 });
await askFirst.click();
return {
...provider,
connectionId: connection.id,
invocationCount: () =>
provider.captures.filter(
(call) => call.method === "tools/call" && call.authorized !== false,
).length,
};
} catch (error) {
await provider.close();
throw error;
}
}