mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Product E2E suite proves that agent work survives a control-plane restart > - A restart chaos campaign found three restart cells that fail in the test harness, not in the product > - A harness failure hides the real restart result, so the campaign cannot classify these cells > - This pull request repairs the harness drift in those cells and names the one failure the harness cannot prevent > - The benefit is that these cells now test restart behavior, or fail with a clear cause ## Linked Issues or Issue Description Refs #14811 (connector setup became one screen). Refs #19475 (completion-update restart fixture). **What happened?** 1. `lifecycle-baseline.runner-codex.local.tool-review-restart` clicked a "Save and continue" button. #14811 removed that button. The cell failed during setup. After that fix, the cell also failed after the restart: the expanded review card does not survive the post-restart reload, so "Approve & run" was not visible. 2. `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` failed with `page.goto: net::ERR_ABORTED` after the controller restart. The old document's Vite client reloads when the replacement server starts, and this aborts the test navigation. The code tolerated only `TimeoutError`. 3. `completion-updates.runner-acpx-claude.local.handoff-completion-restart` waited for the source agent to run an instructed wait command. Claude replied without the command (2/2 attempts), so the restart never occurred. The cell then failed with a generic timeout. **Expected behavior** Each cell reaches its restart boundary and tests restart behavior. If the boundary cannot be reached, the cell names the cause. **Steps to reproduce** Run `pnpm test:e2e:runner -- --max-parallel 1 --max-automatic-retries 0 --id <cell>` for each cell above on `master`. ## What Changed - `tests/runner-e2e/connection-reviews.ts`: accept the one-screen "Check link" step and the older "Save and continue" step. This uses the same regex approach as `connection-flow.ts`. - `tests/runner-e2e/runner.spec.ts` (tool review): click "Review request" after the restart reload, on the document that acts on the card. - `tests/runner-e2e/chat-restart.ts`: add `gotoAfterServerRestart`. It retries only `net::ERR_ABORTED`, within the caller's deadline and at most 5 attempts. It still accepts `TimeoutError`. It rethrows all other errors. The structured-question restart uses it. The existing sentinel and durable-state assertions still prove the new document. - `tests/runner-e2e/completion-updates.ts` and `completion-update-flow.ts`: add `completionRestartGateSkipped` and `completionRestartGateTimeoutDetail`. The gate poll stops at once with `restart boundary not reached (agent skipped gate)` when no completion turn is active and the turn that carries this task's Done facts (`chatCompletionUpdates`, as in `completionDelivery`) succeeded or published its reply before the gate reports `waiting`. A failed or cancelled turn is not blamed on the agent, because delivery can retry it; the timeout detail names it instead. The post-restart chat navigation now uses `gotoAfterServerRestart`. - A fixture-level gate is not possible without a product hook. The native run finalizer publishes the completion reply inside its own transaction, through the issue service closure. The isolated-server entrypoint can patch only class prototypes and HTTP responses, and the reply uses no tool call or HTTP request. - `tests/runner-e2e/README.md`: document the skipped-gate failure. - Docs: use `pnpm --filter @paperclipai/paperclip-runner run build:runner-binaries` in `tests/runner-e2e/README.md`, `tests/restart-chaos/README.md`, `tests/restart-chaos/main.ts` and `doc/DEVELOPING.md`. - No product code changes. ## Verification - Unit: `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-restart.test.ts tests/runner-e2e/completion-updates.test.ts tests/runner-e2e/connection-reviews.test.ts` (36 passed). - Typecheck: `node scripts/typecheck-test-projects.mjs tests/runner-e2e` passed. - Live cells, local, `--max-parallel 1 --max-automatic-retries 0`, macOS, load average 35–65: - `lifecycle-baseline.runner-codex.local.tool-review-restart`: PASS (first run found the review-card reload drift, fixed in the second commit). - `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`: PASS. A first attempt failed before the restart with `PRP command session.open timed out` at load ~36 (environment). - `completion-updates.runner-acpx-claude.local.handoff-completion-restart`: FAIL as designed, twice (before and after the review fix), with `restart boundary not reached (agent skipped gate): completion turn … ended succeeded without the reference gate reporting waiting`. Claude skipped the wait both times, so this cell still does not exercise a restart. ## Risks - Low risk. Test harness and docs only. - The navigation retry could hide a real abort. It is limited to `net::ERR_ABORTED`, 5 attempts and the deadline, and the test still asserts that a new document loaded. - The ACPX Claude completion restart cell still does not exercise a restart when the agent skips the gate. It now reports this cause by name. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, browser test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
116 lines
4.3 KiB
TypeScript
116 lines
4.3 KiB
TypeScript
import { randomBytes } from "node:crypto";
|
|
import { expect, type Page } from "@playwright/test";
|
|
import type { RunnerApi } from "./api.js";
|
|
import { startReviewProvider } from "../fixtures/connection-review-provider.js";
|
|
|
|
/** The button after the URL step: Check link today, Save and continue before #14811. */
|
|
export const CONNECTOR_SETUP_NEXT_STEP = /^(Save and continue|Check link)$/i;
|
|
const CHECK_LINK = /^Check link$/i;
|
|
|
|
/** Uses the same production connection and permission screens as the fixture browser suite. */
|
|
export async function setupConnectionReview(input: {
|
|
page: Page;
|
|
api: RunnerApi;
|
|
prefix: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
marker: string;
|
|
authenticated?: boolean;
|
|
}) {
|
|
const credential = input.authenticated
|
|
? randomBytes(32).toString("hex")
|
|
: undefined;
|
|
const provider = await startReviewProvider(input.marker, credential);
|
|
try {
|
|
const { page, api } = input;
|
|
let connection: { id: string };
|
|
if (credential) {
|
|
// Setup through the public API avoids putting this fixture credential in
|
|
// browser traces or the agent's environment. Approval remains a UI action.
|
|
const connected = await api.postSensitive<any>(
|
|
`/api/companies/${input.companyId}/tools/apps/connect`,
|
|
{
|
|
link: provider.url,
|
|
name: "Studio Page Service",
|
|
authMode: "bearer",
|
|
credentialValues: { "credentials.authorization": credential },
|
|
grantKind: "organization",
|
|
},
|
|
);
|
|
const ids = [
|
|
...connected.actions.readOnly,
|
|
...connected.actions.canMakeChanges,
|
|
].map((a: any) => a.catalogEntryId);
|
|
await api.post(
|
|
`/api/companies/${input.companyId}/tools/apps/${connected.connectionId}/finish`,
|
|
{
|
|
enabledCatalogEntryIds: ids,
|
|
askFirstCatalogEntryIds: ids,
|
|
access: { agentIds: [input.agentId] },
|
|
},
|
|
);
|
|
connection = { id: connected.connectionId };
|
|
} else {
|
|
await page.goto(`/${input.prefix}/apps`, { waitUntil: "domcontentloaded" });
|
|
const connector = page
|
|
.getByRole("list", { name: "Connector list" })
|
|
.getByRole("listitem")
|
|
.filter({ hasText: "Connect your own tool" });
|
|
await connector
|
|
.getByRole("button", { name: "Connect", exact: true })
|
|
.click();
|
|
await connector
|
|
.getByRole("button", { name: "Connect your own MCP server" })
|
|
.click();
|
|
await page
|
|
.getByPlaceholder("https://example.com/actions")
|
|
.fill(provider.url);
|
|
await page.getByRole("button", { name: "Continue", exact: true }).click();
|
|
// #14811 made connector setup one screen: Continue now leads straight to
|
|
// Check link. Accept the older intermediate "Save and continue" step too,
|
|
// as connection-flow.ts does, so either layout reaches the link check.
|
|
const next = page
|
|
.getByRole("button", { name: CONNECTOR_SETUP_NEXT_STEP })
|
|
.first();
|
|
await expect(next).toBeVisible();
|
|
if (!CHECK_LINK.test((await next.textContent()) ?? "")) await next.click();
|
|
await page.getByRole("button", { name: CHECK_LINK }).click();
|
|
await expect(
|
|
page.getByRole("heading", { name: /is ready/i }),
|
|
).toBeVisible();
|
|
const {
|
|
connections: [createdConnection],
|
|
} = await api.get<{ connections: Array<{ id: string }> }>(
|
|
`/api/companies/${input.companyId}/tools/connections`,
|
|
);
|
|
connection = createdConnection!;
|
|
}
|
|
const installed = await api.request.put(
|
|
`/api/tool-connections/${connection.id}/installs`,
|
|
{
|
|
data: { installs: [{ targetType: "agent", targetId: input.agentId }] },
|
|
},
|
|
);
|
|
expect(installed.ok()).toBe(true);
|
|
await page.goto(`/${input.prefix}/apps/${connection.id}/permissions`, {
|
|
waitUntil: "domcontentloaded",
|
|
});
|
|
const askFirst = page.getByRole("radio", {
|
|
name: "List fixture pages: Ask first",
|
|
});
|
|
await expect(askFirst).toBeVisible({ timeout: 30_000 });
|
|
await askFirst.click();
|
|
return {
|
|
...provider,
|
|
connectionId: connection.id,
|
|
invocationCount: () =>
|
|
provider.captures.filter(
|
|
(call) => call.method === "tools/call" && call.authorized !== false,
|
|
).length,
|
|
};
|
|
} catch (error) {
|
|
await provider.close();
|
|
throw error;
|
|
}
|
|
}
|