Files
PaperClipAI/tests/runner-e2e/api.ts
T
DottaandPaperclip d3f2a60e79 test(e2e): reuse core journeys across deployment targets (#18222)
## Thinking Path

> - Paperclip manages agents and their work.
> - Product E2E tests verify work through the browser and public APIs.
> - The classic matrix already tests several common task stories.
> - Its older stories depend on local fixture setup and workspace
access.
> - Deployment tests need the same story checks with a separate target
adapter.
> - This PR shares those checks and adds five explicit journey
selections.
> - A deployment pass still requires evidence from that deployment.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Product E2E journey reuse across managed instances and deployed Cloud
instances.

**Current behavior**

The deployed-target catalog covers task creation, assignment, Ask, and
fresh onboarding. Plan revision and Everyday stories run through the
classic fixture driver. Several story steps assume a local workspace.

**Proposed behavior**

Add explicit Plan revision, artifact revision, protected-action approval
and decline, Skill Studio, and Stop/redirect journeys. Share the main
browser sequences and content checks with the classic driver. Let target
adapters provide authentication, native agents, a reachable service
fixture, and owned cleanup.

**Reason and benefit**

A local pass cannot verify the deployed application. Shared story code
lets a deployed target exercise the same behavior without copying an
entire test or starting a replacement server.

**Breaking changes**

No product API or default matrix selection changes. Passed deployment
results must satisfy the declared story checks and work-run count.
Unsupported target capabilities fail before a story runs.

Related public work: #17262 and #18098. Upstream prerequisite: #18230.

## What Changed

- Extract shared Plan, artifact download/oracle, protected service,
Skill Studio, and Stop/redirect helpers. Keep classic callers on those
helpers.
- Add five explicit paid journey selections with owned-task observations
and native company, agent, model, and environment checks.
- Include classic prompts, delivery/decision code, and the decline
grader in the source-definition hash. Keep classic transient-read
retries and reject deployment binding failures immediately. Match cookie
directory boundaries in sensitive fixture requests.
- Require complete story checks and expected run receipts before
retaining a passing result. Keep failed evidence and private downloads.
- Add a secret-safe request method for deployment fixture credentials.
Add the missing declaration for the existing manual evidence policy
module so the E2E project can typecheck.
- Document target responsibilities and promotion after a real deployment
pass. This source catalog does not enroll candidates in a daily plan.

## Verification

- E2E TypeScript check: passed, including the review follow-up.
- Review follow-up: 30 tests passed across four files, covering
transient Plan reads, immediate binding rejection, cookie-path scope,
agent receipts, and artifact evidence. Stop now waits for a saved,
still-active native turn before the browser click and retains its
startup events. Counterexamples reject claimed-but-unstarted,
already-completed, interrupted, and non-native work. Shared observation
and API helpers also enter the definition fingerprint. Cancellation
snapshots are captured before lease cleanup. A direct terminal-boundary
assertion rejects native work after cancellation even if it is already
in the first snapshot; five cleanup-window counterexamples are covered.
The Stop-output regression first failed for both native `item.delta` and
`item.completed`; the fixed check detects both while allowing
accounting-only updates. A report-catalog test first failed because the
sandbox denied the tsx child IPC pipe (`listen EPERM`); the unchanged
test passed outside the sandbox.
- Affected journey, catalog, API, service fixture, and delivery checks:
313 tests passed across 19 files.
- Workspace build: passed after rebase.
- Workspace typecheck: passed. The default repository run was attempted.
Repeated embedded PostgreSQL startup failures prevented local database
coverage; it was stopped without claiming a full pass. Linux CI must
supply that coverage.
- Broad E2E support run: 4,793 passed, 23 failed, 2 skipped. Failures
include existing stale image/Pi assertions and process/startup timeouts
under host load. The affected tests pass separately. This is not a claim
that the broad suite is green.
- Upstream CI prerequisites: this branch is rebased on the merged #18230
repair. The earlier conflicting Turbo flags no longer apply. The
shell-hook fix is also supplied by that upstream change. Linux
shell-hook investigation: 19 focused tests and adapter TypeScript
passed. The exact staged-config sequence reproduced CI's 500 in a pinned
Node 24 Linux container. Node's default stdin socket at shell level zero
selects Bash remote-shell startup and reads `.bashrc` instead of
`BASH_ENV`. Ignored stdin and a nested shell each produce 1000. The
fixture now ignores stdin; the production hook and 1000 assertion are
unchanged. See [Bash startup
rules](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files).
- GitHub CI on the earlier head passed most suites but failed two
browser tests (a fixture cleanup lock timeout and a fixture run staying
queued) plus the shell-hook assertion. The lock-timeout case received
one retry, which was cancelled by a later source push and does not
establish recovery. These failures remain part of the verification
record.
- Real Cloud qualification: the private target adapter is reviewed and
merged. These candidates must pass there before daily-plan promotion.

Current source review: Greptile 5/5 and Paperclip Review 5/5 on
`a9df1d6e8d880710fce39b87e689e07e1c8e6edc`, with zero unresolved review
threads. The full [GitHub CI
run](https://github.com/paperclipai/paperclip/actions/runs/40974946081)
passed on that exact head, including both required aggregate checks.

## Risks

Shared helper changes affect classic callers. The independent ZIP oracle
and visible delivery requirements remain in place. Stop requires native
acknowledgement, terminal event evidence, resource cleanup, and no later
original-turn activity. The managed local target still lacks paid-agent
fixtures and reports these journeys as unsupported.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, terminal use, and code
execution. The exact serving model ID and context window are not exposed
to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes

- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 18:58:26 -05:00

180 lines
7.0 KiB
TypeScript

import type { APIRequestContext, APIResponse } from "@playwright/test";
export interface JsonRecord {
[key: string]: unknown;
}
async function failureMessage(response: APIResponse, method: string) {
const text = await response.text().catch(() => "");
return `${method} ${response.url()} returned ${response.status()}${text ? `: ${text}` : ""}`;
}
/** Preserve HTTP status independently of diagnostic response text. */
export class RunnerApiHttpError extends Error {
constructor(readonly status: number, message: string) { super(message); this.name = "RunnerApiHttpError"; }
}
/** Admission-only diagnostics. Never retain the rejected value or its cause. */
export class RemoteAdmissionReadError extends Error {
constructor(
readonly endpoint: "issue" | "run" | "leases",
readonly failureClass: "candidate_failure" | "transient_infrastructure" | "permanent_infrastructure",
) {
super(`Remote admission ${endpoint} read failed (${failureClass}); diagnostics withheld`);
this.name = "RemoteAdmissionReadError";
}
}
export class RunnerApi {
readonly baseURL: string;
private browserCookie?: string;
setBrowserSession(cookie: string) { this.browserCookie = cookie; }
private sessionOptions() {
return this.browserCookie ? { headers: { Cookie: this.browserCookie, Origin: this.baseURL } } : {};
}
constructor(readonly request: APIRequestContext) {
const port = process.env.PAPERCLIP_RUNNER_E2E_PORT?.trim();
if (!port) throw new Error("PAPERCLIP_RUNNER_E2E_PORT is required");
this.baseURL = `http://127.0.0.1:${port}`;
}
async get<T>(path: string, options?: { timeout: number }): Promise<T> {
const response = await this.request.get(path, { ...this.sessionOptions(), ...options });
if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET"));
return response.json() as Promise<T>;
}
async bytes(path: string): Promise<Buffer> {
const response = await this.request.get(path, this.sessionOptions());
if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET"));
return response.body();
}
async post<T>(path: string, data?: unknown, options?: { timeout: number }): Promise<T> {
const response = await this.request.post(path, { data, ...this.sessionOptions(), ...options });
if (!response.ok()) throw new Error(await failureMessage(response, "POST"));
return response.json() as Promise<T>;
}
/**
* Playwright traces APIRequestContext request bodies. Secret creation must
* still use the public API, but it goes through Node fetch so plaintext is
* never serialized into trace/blob evidence before Paperclip encrypts it.
*/
async postSensitive<T>(path: string, data: unknown): Promise<T> {
return this.sensitive<T>("POST", path, data);
}
async putSensitive<T>(path: string, data: unknown): Promise<T> {
return this.sensitive<T>("PUT", path, data);
}
private async sensitive<T>(method: "POST" | "PUT", path: string, data: unknown): Promise<T> {
const url = new URL(path, this.baseURL);
if (url.origin !== this.baseURL || !url.pathname.startsWith("/api/")) throw new Error("Sensitive request must use the fixture API");
let response: Response;
try {
response = await fetch(url, {
method, redirect: "error", signal: AbortSignal.timeout(15_000),
headers: { "content-type": "application/json", ...this.sessionOptions().headers },
body: JSON.stringify(data),
});
} catch { throw new Error(`Sensitive ${method} request failed; details withheld`); }
if (!response.ok) {
await response.body?.cancel();
throw new Error(`Sensitive ${method} ${url.pathname} returned ${response.status}; response body withheld`);
}
return response.json() as Promise<T>;
}
async patch<T>(path: string, data: unknown): Promise<T> {
const response = await this.request.patch(path, { data, ...this.sessionOptions() });
if (!response.ok())
throw new Error(await failureMessage(response, "PATCH"));
return response.json() as Promise<T>;
}
/** Use the public revision fence when configuring a fixture's instruction entry. */
async saveAgentInstructions(agentId: string, content: string): Promise<void> {
const path = `/api/agents/${agentId}/instructions-bundle/file`;
const current = await this.request.get(`${path}?path=AGENTS.md`);
let baseHash: string | null = null;
if (current.ok()) {
const detail = await current.json();
if (typeof detail.contentHash !== "string" || !detail.contentHash) {
throw new Error("Existing fixture instructions have no revision hash");
}
baseHash = detail.contentHash;
} else if (current.status() !== 404) {
throw new Error(await failureMessage(current, "GET"));
}
const saved = await this.request.put(path, { data: { path: "AGENTS.md", content, baseHash } });
if (!saved.ok()) throw new Error(await failureMessage(saved, "PUT"));
}
async delete(
path: string,
options?: { allowNotFound?: boolean },
): Promise<void> {
const response = await this.request.delete(path, this.sessionOptions());
if (response.ok() || (options?.allowNotFound && response.status() === 404))
return;
throw new Error(await failureMessage(response, "DELETE"));
}
}
export class ObservedStateTimeout extends Error {
constructor(label: string, readonly failureClass: "candidate_failure" | "transient_infrastructure", detail?: string) {
super(`Timed out waiting for ${label}; ${detail ?? "the observed state did not satisfy the condition"}. See the saved state evidence.`);
this.name = "ObservedStateTimeout";
}
}
export async function pollUntil<T>(input: {
label: string;
deadlineAt: number;
load: () => Promise<T>;
accept: (value: T) => boolean;
reject?: (value: T) => string | undefined;
intervalMs?: number;
retryOnError?: (error: unknown) => boolean;
timeoutFailureClass?: "candidate_failure" | "transient_infrastructure";
timeoutDetail?: (value: T | undefined) => string | undefined;
}): Promise<T> {
let last: T | undefined;
let lastError: unknown;
while (Date.now() < input.deadlineAt) {
try {
last = await input.load();
if (input.accept(last)) return last;
const rejection = input.reject?.(last);
if (rejection) {
throw new Error(`Stopped waiting for ${input.label}: ${rejection}`);
}
lastError = undefined;
} catch (error) {
if (input.retryOnError && !input.retryOnError(error)) throw error;
if (
error instanceof Error &&
error.message.startsWith(`Stopped waiting for ${input.label}:`)
) {
throw error;
}
lastError = error;
}
await new Promise((resolve) =>
setTimeout(resolve, input.intervalMs ?? 2_000),
);
}
if (lastError instanceof Error) {
throw new Error(`Timed out waiting for ${input.label}: ${lastError.message}`, { cause: lastError });
}
throw new ObservedStateTimeout(
input.label,
input.timeoutFailureClass ?? "candidate_failure",
input.timeoutDetail?.(last),
);
}