mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 06:04:11 +01:00
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests verify work through the browser and public APIs. > - The classic matrix already tests several common task stories. > - Its older stories depend on local fixture setup and workspace access. > - Deployment tests need the same story checks with a separate target adapter. > - This PR shares those checks and adds five explicit journey selections. > - A deployment pass still requires evidence from that deployment. ## Linked Issues or Issue Description **What existing behavior does this improve?** Product E2E journey reuse across managed instances and deployed Cloud instances. **Current behavior** The deployed-target catalog covers task creation, assignment, Ask, and fresh onboarding. Plan revision and Everyday stories run through the classic fixture driver. Several story steps assume a local workspace. **Proposed behavior** Add explicit Plan revision, artifact revision, protected-action approval and decline, Skill Studio, and Stop/redirect journeys. Share the main browser sequences and content checks with the classic driver. Let target adapters provide authentication, native agents, a reachable service fixture, and owned cleanup. **Reason and benefit** A local pass cannot verify the deployed application. Shared story code lets a deployed target exercise the same behavior without copying an entire test or starting a replacement server. **Breaking changes** No product API or default matrix selection changes. Passed deployment results must satisfy the declared story checks and work-run count. Unsupported target capabilities fail before a story runs. Related public work: #17262 and #18098. Upstream prerequisite: #18230. ## What Changed - Extract shared Plan, artifact download/oracle, protected service, Skill Studio, and Stop/redirect helpers. Keep classic callers on those helpers. - Add five explicit paid journey selections with owned-task observations and native company, agent, model, and environment checks. - Include classic prompts, delivery/decision code, and the decline grader in the source-definition hash. Keep classic transient-read retries and reject deployment binding failures immediately. Match cookie directory boundaries in sensitive fixture requests. - Require complete story checks and expected run receipts before retaining a passing result. Keep failed evidence and private downloads. - Add a secret-safe request method for deployment fixture credentials. Add the missing declaration for the existing manual evidence policy module so the E2E project can typecheck. - Document target responsibilities and promotion after a real deployment pass. This source catalog does not enroll candidates in a daily plan. ## Verification - E2E TypeScript check: passed, including the review follow-up. - Review follow-up: 30 tests passed across four files, covering transient Plan reads, immediate binding rejection, cookie-path scope, agent receipts, and artifact evidence. Stop now waits for a saved, still-active native turn before the browser click and retains its startup events. Counterexamples reject claimed-but-unstarted, already-completed, interrupted, and non-native work. Shared observation and API helpers also enter the definition fingerprint. Cancellation snapshots are captured before lease cleanup. A direct terminal-boundary assertion rejects native work after cancellation even if it is already in the first snapshot; five cleanup-window counterexamples are covered. The Stop-output regression first failed for both native `item.delta` and `item.completed`; the fixed check detects both while allowing accounting-only updates. A report-catalog test first failed because the sandbox denied the tsx child IPC pipe (`listen EPERM`); the unchanged test passed outside the sandbox. - Affected journey, catalog, API, service fixture, and delivery checks: 313 tests passed across 19 files. - Workspace build: passed after rebase. - Workspace typecheck: passed. The default repository run was attempted. Repeated embedded PostgreSQL startup failures prevented local database coverage; it was stopped without claiming a full pass. Linux CI must supply that coverage. - Broad E2E support run: 4,793 passed, 23 failed, 2 skipped. Failures include existing stale image/Pi assertions and process/startup timeouts under host load. The affected tests pass separately. This is not a claim that the broad suite is green. - Upstream CI prerequisites: this branch is rebased on the merged #18230 repair. The earlier conflicting Turbo flags no longer apply. The shell-hook fix is also supplied by that upstream change. Linux shell-hook investigation: 19 focused tests and adapter TypeScript passed. The exact staged-config sequence reproduced CI's 500 in a pinned Node 24 Linux container. Node's default stdin socket at shell level zero selects Bash remote-shell startup and reads `.bashrc` instead of `BASH_ENV`. Ignored stdin and a nested shell each produce 1000. The fixture now ignores stdin; the production hook and 1000 assertion are unchanged. See [Bash startup rules](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files). - GitHub CI on the earlier head passed most suites but failed two browser tests (a fixture cleanup lock timeout and a fixture run staying queued) plus the shell-hook assertion. The lock-timeout case received one retry, which was cancelled by a later source push and does not establish recovery. These failures remain part of the verification record. - Real Cloud qualification: the private target adapter is reviewed and merged. These candidates must pass there before daily-plan promotion. Current source review: Greptile 5/5 and Paperclip Review 5/5 on `a9df1d6e8d880710fce39b87e689e07e1c8e6edc`, with zero unresolved review threads. The full [GitHub CI run](https://github.com/paperclipai/paperclip/actions/runs/40974946081) passed on that exact head, including both required aggregate checks. ## Risks Shared helper changes affect classic callers. The independent ZIP oracle and visible delivery requirements remain in place. Stop requires native acknowledgement, terminal event evidence, resource cleanup, and no later original-turn activity. The managed local target still lacks paid-agent fixtures and reports these journeys as unsupported. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, terminal use, and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
180 lines
7.0 KiB
TypeScript
180 lines
7.0 KiB
TypeScript
import type { APIRequestContext, APIResponse } from "@playwright/test";
|
|
|
|
export interface JsonRecord {
|
|
[key: string]: unknown;
|
|
}
|
|
|
|
async function failureMessage(response: APIResponse, method: string) {
|
|
const text = await response.text().catch(() => "");
|
|
return `${method} ${response.url()} returned ${response.status()}${text ? `: ${text}` : ""}`;
|
|
}
|
|
|
|
/** Preserve HTTP status independently of diagnostic response text. */
|
|
export class RunnerApiHttpError extends Error {
|
|
constructor(readonly status: number, message: string) { super(message); this.name = "RunnerApiHttpError"; }
|
|
}
|
|
|
|
/** Admission-only diagnostics. Never retain the rejected value or its cause. */
|
|
export class RemoteAdmissionReadError extends Error {
|
|
constructor(
|
|
readonly endpoint: "issue" | "run" | "leases",
|
|
readonly failureClass: "candidate_failure" | "transient_infrastructure" | "permanent_infrastructure",
|
|
) {
|
|
super(`Remote admission ${endpoint} read failed (${failureClass}); diagnostics withheld`);
|
|
this.name = "RemoteAdmissionReadError";
|
|
}
|
|
}
|
|
|
|
export class RunnerApi {
|
|
readonly baseURL: string;
|
|
private browserCookie?: string;
|
|
|
|
setBrowserSession(cookie: string) { this.browserCookie = cookie; }
|
|
private sessionOptions() {
|
|
return this.browserCookie ? { headers: { Cookie: this.browserCookie, Origin: this.baseURL } } : {};
|
|
}
|
|
|
|
constructor(readonly request: APIRequestContext) {
|
|
const port = process.env.PAPERCLIP_RUNNER_E2E_PORT?.trim();
|
|
if (!port) throw new Error("PAPERCLIP_RUNNER_E2E_PORT is required");
|
|
this.baseURL = `http://127.0.0.1:${port}`;
|
|
}
|
|
|
|
async get<T>(path: string, options?: { timeout: number }): Promise<T> {
|
|
const response = await this.request.get(path, { ...this.sessionOptions(), ...options });
|
|
if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET"));
|
|
return response.json() as Promise<T>;
|
|
}
|
|
|
|
async bytes(path: string): Promise<Buffer> {
|
|
const response = await this.request.get(path, this.sessionOptions());
|
|
if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET"));
|
|
return response.body();
|
|
}
|
|
|
|
async post<T>(path: string, data?: unknown, options?: { timeout: number }): Promise<T> {
|
|
const response = await this.request.post(path, { data, ...this.sessionOptions(), ...options });
|
|
if (!response.ok()) throw new Error(await failureMessage(response, "POST"));
|
|
return response.json() as Promise<T>;
|
|
}
|
|
|
|
/**
|
|
* Playwright traces APIRequestContext request bodies. Secret creation must
|
|
* still use the public API, but it goes through Node fetch so plaintext is
|
|
* never serialized into trace/blob evidence before Paperclip encrypts it.
|
|
*/
|
|
async postSensitive<T>(path: string, data: unknown): Promise<T> {
|
|
return this.sensitive<T>("POST", path, data);
|
|
}
|
|
|
|
async putSensitive<T>(path: string, data: unknown): Promise<T> {
|
|
return this.sensitive<T>("PUT", path, data);
|
|
}
|
|
|
|
private async sensitive<T>(method: "POST" | "PUT", path: string, data: unknown): Promise<T> {
|
|
const url = new URL(path, this.baseURL);
|
|
if (url.origin !== this.baseURL || !url.pathname.startsWith("/api/")) throw new Error("Sensitive request must use the fixture API");
|
|
let response: Response;
|
|
try {
|
|
response = await fetch(url, {
|
|
method, redirect: "error", signal: AbortSignal.timeout(15_000),
|
|
headers: { "content-type": "application/json", ...this.sessionOptions().headers },
|
|
body: JSON.stringify(data),
|
|
});
|
|
} catch { throw new Error(`Sensitive ${method} request failed; details withheld`); }
|
|
if (!response.ok) {
|
|
await response.body?.cancel();
|
|
throw new Error(`Sensitive ${method} ${url.pathname} returned ${response.status}; response body withheld`);
|
|
}
|
|
return response.json() as Promise<T>;
|
|
}
|
|
|
|
async patch<T>(path: string, data: unknown): Promise<T> {
|
|
const response = await this.request.patch(path, { data, ...this.sessionOptions() });
|
|
if (!response.ok())
|
|
throw new Error(await failureMessage(response, "PATCH"));
|
|
return response.json() as Promise<T>;
|
|
}
|
|
|
|
/** Use the public revision fence when configuring a fixture's instruction entry. */
|
|
async saveAgentInstructions(agentId: string, content: string): Promise<void> {
|
|
const path = `/api/agents/${agentId}/instructions-bundle/file`;
|
|
const current = await this.request.get(`${path}?path=AGENTS.md`);
|
|
let baseHash: string | null = null;
|
|
if (current.ok()) {
|
|
const detail = await current.json();
|
|
if (typeof detail.contentHash !== "string" || !detail.contentHash) {
|
|
throw new Error("Existing fixture instructions have no revision hash");
|
|
}
|
|
baseHash = detail.contentHash;
|
|
} else if (current.status() !== 404) {
|
|
throw new Error(await failureMessage(current, "GET"));
|
|
}
|
|
const saved = await this.request.put(path, { data: { path: "AGENTS.md", content, baseHash } });
|
|
if (!saved.ok()) throw new Error(await failureMessage(saved, "PUT"));
|
|
}
|
|
|
|
async delete(
|
|
path: string,
|
|
options?: { allowNotFound?: boolean },
|
|
): Promise<void> {
|
|
const response = await this.request.delete(path, this.sessionOptions());
|
|
if (response.ok() || (options?.allowNotFound && response.status() === 404))
|
|
return;
|
|
throw new Error(await failureMessage(response, "DELETE"));
|
|
}
|
|
}
|
|
|
|
export class ObservedStateTimeout extends Error {
|
|
constructor(label: string, readonly failureClass: "candidate_failure" | "transient_infrastructure", detail?: string) {
|
|
super(`Timed out waiting for ${label}; ${detail ?? "the observed state did not satisfy the condition"}. See the saved state evidence.`);
|
|
this.name = "ObservedStateTimeout";
|
|
}
|
|
}
|
|
|
|
export async function pollUntil<T>(input: {
|
|
label: string;
|
|
deadlineAt: number;
|
|
load: () => Promise<T>;
|
|
accept: (value: T) => boolean;
|
|
reject?: (value: T) => string | undefined;
|
|
intervalMs?: number;
|
|
retryOnError?: (error: unknown) => boolean;
|
|
timeoutFailureClass?: "candidate_failure" | "transient_infrastructure";
|
|
timeoutDetail?: (value: T | undefined) => string | undefined;
|
|
}): Promise<T> {
|
|
let last: T | undefined;
|
|
let lastError: unknown;
|
|
while (Date.now() < input.deadlineAt) {
|
|
try {
|
|
last = await input.load();
|
|
if (input.accept(last)) return last;
|
|
const rejection = input.reject?.(last);
|
|
if (rejection) {
|
|
throw new Error(`Stopped waiting for ${input.label}: ${rejection}`);
|
|
}
|
|
lastError = undefined;
|
|
} catch (error) {
|
|
if (input.retryOnError && !input.retryOnError(error)) throw error;
|
|
if (
|
|
error instanceof Error &&
|
|
error.message.startsWith(`Stopped waiting for ${input.label}:`)
|
|
) {
|
|
throw error;
|
|
}
|
|
lastError = error;
|
|
}
|
|
await new Promise((resolve) =>
|
|
setTimeout(resolve, input.intervalMs ?? 2_000),
|
|
);
|
|
}
|
|
if (lastError instanceof Error) {
|
|
throw new Error(`Timed out waiting for ${input.label}: ${lastError.message}`, { cause: lastError });
|
|
}
|
|
throw new ObservedStateTimeout(
|
|
input.label,
|
|
input.timeoutFailureClass ?? "candidate_failure",
|
|
input.timeoutDetail?.(last),
|
|
);
|
|
}
|