mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests verify work through the browser and public APIs. > - The classic matrix already tests several common task stories. > - Its older stories depend on local fixture setup and workspace access. > - Deployment tests need the same story checks with a separate target adapter. > - This PR shares those checks and adds five explicit journey selections. > - A deployment pass still requires evidence from that deployment. ## Linked Issues or Issue Description **What existing behavior does this improve?** Product E2E journey reuse across managed instances and deployed Cloud instances. **Current behavior** The deployed-target catalog covers task creation, assignment, Ask, and fresh onboarding. Plan revision and Everyday stories run through the classic fixture driver. Several story steps assume a local workspace. **Proposed behavior** Add explicit Plan revision, artifact revision, protected-action approval and decline, Skill Studio, and Stop/redirect journeys. Share the main browser sequences and content checks with the classic driver. Let target adapters provide authentication, native agents, a reachable service fixture, and owned cleanup. **Reason and benefit** A local pass cannot verify the deployed application. Shared story code lets a deployed target exercise the same behavior without copying an entire test or starting a replacement server. **Breaking changes** No product API or default matrix selection changes. Passed deployment results must satisfy the declared story checks and work-run count. Unsupported target capabilities fail before a story runs. Related public work: #17262 and #18098. Upstream prerequisite: #18230. ## What Changed - Extract shared Plan, artifact download/oracle, protected service, Skill Studio, and Stop/redirect helpers. Keep classic callers on those helpers. - Add five explicit paid journey selections with owned-task observations and native company, agent, model, and environment checks. - Include classic prompts, delivery/decision code, and the decline grader in the source-definition hash. Keep classic transient-read retries and reject deployment binding failures immediately. Match cookie directory boundaries in sensitive fixture requests. - Require complete story checks and expected run receipts before retaining a passing result. Keep failed evidence and private downloads. - Add a secret-safe request method for deployment fixture credentials. Add the missing declaration for the existing manual evidence policy module so the E2E project can typecheck. - Document target responsibilities and promotion after a real deployment pass. This source catalog does not enroll candidates in a daily plan. ## Verification - E2E TypeScript check: passed, including the review follow-up. - Review follow-up: 30 tests passed across four files, covering transient Plan reads, immediate binding rejection, cookie-path scope, agent receipts, and artifact evidence. Stop now waits for a saved, still-active native turn before the browser click and retains its startup events. Counterexamples reject claimed-but-unstarted, already-completed, interrupted, and non-native work. Shared observation and API helpers also enter the definition fingerprint. Cancellation snapshots are captured before lease cleanup. A direct terminal-boundary assertion rejects native work after cancellation even if it is already in the first snapshot; five cleanup-window counterexamples are covered. The Stop-output regression first failed for both native `item.delta` and `item.completed`; the fixed check detects both while allowing accounting-only updates. A report-catalog test first failed because the sandbox denied the tsx child IPC pipe (`listen EPERM`); the unchanged test passed outside the sandbox. - Affected journey, catalog, API, service fixture, and delivery checks: 313 tests passed across 19 files. - Workspace build: passed after rebase. - Workspace typecheck: passed. The default repository run was attempted. Repeated embedded PostgreSQL startup failures prevented local database coverage; it was stopped without claiming a full pass. Linux CI must supply that coverage. - Broad E2E support run: 4,793 passed, 23 failed, 2 skipped. Failures include existing stale image/Pi assertions and process/startup timeouts under host load. The affected tests pass separately. This is not a claim that the broad suite is green. - Upstream CI prerequisites: this branch is rebased on the merged #18230 repair. The earlier conflicting Turbo flags no longer apply. The shell-hook fix is also supplied by that upstream change. Linux shell-hook investigation: 19 focused tests and adapter TypeScript passed. The exact staged-config sequence reproduced CI's 500 in a pinned Node 24 Linux container. Node's default stdin socket at shell level zero selects Bash remote-shell startup and reads `.bashrc` instead of `BASH_ENV`. Ignored stdin and a nested shell each produce 1000. The fixture now ignores stdin; the production hook and 1000 assertion are unchanged. See [Bash startup rules](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files). - GitHub CI on the earlier head passed most suites but failed two browser tests (a fixture cleanup lock timeout and a fixture run staying queued) plus the shell-hook assertion. The lock-timeout case received one retry, which was cancelled by a later source push and does not establish recovery. These failures remain part of the verification record. - Real Cloud qualification: the private target adapter is reviewed and merged. These candidates must pass there before daily-plan promotion. Current source review: Greptile 5/5 and Paperclip Review 5/5 on `a9df1d6e8d880710fce39b87e689e07e1c8e6edc`, with zero unresolved review threads. The full [GitHub CI run](https://github.com/paperclipai/paperclip/actions/runs/40974946081) passed on that exact head, including both required aggregate checks. ## Risks Shared helper changes affect classic callers. The independent ZIP oracle and visible delivery requirements remain in place. Stop requires native acknowledgement, terminal event evidence, resource cleanup, and no later original-turn activity. The managed local target still lacks paid-agent fixtures and reports these journeys as unsupported. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, terminal use, and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
62 lines
3.4 KiB
TypeScript
62 lines
3.4 KiB
TypeScript
import type { Page } from "@playwright/test";
|
|
type Row = Record<string, any>;
|
|
|
|
/** Work emitted by a retired turn, excluding accounting and control-plane bookkeeping. */
|
|
export function nativeWorkEvents<T extends Row>(events: T[]): T[] {
|
|
return events.filter(row => {
|
|
const kind = row.payload?.prpEvent?.eventType;
|
|
return typeof kind === "string" && /^(tool\.execution\.|turn\.|message\.|item\.)/.test(kind);
|
|
});
|
|
}
|
|
|
|
/** Detect post-cancellation output even if it arrived before the first snapshot. */
|
|
export function nativeCancellationHasNoLaterWork(events: Row[]): boolean {
|
|
const work = nativeWorkEvents(events);
|
|
const cancellation = work.findIndex(row => row.payload.prpEvent.eventType === "turn.cancelled");
|
|
return cancellation >= 0 && cancellation === work.length - 1;
|
|
}
|
|
|
|
/** A claimed run is not proof that a cancellable native turn has started. */
|
|
export function activeNativeTurnStarted(run: Row | undefined, events: Row[]): boolean {
|
|
const latestTurn = events.map(row => row.payload?.prpEvent?.eventType)
|
|
.filter(kind => ["turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(kind)).at(-1);
|
|
return run?.runtimeMode === "native" && run.status === "running" && latestTurn === "turn.started";
|
|
}
|
|
|
|
/** Shared story order; target adapters supply authoritative retirement observations. */
|
|
export async function stopAndRedirectThroughUi(input: {
|
|
page: Page; nonce: string;
|
|
awaitRetirement(): Promise<void>;
|
|
beforeRedirect(): Promise<void>;
|
|
reply(message: string): Promise<unknown>;
|
|
}) {
|
|
await input.page.getByTestId("task-chat-composer-stop").last().click();
|
|
await input.awaitRetirement();
|
|
await input.beforeRedirect();
|
|
const marker = `The studio is ready. Reference ${input.nonce}.`;
|
|
const reply = await input.reply(`Change direction. Leave the project as it is. Reply with just this short note: "${marker}"`);
|
|
await input.page.reload();
|
|
return { marker, reply };
|
|
}
|
|
|
|
/** The native control plane must acknowledge this run's Stop, including resource release. */
|
|
export function nativeStopAcknowledged(run: Row, companyId: string, issueId: string, userId: string) {
|
|
const stop = run.resultJson?.nativeCancellation;
|
|
const result = run.resultJson ?? {};
|
|
const startup = result.startupCancellation;
|
|
const cancellation = result.cancellation;
|
|
return run.runtimeMode === "native" && run.status === "cancelled" && run.companyId === companyId && (run.nativeIssueId ?? run.issueId) === issueId
|
|
&& stop?.schema === "paperclip.native-cancellation.v1" && stop.companyId === companyId
|
|
&& stop.runId === run.id && stop.issueId === issueId && stop.scope === "run" && stop.dispatched === true
|
|
&& stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only"
|
|
&& stop.effects?.length === 1 && stop.effects[0] === "release_run_resources"
|
|
&& typeof stop.intentAuditId === "string" && typeof stop.acknowledgementAuditId === "string"
|
|
&& stop.intentAuditId !== stop.acknowledgementAuditId
|
|
&& /^native-cancellation:[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(stop.intentId ?? "")
|
|
&& result.cancelledByActorType === "user" && result.cancelledByUserId === userId
|
|
&& cancellation?.source === "operator" && cancellation.expected === true
|
|
&& cancellation.initiator?.type === "user" && cancellation.initiator.id === userId
|
|
&& typeof startup?.requestedAt === "string" && Number.isFinite(Date.parse(startup.requestedAt))
|
|
&& startup.beforeNativeSelection === false;
|
|
}
|