Files
PaperClipAI/tests/runner-e2e/user-actions.ts
T
DottaandPaperclip 53732c811b fix(evals): classify latency from observation bounds (#16829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E probes measure when acknowledgements and replies become
visible.
> - A poll gives a time interval for an event.
> - The first positive poll can occur after a target that the event
could still meet.
> - This pull request compares the full interval with a strict or
inclusive target.
> - Reports retain the raw observations and the original verdict when
the timing verdict changes.

## Linked Issues or Issue Description

**What happened?**

A historical acknowledgement was absent at 1901 ms and first present at
2427 ms. A report marked the 2000 ms target as failed. The interval
crosses that target, so the available evidence cannot give a pass or
fail verdict.

**Expected behavior**

Return pass when the full interval meets the target. Return fail when
the full interval rules out the target. Return inconclusive for overlap,
invalid evidence, or insufficient evidence. Respect strict and inclusive
targets and open and closed endpoints.

**Steps to reproduce**

Run the deterministic timing tests. Use the acknowledgement interval
`(1901, 2427] ms` with `<= 2000 ms`. Compare the late reply interval
`(52965, 53334] ms` with `<= 30000 ms`. The first is inconclusive. The
second fails.

**Paperclip version or commit**

The source includes master at
`14afc12124017d4ec30d0f4bfd6892f8ce475d21`.

Related PR:
[#13710](https://github.com/paperclipai/paperclip/pull/13710) changed
chat latency and session reuse. This change corrects probe evidence and
report verdicts.

## What Changed

- Add a shared interval classifier with explicit endpoint and target
operators.
- Start observation before dispatch. Use a monotonic clock. Bound
asynchronous reads by request start and completion.
- Measure the first visible response separately from message creation
and final settlement. Share browser reads with an optional
acknowledgement probe.
- Keep raw bounds, clock resolution, message metadata, failed reads, and
original verdicts in retained report evidence.
- Add deterministic coverage for threshold equality, missing data,
edited messages, asynchronous reads, teardown, report correction, and a
finite-domain oracle.
- Bound observer shutdown to one second across pending and final reads.
Retain timeout counts. Ignore late completions and start no more reads
after shutdown.
- Document the timing contract in the existing Product E2E guide.
- Keep current startup reuse evidence after the master merge. Update
four stale test expectations for qualified Copilot admission. Keep the
explicit suite and provider restrictions.

## Verification

- Current head: `c42e87e6d3b4b347e7cb176f56819c1d0abe106d`.
- Passed: `pnpm test:e2e:runner:unit` (3,287 Vitest tests and 128 native
completion tests; two explicit skips).
- Passed: `pnpm test:e2e:runner:typecheck`, `pnpm -r typecheck`, `pnpm
build`, `pnpm check:token-gates`, and `git diff origin/master --check`.
- Passed: [exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39607140418),
including both required checks (`ci / verify` and `ci / e2e`), all
general and serialized test jobs, all eight browser shards, build,
canary dry run, and the clean public npm-install smoke.
- Greptile reviewed the complete current diff and returned [5/5 with no
actionable
issues](https://github.com/paperclipai/paperclip/pull/16829#issuecomment-6266704814).
All review threads are resolved.
- The redundant local `pnpm test:run` was stopped after the exact-head
CI test jobs passed. It did not produce a complete local verdict; the
full repository test evidence is the CI run above.
- The [earlier broad
run](https://github.com/paperclipai/paperclip/actions/runs/39249282597)
failed on the old head. Its full test command timed out after 15
minutes. Five E2E support failures came from ambient `NODE_PATH` and a
missing historical Git object. Current master supplies the
closed-environment launcher and retained source fixture. The full
support suite passes with these fixes.
- After the current master merge, four stale Copilot assertions and the
nested unit launcher failed. Copilot is now qualified. Those assertions
now require no qualification override and retain the rejected
suite/provider cases. The full documented unit command passes.
- The first local typecheck after the merge used stale generated package
declarations and failed. The typechecks pass after `pnpm build`.
- These checks use synthetic timing evidence. They make no live provider
calls and claim no new live latency measurement.

## Risks

- The browser diagnostic depends on the existing message and receipt DOM
markers. Controlled tests cover visibility, receipt separation,
concurrent reads, and teardown.
- Corrected timing verdicts can become inconclusive. The report keeps
the original verdict and does not change the overall workflow grade.
- This change adds no database migration or product surface.

## Model Used

- OpenAI GPT-6 Codex. Model family: GPT-6. This session does not expose
a more specific model ID or context window size. The agent used
reasoning, shell tools, code inspection, and deterministic tests. The
earlier implementation session did not retain its exact model ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 08:31:22 -05:00

144 lines
6.7 KiB
TypeScript

import { expect, type Locator, type Page } from "@playwright/test";
/** Rich editors interpret pasted Markdown; fill inserts literal paragraph text. */
export async function fillTaskPrompt(editor: Locator, prompt: string): Promise<void> {
if (!/^ {0,3}(?:`{3,}|~{3,})/m.test(prompt)
|| await editor.evaluate(element => element.tagName === "TEXTAREA")) {
await editor.fill(prompt);
return;
}
await editor.fill("");
await editor.focus();
await editor.evaluate((element, text) => {
const clipboardData = new DataTransfer();
clipboardData.setData("text/plain", text);
const event = new ClipboardEvent("paste", { clipboardData, bubbles: true, cancelable: true });
element.dispatchEvent(event);
if (!event.defaultPrevented) throw new Error("Task editor did not accept the Markdown paste");
}, prompt);
}
export async function createTaskThroughUi(input: {
page: Page;
issuePrefix: string;
agentName: string;
title: string;
prompt: string;
workMode: "standard" | "planning" | "ask";
projectName?: string;
requireExplicitTitle?: boolean;
attachments?: readonly string[];
beforeDispatch?: () => void;
}) {
// Search's public creation action exposes the explicit title field. Strict
// native-operation fixtures use it so automatic task naming is not an extra
// provider mutation before the operation whose permission is under test.
const issuesUrl = input.requireExplicitTitle
? `/${encodeURIComponent(input.issuePrefix)}/search?scope=issues&q=${encodeURIComponent(`"${input.title}"`)}`
: `/${encodeURIComponent(input.issuePrefix)}/issues`;
const newTask = input.requireExplicitTitle
? input.page.getByRole("button", { name: "Create task from this query", exact: true })
: input.page.getByRole("button", { name: /^New task$/i }).first();
let bootstrapError: unknown;
for (let bootstrapAttempt = 1; bootstrapAttempt <= 3; bootstrapAttempt += 1) {
try {
await input.page.goto(issuesUrl, {
waitUntil: "domcontentloaded",
timeout: 30_000,
});
await newTask.waitFor({ state: "visible", timeout: 20_000 });
bootstrapError = undefined;
break;
} catch (error) {
bootstrapError = error;
if (bootstrapAttempt < 3) await input.page.waitForTimeout(1_000);
}
}
if (bootstrapError) {
throw new Error(
`Browser bootstrap failed before task creation: ${bootstrapError instanceof Error ? bootstrapError.message : String(bootstrapError)}`,
{ cause: bootstrapError },
);
}
await newTask.click();
const dialog = input.page.getByRole("dialog", { name: "New task", exact: true });
await expect(dialog).toBeVisible();
const titleInput = dialog.getByRole("textbox", { name: "Task title", exact: true });
if (input.requireExplicitTitle) {
await expect(titleInput).toBeVisible();
await titleInput.fill(input.title);
} else if (await titleInput.isVisible()) await titleInput.fill(input.title);
await fillTaskPrompt(dialog.getByRole("textbox", { name: "editable markdown", exact: true }), input.prompt);
if (input.workMode !== "standard") {
await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
await input.page.getByTestId(input.workMode === "planning" ? "composer-add-plan" : "composer-add-ask").click();
}
await dialog
.getByRole("button", { name: "Select assignee", exact: true })
.click();
await input.page.getByRole("searchbox", { name: "Search assignees", exact: true }).fill(input.agentName);
await input.page.getByRole("option").filter({ has: input.page.getByText(input.agentName, { exact: true }) }).click();
await expect(input.page.getByRole("searchbox", { name: "Search assignees", exact: true })).toBeHidden();
if (input.projectName) {
// Selecting the assignee advances focus to this selector and opens it.
// Focus is idempotent here; clicking would toggle an already-open popover
// closed before the search field can be filled.
const projectTrigger = dialog.getByRole("group", { name: "Task project and worktrees", exact: true }).getByRole("button").first();
await projectTrigger.focus();
const projectSearch = input.page.getByPlaceholder(/^Search projects(?:…|\.\.\.)?$/i);
await expect(projectSearch).toBeVisible({ timeout: 2_000 }).catch(async () => {
if (await projectTrigger.getAttribute("aria-expanded") !== "true") await projectTrigger.click();
await expect(projectSearch).toBeVisible();
});
await projectSearch.fill(input.projectName);
await input.page.getByText(input.projectName, { exact: true }).last().click();
}
if (input.attachments?.length) {
const chooser = input.page.waitForEvent("filechooser");
const upload = dialog.getByRole("button", { name: "Upload", exact: true });
if (await upload.count()) await upload.click();
else {
await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
await input.page.getByRole("menuitem", { name: /^Files and images/ }).click();
}
await (await chooser).setFiles([...input.attachments]);
// Upload finishes as part of task creation; the dialog retains the selected files.
}
// The prompt-only composer generates a provisional title which the provider
// can rename immediately. Bind the fixture to the actual creation response.
input.beforeDispatch?.();
const submittedAtMs = Date.now();
const [response] = await Promise.all([
input.page.waitForResponse(response => response.request().method() === "POST"
&& /^\/api\/companies\/[^/]+\/issues$/.test(new URL(response.url()).pathname)),
dialog.getByRole("button", { name: "Create task", exact: true }).click(),
]);
expect(response.status()).toBe(201);
const issue = await response.json() as { id: string; companyId: string; title: string; titleNeedsGeneration: boolean };
expect(new URL(response.url()).pathname).toBe(`/api/companies/${issue.companyId}/issues`);
expect(issue.id).toEqual(expect.any(String));
expect(issue.id.length).toBeGreaterThan(0);
if (input.requireExplicitTitle) {
expect(issue.title).toBe(input.title);
expect(issue.titleNeedsGeneration).toBe(false);
}
// Creation precedes uploads; allow them to finish before navigation.
await expect(dialog).not.toBeVisible({ timeout: 30_000 });
return { submittedAtMs, issueId: issue.id };
}
export async function submitTaskReply(
page: Page,
body: string,
): Promise<number> {
const composer = page.getByTestId("task-chat-composer-input").last();
await expect(composer).toBeVisible({ timeout: 30_000 });
await composer
.locator('[contenteditable="true"], textarea')
.first()
.fill(body);
const submittedAtMs = Date.now();
await page.getByTestId("task-chat-composer-send").last().click();
return submittedAtMs;
}