mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 09:07:46 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E probes measure when acknowledgements and replies become visible. > - A poll gives a time interval for an event. > - The first positive poll can occur after a target that the event could still meet. > - This pull request compares the full interval with a strict or inclusive target. > - Reports retain the raw observations and the original verdict when the timing verdict changes. ## Linked Issues or Issue Description **What happened?** A historical acknowledgement was absent at 1901 ms and first present at 2427 ms. A report marked the 2000 ms target as failed. The interval crosses that target, so the available evidence cannot give a pass or fail verdict. **Expected behavior** Return pass when the full interval meets the target. Return fail when the full interval rules out the target. Return inconclusive for overlap, invalid evidence, or insufficient evidence. Respect strict and inclusive targets and open and closed endpoints. **Steps to reproduce** Run the deterministic timing tests. Use the acknowledgement interval `(1901, 2427] ms` with `<= 2000 ms`. Compare the late reply interval `(52965, 53334] ms` with `<= 30000 ms`. The first is inconclusive. The second fails. **Paperclip version or commit** The source includes master at `14afc12124017d4ec30d0f4bfd6892f8ce475d21`. Related PR: [#13710](https://github.com/paperclipai/paperclip/pull/13710) changed chat latency and session reuse. This change corrects probe evidence and report verdicts. ## What Changed - Add a shared interval classifier with explicit endpoint and target operators. - Start observation before dispatch. Use a monotonic clock. Bound asynchronous reads by request start and completion. - Measure the first visible response separately from message creation and final settlement. Share browser reads with an optional acknowledgement probe. - Keep raw bounds, clock resolution, message metadata, failed reads, and original verdicts in retained report evidence. - Add deterministic coverage for threshold equality, missing data, edited messages, asynchronous reads, teardown, report correction, and a finite-domain oracle. - Bound observer shutdown to one second across pending and final reads. Retain timeout counts. Ignore late completions and start no more reads after shutdown. - Document the timing contract in the existing Product E2E guide. - Keep current startup reuse evidence after the master merge. Update four stale test expectations for qualified Copilot admission. Keep the explicit suite and provider restrictions. ## Verification - Current head: `c42e87e6d3b4b347e7cb176f56819c1d0abe106d`. - Passed: `pnpm test:e2e:runner:unit` (3,287 Vitest tests and 128 native completion tests; two explicit skips). - Passed: `pnpm test:e2e:runner:typecheck`, `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, and `git diff origin/master --check`. - Passed: [exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/39607140418), including both required checks (`ci / verify` and `ci / e2e`), all general and serialized test jobs, all eight browser shards, build, canary dry run, and the clean public npm-install smoke. - Greptile reviewed the complete current diff and returned [5/5 with no actionable issues](https://github.com/paperclipai/paperclip/pull/16829#issuecomment-6266704814). All review threads are resolved. - The redundant local `pnpm test:run` was stopped after the exact-head CI test jobs passed. It did not produce a complete local verdict; the full repository test evidence is the CI run above. - The [earlier broad run](https://github.com/paperclipai/paperclip/actions/runs/39249282597) failed on the old head. Its full test command timed out after 15 minutes. Five E2E support failures came from ambient `NODE_PATH` and a missing historical Git object. Current master supplies the closed-environment launcher and retained source fixture. The full support suite passes with these fixes. - After the current master merge, four stale Copilot assertions and the nested unit launcher failed. Copilot is now qualified. Those assertions now require no qualification override and retain the rejected suite/provider cases. The full documented unit command passes. - The first local typecheck after the merge used stale generated package declarations and failed. The typechecks pass after `pnpm build`. - These checks use synthetic timing evidence. They make no live provider calls and claim no new live latency measurement. ## Risks - The browser diagnostic depends on the existing message and receipt DOM markers. Controlled tests cover visibility, receipt separation, concurrent reads, and teardown. - Corrected timing verdicts can become inconclusive. The report keeps the original verdict and does not change the overall workflow grade. - This change adds no database migration or product surface. ## Model Used - OpenAI GPT-6 Codex. Model family: GPT-6. This session does not expose a more specific model ID or context window size. The agent used reasoning, shell tools, code inspection, and deterministic tests. The earlier implementation session did not retain its exact model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
144 lines
6.7 KiB
TypeScript
144 lines
6.7 KiB
TypeScript
import { expect, type Locator, type Page } from "@playwright/test";
|
|
|
|
/** Rich editors interpret pasted Markdown; fill inserts literal paragraph text. */
|
|
export async function fillTaskPrompt(editor: Locator, prompt: string): Promise<void> {
|
|
if (!/^ {0,3}(?:`{3,}|~{3,})/m.test(prompt)
|
|
|| await editor.evaluate(element => element.tagName === "TEXTAREA")) {
|
|
await editor.fill(prompt);
|
|
return;
|
|
}
|
|
await editor.fill("");
|
|
await editor.focus();
|
|
await editor.evaluate((element, text) => {
|
|
const clipboardData = new DataTransfer();
|
|
clipboardData.setData("text/plain", text);
|
|
const event = new ClipboardEvent("paste", { clipboardData, bubbles: true, cancelable: true });
|
|
element.dispatchEvent(event);
|
|
if (!event.defaultPrevented) throw new Error("Task editor did not accept the Markdown paste");
|
|
}, prompt);
|
|
}
|
|
|
|
export async function createTaskThroughUi(input: {
|
|
page: Page;
|
|
issuePrefix: string;
|
|
agentName: string;
|
|
title: string;
|
|
prompt: string;
|
|
workMode: "standard" | "planning" | "ask";
|
|
projectName?: string;
|
|
requireExplicitTitle?: boolean;
|
|
attachments?: readonly string[];
|
|
beforeDispatch?: () => void;
|
|
}) {
|
|
// Search's public creation action exposes the explicit title field. Strict
|
|
// native-operation fixtures use it so automatic task naming is not an extra
|
|
// provider mutation before the operation whose permission is under test.
|
|
const issuesUrl = input.requireExplicitTitle
|
|
? `/${encodeURIComponent(input.issuePrefix)}/search?scope=issues&q=${encodeURIComponent(`"${input.title}"`)}`
|
|
: `/${encodeURIComponent(input.issuePrefix)}/issues`;
|
|
const newTask = input.requireExplicitTitle
|
|
? input.page.getByRole("button", { name: "Create task from this query", exact: true })
|
|
: input.page.getByRole("button", { name: /^New task$/i }).first();
|
|
let bootstrapError: unknown;
|
|
for (let bootstrapAttempt = 1; bootstrapAttempt <= 3; bootstrapAttempt += 1) {
|
|
try {
|
|
await input.page.goto(issuesUrl, {
|
|
waitUntil: "domcontentloaded",
|
|
timeout: 30_000,
|
|
});
|
|
await newTask.waitFor({ state: "visible", timeout: 20_000 });
|
|
bootstrapError = undefined;
|
|
break;
|
|
} catch (error) {
|
|
bootstrapError = error;
|
|
if (bootstrapAttempt < 3) await input.page.waitForTimeout(1_000);
|
|
}
|
|
}
|
|
if (bootstrapError) {
|
|
throw new Error(
|
|
`Browser bootstrap failed before task creation: ${bootstrapError instanceof Error ? bootstrapError.message : String(bootstrapError)}`,
|
|
{ cause: bootstrapError },
|
|
);
|
|
}
|
|
await newTask.click();
|
|
const dialog = input.page.getByRole("dialog", { name: "New task", exact: true });
|
|
await expect(dialog).toBeVisible();
|
|
const titleInput = dialog.getByRole("textbox", { name: "Task title", exact: true });
|
|
if (input.requireExplicitTitle) {
|
|
await expect(titleInput).toBeVisible();
|
|
await titleInput.fill(input.title);
|
|
} else if (await titleInput.isVisible()) await titleInput.fill(input.title);
|
|
await fillTaskPrompt(dialog.getByRole("textbox", { name: "editable markdown", exact: true }), input.prompt);
|
|
if (input.workMode !== "standard") {
|
|
await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
|
|
await input.page.getByTestId(input.workMode === "planning" ? "composer-add-plan" : "composer-add-ask").click();
|
|
}
|
|
await dialog
|
|
.getByRole("button", { name: "Select assignee", exact: true })
|
|
.click();
|
|
await input.page.getByRole("searchbox", { name: "Search assignees", exact: true }).fill(input.agentName);
|
|
await input.page.getByRole("option").filter({ has: input.page.getByText(input.agentName, { exact: true }) }).click();
|
|
await expect(input.page.getByRole("searchbox", { name: "Search assignees", exact: true })).toBeHidden();
|
|
if (input.projectName) {
|
|
// Selecting the assignee advances focus to this selector and opens it.
|
|
// Focus is idempotent here; clicking would toggle an already-open popover
|
|
// closed before the search field can be filled.
|
|
const projectTrigger = dialog.getByRole("group", { name: "Task project and worktrees", exact: true }).getByRole("button").first();
|
|
await projectTrigger.focus();
|
|
const projectSearch = input.page.getByPlaceholder(/^Search projects(?:…|\.\.\.)?$/i);
|
|
await expect(projectSearch).toBeVisible({ timeout: 2_000 }).catch(async () => {
|
|
if (await projectTrigger.getAttribute("aria-expanded") !== "true") await projectTrigger.click();
|
|
await expect(projectSearch).toBeVisible();
|
|
});
|
|
await projectSearch.fill(input.projectName);
|
|
await input.page.getByText(input.projectName, { exact: true }).last().click();
|
|
}
|
|
if (input.attachments?.length) {
|
|
const chooser = input.page.waitForEvent("filechooser");
|
|
const upload = dialog.getByRole("button", { name: "Upload", exact: true });
|
|
if (await upload.count()) await upload.click();
|
|
else {
|
|
await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
|
|
await input.page.getByRole("menuitem", { name: /^Files and images/ }).click();
|
|
}
|
|
await (await chooser).setFiles([...input.attachments]);
|
|
// Upload finishes as part of task creation; the dialog retains the selected files.
|
|
}
|
|
// The prompt-only composer generates a provisional title which the provider
|
|
// can rename immediately. Bind the fixture to the actual creation response.
|
|
input.beforeDispatch?.();
|
|
const submittedAtMs = Date.now();
|
|
const [response] = await Promise.all([
|
|
input.page.waitForResponse(response => response.request().method() === "POST"
|
|
&& /^\/api\/companies\/[^/]+\/issues$/.test(new URL(response.url()).pathname)),
|
|
dialog.getByRole("button", { name: "Create task", exact: true }).click(),
|
|
]);
|
|
expect(response.status()).toBe(201);
|
|
const issue = await response.json() as { id: string; companyId: string; title: string; titleNeedsGeneration: boolean };
|
|
expect(new URL(response.url()).pathname).toBe(`/api/companies/${issue.companyId}/issues`);
|
|
expect(issue.id).toEqual(expect.any(String));
|
|
expect(issue.id.length).toBeGreaterThan(0);
|
|
if (input.requireExplicitTitle) {
|
|
expect(issue.title).toBe(input.title);
|
|
expect(issue.titleNeedsGeneration).toBe(false);
|
|
}
|
|
// Creation precedes uploads; allow them to finish before navigation.
|
|
await expect(dialog).not.toBeVisible({ timeout: 30_000 });
|
|
return { submittedAtMs, issueId: issue.id };
|
|
}
|
|
|
|
export async function submitTaskReply(
|
|
page: Page,
|
|
body: string,
|
|
): Promise<number> {
|
|
const composer = page.getByTestId("task-chat-composer-input").last();
|
|
await expect(composer).toBeVisible({ timeout: 30_000 });
|
|
await composer
|
|
.locator('[contenteditable="true"], textarea')
|
|
.first()
|
|
.fill(body);
|
|
const submittedAtMs = Date.now();
|
|
await page.getByTestId("task-chat-composer-send").last().click();
|
|
return submittedAtMs;
|
|
}
|