Files
PaperClipAI/ui/src/components/ExecutionBlockerNotice.test.tsx
T
DottaandPaperclip 5842185e4f fix: surface native cleanup quarantine and add chat qualification evals (#13775)
## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - Agent Chat needs reliable native execution before native runners
become the onboarding default.
> - Existing stories covered idle reassignment and controller restart,
but not an executing worker handoff or worker process loss.
> - Status answer tests also need to reject stale claims and invented
facts.
> - This pull request adds six opt-in full-stack cells with independent
state assertions and retained evidence.
> - The probes exposed a misleading Retry across server projection and
recovery-banner paths; the fix reports the blocked recovery honestly.
> - The tests preserve failures without changing recovery policy,
production prompts, or onboarding defaults.

## Linked Issues or Issue Description

Refs: #13762. Related: #13765 (Retry targets the latest failed attempt),
#13753 (task context ownership), #13746 (native recovery work).

## What Changed

- Add active reassignment with saved draft and plan preservation,
old-worker cancellation, and successor completion checks.
- Preserve recovery-needed projection when native cleanup fails before
its coordinator exists, refuse a generic retry that would immediately
fail again, and replace the recovery banner's misleading Retry with
Inspect run.
- Add verified local worker process loss with a required successful
continuation; retain a failing qualification result when recovery is
unavailable, while independently verifying the UI/API refuse doomed
retries.
- Add two-turn factual answer checks for current blockers, stale claims,
inactive backlog work, and unknown facts. Retain prose for separate
semantic review.
- Add positive and negative oracle calibration and document fault
isolation, cleanup, billing, and qualification limits.

## Verification

- Eval TypeScript check passes.
- All 442 eval support tests pass locally. The 89 focused server tests
and server typecheck pass. Six recovery-banner UI tests and token gates
pass.
- Initial new-cell campaign:
https://github.com/paperclipai/paperclip/actions/runs/35657128077. All
six results are retained; four failed on fixture-contract issues and two
exposed real worker cleanup quarantine.
- All 26 existing native onboarding cells:
https://github.com/paperclipai/paperclip/actions/runs/35656761484 (26/26
passed on master 846336e5a, all cleanup passed).
- Intermediate handoff/fault campaign:
https://github.com/paperclipai/paperclip/actions/runs/35657945095 (four
retained failures: two overly strict draft oracles, two real crash
quarantines).
- Final active handoff:
https://github.com/paperclipai/paperclip/actions/runs/35659014397 (2/2
passed on cf6d4ae3a; both cleanup passed).
- Clarified answer-quality fixtures:
https://github.com/paperclipai/paperclip/actions/runs/35658262695 (2/2
passed on 4a26f10be; both cleanup passed; all four answers semantically
reviewed).
- Quarantine guard regression campaign:
https://github.com/paperclipai/paperclip/actions/runs/35659580100 (both
API requests correctly refused with 409/no second run, but exposed a
separate misleading Retry in the recovery banner and a fixture wait on a
non-admitted run).
- Final quarantine guard verification:
https://github.com/paperclipai/paperclip/actions/runs/35661067305
(147e42f7e: both providers verify Inspect run/no Retry, HTTP 409, one
retained run, unchanged saved plan, and successful disposable cleanup.
Both evals intentionally remain red with
`worker_crash_recovery_unqualified`; no successful continuation exists).
The preceding campaign 35658772755 never ran provider cases because
GitHub artifact finalization returned HTTP 403.
- Full repository CI passes on 147e42f7e: typecheck, tests, build, and
browser gates. One unchanged local-service-supervisor readiness test
failed initially; its six-test file passed in isolation and the failed
shard passed on its single rerun. Latest-head rollup: 54 successful, 2
intentionally skipped, no failed or pending checks. Greptile is 5/5 with
zero unresolved findings.
- See tests/runner-e2e/QUALIFICATION-2026-09-21.md for retained attempts
and semantic review. Published reports:
[onboarding](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35656761484-1/),
[handoff](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35659014397-1/),
[grounded
answers](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35658262695-1/),
[crash guards and unqualified
recovery](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35661067305-1/).

## Risks

- Paid cells are explicit-only and local-only. The fault fixture signals
only the exact native run PID after checking its identity.
- Live worker-loss probes currently fail on cleanup quarantine for both
providers. The eval must remain red until there is a usable recovery,
even when preservation and refusal checks pass. Verified cleanup with a
fresh attempt versus exact-session resume remains a product decision.
- Structured facts alone do not qualify prose quality; semantic review
remains separate.
- Onboarding uses the existing runtime switch after the real wizard and
before provider execution. Native UI selection and public defaults
remain unchanged.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact served snapshot and context-window size are not
exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-21 18:13:01 -05:00

83 lines
5.1 KiB
TypeScript

// @vitest-environment jsdom
import { act, type AnchorHTMLAttributes } from "react";
import { createRoot, type Root } from "react-dom/client";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { ExecutionBlockerNotice } from "./ExecutionBlockerNotice";
import { agentsApi } from "../api/agents";
import { activityApi } from "../api/activity";
vi.mock("../lib/router", () => ({
Link: ({ to, ...props }: AnchorHTMLAttributes<HTMLAnchorElement> & { to: string }) => <a href={to} {...props} />,
}));
vi.mock("../api/agents", () => ({ agentsApi: { retryFailedRun: vi.fn() } }));
vi.mock("../api/activity", () => ({ activityApi: { runsForIssue: vi.fn() } }));
describe("stopped task recovery notice", () => {
let root: Root;
let container: HTMLDivElement;
let client: QueryClient;
const onRetried = vi.fn();
beforeEach(async () => {
vi.clearAllMocks();
container = document.createElement("div"); document.body.append(container);
root = createRoot(container);
client = new QueryClient({ defaultOptions: { queries: { retry: false }, mutations: { retry: false } } });
vi.mocked(activityApi.runsForIssue).mockResolvedValue([{ runId: "failed-run", agentId: "agent", status: "failed" }] as never);
await act(async () => root.render(<QueryClientProvider client={client}>
<ExecutionBlockerNotice companyId="company" issueId="task" onRetried={onRetried} blocker={{
recoveryActionId: "recovery", runId: "failed-run", agentId: "agent", cause: "legacy_execution_requires_reconciliation",
nextAction: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.",
}} />
</QueryClientProvider>));
await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
});
afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); });
it("shows only the requested sentence and Retry, inside a distinct recovery container", () => {
const notice = container.querySelector('[role="status"][aria-label="Task recovery"]')!;
expect(notice.textContent).toBe("Automatic recovery of this task stopped.Retry");
expect(notice.classList.contains("border")).toBe(true);
expect(notice.classList.contains("bg-muted")).toBe(true);
expect(notice.querySelector("a")).toBeNull();
});
it("keeps the required next action for other reconciliation causes", async () => {
await act(async () => root.render(<QueryClientProvider client={client}>
<ExecutionBlockerNotice companyId="company" issueId="task" onRetried={onRetried} blocker={{
recoveryActionId: "recovery", runId: "failed-run", agentId: "agent", cause: "action_outcome_unknown",
nextAction: "Verify the external action outcome before continuing.",
}} />
</QueryClientProvider>));
expect(container.textContent).toContain("Verify the external action outcome before continuing.");
expect(container.textContent).not.toContain("Automatic recovery of this task stopped.");
});
it.each(["native_continuation_requires_reconciliation", "native_session_cleanup_quarantined"])("links to the source run instead of offering a retry rejected by %s", async (cause) => {
await act(async () => root.render(<QueryClientProvider client={client}>
<ExecutionBlockerNotice companyId="company" issueId="task" onRetried={onRetried} blocker={{
recoveryActionId: "recovery", runId: "failed-run", agentId: "agent",
cause,
nextAction: "Inspect the original failure and reconcile the previous execution before continuing.",
}} />
</QueryClientProvider>));
expect(container.textContent).toContain("Recovery needed.");
expect(container.textContent).not.toContain("Retry");
const link = container.querySelector("a")!;
expect(link.textContent).toBe("Inspect run");
expect(link.getAttribute("href")).toBe("/agents/agent/runs/failed-run");
expect(agentsApi.retryFailedRun).not.toHaveBeenCalled();
});
it("retries the exact failed run and refreshes the task", async () => {
vi.mocked(agentsApi.retryFailedRun).mockResolvedValue({} as never);
await act(async () => container.querySelector<HTMLButtonElement>("button")!.click());
await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
expect(agentsApi.retryFailedRun).toHaveBeenCalledWith("agent", "failed-run", "company");
expect(onRetried).toHaveBeenCalledOnce();
});
it("shows a failed Retry in the same container and allows another attempt", async () => {
vi.mocked(agentsApi.retryFailedRun).mockRejectedValue(new Error("Environment cleanup is still running."));
await act(async () => container.querySelector<HTMLButtonElement>("button")!.click());
await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
expect(container.querySelector('[role="alert"]')?.textContent).toBe("Environment cleanup is still running.");
expect(container.querySelector<HTMLButtonElement>("button")!.disabled).toBe(false);
expect(onRetried).not.toHaveBeenCalled();
});
});