Files
PaperClipAI/tests/runner-e2e/continuation-approval.ts
T
Dotta 071b4dd5c3 fix(evals): preserve continuation restart and approval boundaries (#19076)
## Thinking Path

> - Paperclip manages work done by AI agents.
> - Product E2E journeys test that work through the UI and public APIs.
> - A continuation must retain completed work across a server restart.
> - A human-only plan request needs a recorded human decision.
> - The current fixtures retain an old Vite client and dismiss the plan
request before sending a comment.
> - This change uses the existing restart guard and approves the exact
current plan through its UI.

## Linked Issues or Issue Description

No separate issue exists. The failures are retained in [the original
complete Product
run](https://github.com/paperclipai/paperclip/actions/runs/41915965118).

**What happened?**

`completed-action-resume` reached its intended completed-child and
pending-question state. Its navigation then hit the original 30-second
limit after a server restart. The retained trace shows the old Vite
client polling during listener replacement. The existing chat restart
wrapper unloads that client, but the continuation callback did not use
it.

`revision-preserves-approval` saved the revised plan and final output.
Its exact current human-only confirmation remained pending. The fixture
had clicked Dismiss to enter a plain approval comment. A comment does
not resolve that card, so the final pending-work checks correctly
failed.

**Expected behavior**

Unload the old client before the existing restart. Use the real Confirm
action on the exact current plan revision. Require its saved human
acceptance before the normal continuation settles. Preserve all original
final checks and limits.

**Steps to reproduce**

Run those two existing continuation cases from source `4740d251`. The
original run preserves both failures. The focused controls below require
no provider credentials.

**Paperclip version or commit**

Original failure source: `4740d251b7032e714d33039ba66a56948238c38f`.

**Deployment mode**

Isolated local Product E2E servers in GitHub Actions.

## What Changed

- Route only the continuation restart callback through the existing
`restartChatServer` wrapper.
- Select the one pending confirmation for the exact saved plan revision.
Recheck it before the UI action.
- Click its scoped Confirm button. Require the same target, accepted
outcome, human resolver and timestamp in the saved receipt.
- Preserve the generic chat Dismiss behavior, ordinary reply cases,
original prompts, models, run bounds, grading and deadlines.
- Add focused unit and browser controls for the approval boundary.

## Verification

- The original full run retains these two failed grades. Cleanup passed
in both; that does not make either case pass.
- Focused continuation unit controls: 44 passed across four files. The
final new helper controls also passed 3/3.
- Three distinct controlled browser checks passed: the existing
old-client restart guard, the exact current human-only Confirm action,
and rejection of a changed revision without any approval POST.
- The browser controls use local HTTP fixtures. They do not prove a
fresh provider journey passed.
- All six files match the tested handoff hashes. `git diff --check`
passes.
- Local harness typecheck reached six diagnostics from stale cached
dependency declarations. The missing exports are in unchanged
adapter-utils and Runner code. No owned-file diagnostic remains. No
source or lockfile workaround was used. Fresh CI typecheck, build and
tests remain required.
- Fresh real Product acceptance has not run for this repair. The
original 20-case result remains 13 passed and 7 failed.


Current-head CI passes at `335683b741df5a3d0b76d035e105f0b44eda832d`:
required `ci / verify` and `ci / e2e` are green. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/41922544434).
Greptile is 5/5 on this head. Ponytail is 4/5 with a ship
recommendation; its checkpoint warning was resolved with the dominating
awaited snapshot/push proof. No source changed after these checks. Real
changed-source journey acceptance remains pending.

## Risks

- The browser controls model the restart and approval boundaries; the
original real cases must still run after delivery.
- A changed or stale confirmation now fails before an approval. A
dismissed card or an agent-authored claim cannot satisfy human
acceptance.
- This fixture correction grants no new product permission and changes
no provider or scenario limit.

## Model Used

OpenAI `gpt-6.1-sol` in Codex, with `max` reasoning effort. The runtime
reports a 475,000-token context window. Used reasoning, code editing,
shell tools and provider-free browser controls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-04 17:39:10 -06:00

83 lines
4.4 KiB
TypeScript

import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { isContinuationPlan, type ContinuationCheckpoint } from "./continuation-scoring.js";
type Target = { type: string; issueId?: string; key?: string; revisionId?: string };
export interface ContinuationApproval {
id: string;
issueId: string;
kind: string;
status: string;
resolverPolicy?: string;
effectiveResolverPolicy?: string;
continuationPolicy?: string;
payload: { acceptLabel?: string; target?: Target };
resolvedByUserId?: string | null;
resolvedByAgentId?: string | null;
resolvedAt?: string | null;
result?: { outcome?: string } | null;
}
/** Select the exact current saved proposal before using its UI. */
export function currentContinuationApproval(checkpoint: ContinuationCheckpoint): ContinuationApproval {
const pending = (checkpoint.interactions as ContinuationApproval[])
.filter(i => i.kind === "request_confirmation" && i.status === "pending");
expect(pending, "one current plan confirmation must await the user's decision").toHaveLength(1);
const card = pending[0]!;
const target = card.payload.target;
expect(card.issueId).toBe(checkpoint.issue.id);
expect(target).toMatchObject({ type: "issue_document", issueId: checkpoint.issue.id });
const document = checkpoint.documents.find(d => d.key === target?.key);
expect(document && isContinuationPlan(document, checkpoint), "approval must target the recorded plan").toBeTruthy();
expect(document?.latestRevisionId, "the plan must have an immutable current revision").toBeTruthy();
expect(target?.revisionId, "approval must target the current plan revision").toBe(document!.latestRevisionId);
return card;
}
function sameTarget(left: Target | undefined, right: Target | undefined) {
return Boolean(left && right && left.type === right.type && left.issueId === right.issueId
&& left.key === right.key && left.revisionId === right.revisionId);
}
export function continuationApprovalAccepted(expected: ContinuationApproval, saved: ContinuationApproval | undefined) {
return Boolean(saved && saved.id === expected.id && saved.issueId === expected.issueId
&& saved.kind === "request_confirmation" && saved.status === "accepted"
&& saved.result?.outcome === "accepted" && sameTarget(expected.payload.target, saved.payload.target)
&& saved.resolvedByUserId?.trim() && !saved.resolvedByAgentId
&& saved.resolvedAt && Number.isFinite(Date.parse(saved.resolvedAt)));
}
/** A chat comment/dismissal is not a human-only card response. Use its real UI. */
export async function confirmContinuationPlan(input: {
page: Page;
api: Pick<RunnerApi, "get">;
checkpoint: ContinuationCheckpoint;
deadlineAt: number;
}) {
const expected = currentContinuationApproval(input.checkpoint);
const path = `/api/issues/${input.checkpoint.issue.id}`;
const [interactions, documents] = await Promise.all([
input.api.get<ContinuationApproval[]>(`${path}/interactions`),
input.api.get<ContinuationCheckpoint["documents"]>(`${path}/documents`),
]);
const current = currentContinuationApproval({ ...input.checkpoint, interactions, documents });
expect(current.id, "the reviewed confirmation must still be current").toBe(expected.id);
expect(sameTarget(current.payload.target, expected.payload.target), "the reviewed plan must not change before approval").toBe(true);
const card = input.page.getByTestId("task-chat-composer-takeover")
.locator(`[id=${JSON.stringify(`interaction-${current.id}`)}]`);
await expect(card, "the exact current confirmation must be visible").toBeVisible();
await card.getByRole("button", { name: current.payload.acceptLabel ?? "Approve", exact: true }).click();
const saved = await pollUntil({
label: "continuation plan human acceptance",
deadlineAt: Math.min(input.deadlineAt, Date.now() + 30_000),
intervalMs: 250,
load: async () => (await input.api.get<ContinuationApproval[]>(`${path}/interactions`)).find(i => i.id === current.id),
accept: result => continuationApprovalAccepted(current, result),
reject: result => result && (!sameTarget(current.payload.target, result.payload.target)
|| !["pending", "accepted"].includes(result.status)
|| result.status === "accepted" && !continuationApprovalAccepted(current, result))
? "the reviewed confirmation changed or did not record human acceptance" : undefined,
});
return saved!;
}