Files
PaperClipAI/tests/runner-e2e/first-task-approval-history-flow.ts
T
DottaandPaperclip 7c66b6b1f6 test(onboarding): cover plan, hiring, and execution approval history (#19069)
## Thinking Path

> - Paperclip manages agent work and human decisions.
> - Onboarding can approve a plan, a hire, and an execution step in
separate turns.
> - Tests covered single approvals more closely than that full history.
> - The repaired gate must retain valid plan approval through later
decisions.
> - Every consumer must recognize the saved confirmation kind and
selected scope.
> - This change tests that contract and registers a complete
interview-to-execution journey.

## Linked Issues or Issue Description

Refs #19043, #17914, #18283.

The earlier tests admitted one accepted proposal at a time.
Decomposition fixtures used standard confirmations. The direct-task E2E
path forbids hiring. Those cases did not cover an accepted checkbox plan
followed by hiring and a separate untargeted execution approval.

## What Changed

- Add eight PostgreSQL contract cases for standard and checkbox plan
approvals. Native tools create the cards. An authenticated human accepts
them through HTTP. A later native run reads the approved revision and
actual selected scope.
- Exercise native task creation, generic REST creation, child REST
creation, and accepted-plan decomposition. Check the assigned child and
one creation audit event. Check durable replay on the three entry points
that support it. A pending execution confirmation still prevents
creation.
- Register the explicit-only `onboarding-approval-history` suite for
local native Codex and ACPX Claude. The browser chooses the interview,
selects only drafting in the saved plan, approves one writer hire, and
approves a separate Start drafting confirmation. The model must hire the
writer and use that writer's own child execution to save the welcome
document.
- Grade each human decision, the unchanged plan revision and selection,
approval timing, one assigned child and worker execution, the exact
writer-authored saved revision, no pending requests, and persistence
after reload. Calibrate the oracle against missing, forged, premature,
duplicate, and changed outcomes.
- Preserve synthetic option choices, document/run bindings, and saved
revision authorship in the existing public first-task evidence
projection. Keep private provider/account data excluded. Document the
workflow, evidence, billing, and cleanup bounds.
- Align the approval-history fixture with its completion oracle: the
onboarding parent must be Done. Test both native profiles against graded
completion evidence and reject a completed child whose parent remains In
review. Count parent heartbeats separately from the writer: accept one
to seven parent records while retaining the twelve-company-run cap and
the independent three-approval/one-writer oracle.
- Retain the existing 52-cell first-task suite. The new suite has one
attempt, seven expected work turns, a twelve-run maximum, a
fifteen-minute deadline, and the existing $5 company task budget. It is
excluded from `--all`.
- Update two stale cleanup assertions to include the existing
thirty-second deletion timeout.

## Verification

- Red control: temporarily restore the pre-fix admission module and run
the new checkbox/native history case. It fails with the expected
`First-task proposal requires human acceptance` error after plan,
hiring, and execution acceptance. Restore the repaired module after the
probe.
- The final post-rebase history matrix and existing onboarding safety
suite passed together: 74 tests, no skips.
- Final focused Runner E2E support passed: 413 Vitest tests and all 370
Node support checks. This covers catalog discovery, journey controls,
wrong-outcome calibration, public evidence, and existing first-task
behavior.
- Review controls: four authorship cases fail before the revision check.
A delayed browser-response control fails before the approval-time fix.
Both pass after repair. The final oracle requires the exact current
document revision, writer ID, native child run ID, matching body, and
readback after reload.
- Credential-free discovery lists both new native cells. No provider
call was made. `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are absent, so
the real provider journey remains unqualified.
- The earlier broad credential-free E2E support run reported 6,728
passing cases, three skipped cases, and nine failures in three unrelated
files. All nine reproduce with the changed E2E source restored to the
unchanged baseline: three expired overhead-fixture cases, one stale Pi
profile-version assertion, and five workflow security assertions. This
change does not alter those files, paid workflow policy, or provider
pins.
- Previous full workspace typecheck/build and [Linux
CI](https://github.com/paperclipai/paperclip/actions/runs/41921754638)
passed on `4f431f5aae99d072cc75027e90d7b847b4c342ca`, including the new
eight-case PostgreSQL file. Ponytail subsequently found a fixture/oracle
terminal-status mismatch. Both native-cell controls fail on that fixture
(`done` observed versus `in_review` expected), while the
parent-in-review negative control passes. A second comparison uses the
actual `matchesRunCount` predicate: graded evidence with six parent runs
and one writer run fails the old exact-seven-parent requirement. Both
fixes pass these controls, all 413 focused support tests, all 370 Node
checks, and the owning E2E project typecheck. [Fresh
CI](https://github.com/paperclipai/paperclip/actions/runs/41936699686)
passed on `91c980b110252da451b92eda597c168517de1276`: 48 successful jobs
and one expected skip. Greptile reviewed this head at 5/5 with no
unresolved findings. Ponytail's terminal-status finding is fixed and its
thread is resolved; its current-head re-review remains queued. Live
provider cells remain unqualified.

## Risks

The production API and approval gate are unchanged. The new paid journey
can expose model or product failures at an intermediate approval
boundary. It retains those failures and uses the existing diagnostics
and cleanup. Registration and deterministic tests do not establish a
live provider pass. The unrelated baseline support failures remain
explicit.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused tests; unrelated
baseline failures are recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 21:46:55 -06:00

52 lines
3.4 KiB
TypeScript

import { expect, type Page } from "@playwright/test";
import { pollUntil, type RunnerApi } from "./api.js";
import { DRAFT_OPTION, PUBLISH_OPTION, approvalHistoryWriter } from "./first-task-approval-history.js";
import type { FirstTaskCheckpoint, Row } from "./first-task-scoring.js";
/** The model produces every proposal and hire; the harness only operates ordinary approval controls. */
export async function runApprovalHistoryDecisions(input: {
page: Page; api: Pick<RunnerApi, "get">; issueId: string; marker: string; deadlineAt: number;
allRuns: () => Promise<Row[]>;
snapshot: (phase: FirstTaskCheckpoint["phase"], at?: string) => Promise<FirstTaskCheckpoint>;
settle: (prior: Set<string>, completion?: boolean) => Promise<void>;
capture: (id: string, label: string, file: string) => Promise<void>;
}) {
const cards = () => input.api.get<Row[]>(`/api/issues/${input.issueId}/interactions`);
const approve = async (card: Row, phase: FirstTaskCheckpoint["phase"]) => {
const before = new Set((await input.allRuns()).map(run => run.id));
const at = new Date().toISOString();
await input.page.getByRole("button", { name: card.payload.acceptLabel
?? (card.kind === "request_confirmation" ? "Approve" : "Confirm selection"), exact: true }).last().click();
await pollUntil({ label: `saved ${phase} decision`, deadlineAt: Math.min(input.deadlineAt, Date.now() + 30_000), intervalMs: 250,
load: cards, accept: rows => rows.some(row => row.id === card.id && row.status === "accepted" && row.resolvedByUserId),
});
await input.snapshot(phase, at);
await input.settle(before, phase === "accepted");
};
const pending = async (predicate: (row: Row) => boolean) => {
const found = (await cards()).filter(row => row.status === "pending" && predicate(row));
expect(found, "One saved approval for this stage must exist").toHaveLength(1);
return found[0]!;
};
const plan = await pending(row => row.kind === "request_checkbox_confirmation" && row.payload?.target?.key === "plan");
expect(plan.payload.options.map((option: Row) => option.label).sort()).toEqual([DRAFT_OPTION, PUBLISH_OPTION].sort());
await input.page.getByRole("checkbox", { name: DRAFT_OPTION, exact: true }).last().check();
await input.page.getByRole("checkbox", { name: PUBLISH_OPTION, exact: true }).last().uncheck();
await approve(plan, "plan-accepted");
await input.snapshot("hiring-proposed");
await input.capture("hiring-proposed", "Hiring waits for approval", "hiring-proposed.png");
const writer = approvalHistoryWriter(input.marker);
const hiring = await pending(row => row.kind === "request_checkbox_confirmation" && !row.payload?.target
&& row.payload?.options?.length === 1 && row.payload.options[0].label.includes(writer));
await input.page.getByRole("checkbox", { name: hiring.payload.options[0].label, exact: true }).last().check();
await approve(hiring, "hiring-accepted");
await input.snapshot("execution-proposed");
await input.capture("execution-proposed", "Execution waits for Start drafting", "execution-proposed.png");
const execution = await pending(row => row.kind === "request_confirmation" && !row.payload?.target
&& /start drafting/i.test(row.payload?.prompt ?? ""));
await approve(execution, "accepted");
await input.snapshot("finished");
await input.page.reload({ waitUntil: "domcontentloaded" });
await input.snapshot("finished");
}