Files
PaperClipAI/tests/runner-e2e/stop-redirect-flow.ts
T
DottaandPaperclip 5bc029bf6c test: wait for native work and accounting before Stop redirect (#18463)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E journeys test the browser, native work and saved
outcomes on a selected deployment.
> - The Stop journey must interrupt real work before it sends a new
direction.
> - Its old boundary accepted a turn-start event before any command or
usage report.
> - A stopped attempt with incomplete usage correctly blocks later work
under the existing budget policy.
> - This pull request waits for a live native command and bound usage,
then checks actual accounting settlement before it sends the redirect.
> - The provider reports first usage after its initial command
completes, so the fixture finishes a harmless readiness call before the
long interruptible hold.
> - The benefit is a stronger test of Stop and continuation with the
same three-run limit.

## Linked Issues or Issue Description

Refs: #18098, #18345, #18354. Related accounting work: #18035.
Integrated upstream prerequisites: #18387, #18465 and #18409. Duplicate
selector PR #18472 was closed as superseded by actual merged #18409.

**What happened?**

A QA attempt clicked Stop soon after a native turn started. The owned
event stream had no command or usage report. Its cancelled run retained
incomplete accounting. The saved redirect received a skipped wake, and
no third run started. The budget guard must keep that missing usage
visible.

**Expected behavior**

The journey must first prove that actual native work is active and that
its current turn has reported complete usage. The fixture must finish a
harmless readiness command before it starts a separate bounded hold.
After Stop and physical cleanup, it must verify a complete accounting
receipt and saved settlement before sending the redirect.

**Steps to reproduce**

The fresh staging attempt failed before Stop because its first 90-second
command ended before the provider emitted its first complete usage. Its
only later eligible commands lasted less than one second. The controlled
browser sequence reproduces that ordering without accepting it.
Completed readiness usage followed by a separate still-active hold
qualifies under the unchanged oracle. Startup alone, command alone,
usage alone, partial usage, incomplete accounting and a missing
settlement timestamp still hold the next action.

## What Changed

- Request a harmless `printf` readiness command first. Wait for its
result, then request `sleep 90` exactly once in a separate shell call
before file edits. This gives the provider an opportunity to emit
current-turn usage before the interruptible hold.
- Reuse the strict native-command observer. Require a bound current-turn
complete usage report with valid nonzero token counts. Reject stale,
partial, foreign and semantic-tool evidence.
- Wait for the stopped run's complete accounting receipt,
`accountingReceiptReady: true`, `costAccountingPending: false` and a
valid saved settlement time before redirecting. A receipt already ready
before Stop is intentional only when the exact cancelled run is actually
accounted with no pending debt; no source/sequence advance or callback
clock order is assumed.
- Exclude canonical raw, partial and complete `item.completed` usage
only when the API row and PRP envelope match the authoritative owned
run, company and agent, and their source event ID, instance ID and
positive sequence agree. Missing context keeps the strict work check. A
self-consistent accounting source can change after Stop. Keep the
original snapshot and reject real tools, turns, messages and other item
events, including productive output from a new source.
- Retain private boundary and accounting proof. Source and sequence
observations do not impose unsupported controller or clock order.
Include the new oracle in the suite definition hash. Keep existing
success-check IDs, exact three runs, model, budget, deadline, Stop proof
and final outcome checks.
- Reuse one binding validator in the active-work boundary and post-Stop
filter. Pre-Stop keeps its current Runner/current-turn restriction. Add
focused evidence tests and a standalone Chromium/control-API sequence
check. The optional pre-Stop hook leaves existing classic callers
unchanged; they do not call the accounting filter.

## Verification

- Current source is `47578138ad357fc11b8b346bfada36d6aa5ab484`, one
narrow fixture commit on reviewed
`b9668ccfba61c59eb496ba18269aee06e2921d8f`. Only the prompt constant,
two ordering controls, the existing Chromium sequence and the journey
documentation changed. All oracle functions, run counts, model, budget,
deadline and check IDs are byte-identical to b966. There is no new
upstream merge or app change.
- Fresh staging failure is preserved in the [sanitized
report](https://pages.paperclip.ing/evals-qa-smoke-41227920892-1/). It
measured harness `5d3f143d0aed4dc7cf472b7891ea3d4d95ab843b` against app
`b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f`; the reviewed Stop helpers
and its flow were unchanged in that harness. The original grade has
`stop_boundary_already_settled`, two succeeded native runs, and complete
cleanup. Authentication, setup, collection and cleanup passed; the
journey failed. This attempt remains failed.
- Causal timing evidence: the first shell command started at event108
and completed at111 about 90 seconds later. The first structured
complete current-turn usage arrived at113, after the hold ended.
Replaying the exact retained event metadata never qualifies that hold.
The only later eligible command intervals lasted about 0.90 and 0.18
seconds. The emitter publishes positive usage when the provider sends
its assistant-message usage update. The fixture now completes a harmless
first call before requesting the long hold; no usage is supplied or
invented by setup.
- On current source, all 137 focused tests and journey contracts passed.
Use `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts
product-journeys/stop-redirect-evidence.test.ts
product-journeys/agent-evidence.test.ts
product-journeys/active-followup.test.ts native-event-envelope.test.ts
product-journeys/contracts.test.ts --maxWorkers 1`. New controls reject
every prefix when first usage arrives only after the hold completes.
They accept the same-turn sequence of completed readiness, genuine
complete usage and a separate active hold, and reject it after that hold
completes.
- Manual Chromium qualification passed: `node --import
./cli/node_modules/tsx/dist/loader.mjs
tests/runner-e2e/stop-redirect-boundary.browser.mjs`. The browser does
not click Stop for the failed ordering or completed readiness alone. It
clicks once for a separate active hold with bound usage. It sends one
redirect only after actual settlement. Foreign usage, productive output
and missing accounting still fail. This isolated control uses controlled
responses and does not call a model or the deployed app.
- Strict scoped TypeScript, maps, diff and secret checks passed. The
official credential-free Node24 QA runtime-import preflight passed
without network access. Full local E2E typing previously encountered old
borrowed workspace declarations; clean CI typecheck passed on b966. A
full local repository build/test was not repeated for this four-file
fixture change. Fresh canonical CI must supply the clean full gates.
- Historical binding proof remains valid: the exact old d499 filter
failed 17 of the final61 Stop tests. The shared validator checks
authoritative run/company/agent plus consistent row/envelope source
identity. Bound raw/partial/complete accounting is allowed after Stop,
including a consistent source change; productive output remains strict.
Pre-Stop still requires its current Runner and turn, complete nonzero
usage and a running command.
- Historical b966 CI `41224054720` passed all59 terminal checks, with
exact-head Greptile and Paperclip Review each5/5 and no open findings.
These are not approvals of current475. Current475 has fresh exact-head
Greptile5/5 and Paperclip Review5/5, with zero unresolved findings and
no omitted thread pages. Canonical CI
[`41229185200`](https://github.com/paperclipai/paperclip/actions/runs/41229185200)
passed on exact475. All59 checks are terminal:55 success,3 expected
skips and1 optional neutral review, with zero failed or pending checks.
No old job was retried. Fresh normal staging run41229965802 passed. Its
[sanitized
report](https://pages.paperclip.ing/evals-qa-smoke-41229965802-1/)
agrees with the independently read final original grade:11 checks, all5
phases, exactly3 native runs and cleanup passed. The final manifest is
complete, with cleanup complete and no pending evidence. Smoke, report
publication and Slack outcome jobs all succeeded.
- The live pass measured harness
`6be20bef6d0c5077adc1ae99562ad84e756ab167` against app
`b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f`. Its Stop oracle, flow,
shared binding validator, unit cases and Chromium control are
byte-identical to reviewed475. The shared agent-flow file adds only the
hiring import and early hiring dispatch; all remaining bytes match. This
records the actual harness and app identities separately.
- Canonical run readback confirmed the first run succeeded, the second
was cancelled and the third succeeded. The cancelled run had a ready
native-producer accounting receipt, no pending accounting and a saved
settlement time. After verified file copyback, Core queued final
destruction and followed its normal five-minute cleanup backoff. The
exact lease naturally reached expired/cleanup-success with a bound
destroyed receipt before the third run started. No operator cleanup,
source change, timeout extension or workflow retry was used. The earlier
failed staging attempt remains failed.
- Current475 is ready for the normal reviewed merge and QA admission
process. Source stays frozen; merge and configuration changes belong to
that process.

## Risks

- Some provider profiles may not expose a valid current-turn usage
report while the command runs. The stronger boundary fails closed in
that case.
- Accounting may remain incomplete after Stop. The redirect stays unsent
within the original deadline. This change does not waive missing debt,
treat it as zero, raise a budget or change provider behavior.
- This repairs the test boundary. It does not prove or repair a general
provider shutdown-usage issue. A separate local late-usage ordering
control is not attributed to this QA incident.

## Model Used

- OpenAI Codex, GPT-6 family. The exact deployment variant and
context-window size are not exposed in this session. Used code
reasoning, repository tools, test execution and browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 13:49:02 -05:00

68 lines
3.9 KiB
TypeScript

import type { Page } from "@playwright/test";
import { isBoundNativePrpEventRow, type NativePrpRunBinding } from "./native-event-envelope.js";
type Row = Record<string, any>;
/** Work emitted by a retired turn, excluding accounting and control-plane bookkeeping. */
export function nativeWorkEvents<T extends Row>(events: T[], binding?: NativePrpRunBinding): T[] {
return events.filter(row => {
const event = row.payload?.prpEvent;
const kind = event?.eventType;
if (kind === "item.completed" && row.eventType === kind
&& isBoundNativePrpEventRow(row, binding) && event.payload?.kind === "usage") return false;
return typeof kind === "string" && /^(tool\.execution\.|turn\.|message\.|item\.)/.test(kind);
});
}
/** Detect post-cancellation output even if it arrived before the first snapshot. */
export function nativeCancellationHasNoLaterWork(events: Row[], binding?: NativePrpRunBinding): boolean {
const work = nativeWorkEvents(events, binding);
const cancellation = work.findIndex(row => row.payload.prpEvent.eventType === "turn.cancelled");
return cancellation >= 0 && cancellation === work.length - 1;
}
/** A claimed run is not proof that a cancellable native turn has started. */
export function activeNativeTurnStarted(run: Row | undefined, events: Row[]): boolean {
const latestTurn = events.map(row => row.payload?.prpEvent?.eventType)
.filter(kind => ["turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(kind)).at(-1);
return run?.runtimeMode === "native" && run.status === "running" && latestTurn === "turn.started";
}
/** Shared story order; target adapters supply authoritative retirement observations. */
export async function stopAndRedirectThroughUi(input: {
page: Page; nonce: string;
beforeStop?(): Promise<void>;
awaitRetirement(): Promise<void>;
beforeRedirect(): Promise<void>;
reply(message: string): Promise<unknown>;
}) {
await input.beforeStop?.();
await input.page.getByTestId("task-chat-composer-stop").last().click();
await input.awaitRetirement();
await input.beforeRedirect();
const marker = `The studio is ready. Reference ${input.nonce}.`;
const reply = await input.reply(`Change direction. Leave the project as it is. Reply with just this short note: "${marker}"`);
await input.page.reload();
return { marker, reply };
}
/** The native control plane must acknowledge this run's Stop, including resource release. */
export function nativeStopAcknowledged(run: Row, companyId: string, issueId: string, userId: string) {
const stop = run.resultJson?.nativeCancellation;
const result = run.resultJson ?? {};
const startup = result.startupCancellation;
const cancellation = result.cancellation;
return run.runtimeMode === "native" && run.status === "cancelled" && run.companyId === companyId && (run.nativeIssueId ?? run.issueId) === issueId
&& stop?.schema === "paperclip.native-cancellation.v1" && stop.companyId === companyId
&& stop.runId === run.id && stop.issueId === issueId && stop.scope === "run" && stop.dispatched === true
&& stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only"
&& stop.effects?.length === 1 && stop.effects[0] === "release_run_resources"
&& typeof stop.intentAuditId === "string" && typeof stop.acknowledgementAuditId === "string"
&& stop.intentAuditId !== stop.acknowledgementAuditId
&& /^native-cancellation:[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(stop.intentId ?? "")
&& result.cancelledByActorType === "user" && result.cancelledByUserId === userId
&& cancellation?.source === "operator" && cancellation.expected === true
&& cancellation.initiator?.type === "user" && cancellation.initiator.id === userId
&& typeof startup?.requestedAt === "string" && Number.isFinite(Date.parse(startup.requestedAt))
&& startup.beforeNativeSelection === false;
}