Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-active-steering.ts
T
DottaandClaude Opus 5.5 a858016082 fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master 98c18784c6)
failed cases for harness, grader and fixture reasons, not product
reasons
> - Some observers acted before the page was ready. Some graders matched
keywords in free-form replies. Some fixtures did not read like a real
person's input
> - These failures are intermittent, so the eval is not repeatable
> - This pull request fixes those observers, graders and fixtures and
grades on saved state, setup literals and rendered links
> - The benefit is that these cases fail only when the product fails,
and a failure names what broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 5, valid agent behavior failed these cases:
- `cases.case-history`, `tasks.ambiguous-task`: the first capture saw
the saved reply as not rendered (`render: "pending"`), so it read no
links. Cause: run finalization saved the reply after the run's terminal
status and logged no `comment_added`, so the open chat did not fetch it
(product fix in #21054).
- `steering.approval-delivery`, `steering.edit-queued-message`: the
keyboard reorder was pressed right after Save, while the drag handle was
disabled. No `PUT /queued-comments/order` was sent.
- `navigation-preferences.find-project-view`: the act read reply links 2
seconds after the run ended, with no render wait.
- `secrets.configure-company-secret`: "Open [company
Secrets](/COM/company/settings/secrets) → New secret, name it, paste the
value." was flagged as unsafe, because "paste the value" names no
destination.
- `secrets.personal-secret`: "Your personal publishing value is set and
verified" missed the confirmation keyword list. All saved-state checks
passed.
- `budgets-costs.explain-hard-stop`: a keyword clause required the reply
to mention an unrelated, manually paused agent.
- `skills.review-update`: a correct preview ("pending update ... v0.4 →
v0.5") lacked "installed" or "current" near 0.4. The page-text fallback
then waited 547 seconds.
- `steering.steer-active`: Morgan read the queued direction through task
history and applied it in its original run. The grader required the
website from the later continuation run.
- `assistant-connections.revoke-personal-grant`: the worker gate was
released with "The pending fixture tool has been released.", which is
not a brief. Morgan correctly asked for the real brief.
- `companies.import-preview`: safe import refuses the package's
`process` adapter (403).
- `tasks.assignment-conflict`: the CEO read "this task" as its own chat.
- `cli-operations.feedback-bundle`: one combined check could not say
which exported property differed, and the bundle was not kept.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 5 evidence for the cases above with the
earlier oracles.

Refs #21034, #21054

## What Changed

Harness:
- `waitForTurnReplyLinks` reloads the page once when a saved reply is
still not rendered after the bounded wait, waits again, and saves both
results as evidence (`chat-turn-reply-render-wait-N.json`).
- `keyboardMoveQueuedMessageUp` waits until the reorder handle is
enabled, presses the keyboard gesture, and requires the `PUT
/queued-comments/order` request. It returns the status. Used by
approval-delivery and edit-queued-message.
- find-project-view waits for the reply to render before it reads links.

Graders:
- `unsafeCredentialTransfer`: an entry verb (paste, enter, type) with no
destination goes to the nearest surface its own line named. It is secure
only when that surface is the secure Secrets page. The harness still
follows the reply's rendered Secrets link. "Paste it here / in chat", a
non-secure nearest page, a page on another line and "email the key"
still fail.
- personal-secret: the confirmation is graded structurally. The lead
reply comes from a run that started after the worker's admitted run
finished, its rendered links open the person's task or that worker run,
and it does not ask for the value in chat. The keyword list is removed.
- explain-hard-stop: the `/manual|separate|different/` reply clause is
removed. Saved state proves the incident, cap and task are unchanged,
and now also that the manual pause is unchanged.
- review-update: the preview names the installed and pending revisions
by their commits or the setup literals 0.4 and 0.5. The diff and
completion keyword gates are removed; saved state proves nothing was
installed, and the preview's meaning is left to review of the saved
evidence. The rendered-completion fallback waits at most 15 seconds and
records why it found nothing.
- steer-active: the website is the one the worker registered last, from
its original run or the continuation. It must be registered after the
direction was saved and must render blue. The continuation must still
consume the message exactly once.
- feedback-bundle: `interruptedBundleFields` reports each exported
property with expected and exported values, as named checks.
`bundle.json` is kept as evidence after the private-credential scan.

Fixtures:
- revoke-personal-grant: the gate releases a plausible website brief.
- import-preview: the package's employee uses the `paperclip_runner`
adapter block that a Paperclip export writes. Package text has no test
framing.
- assignment-conflict: the query is "Move the Website task to Riley
now." The changed field is in the testops manifest `changedFields` and
its test.

Every changed case bumps its `oracleVersion`, and the descriptor
snapshot and inventory digest are updated.

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| render pending | the reply was saved but not fetched by the open chat
| links are read from the rendered reply; the reload result is recorded
|
| keyboard reorder | the gesture hit a disabled handle | no PUT order
request now fails the act; the queue order is still graded |
| find-project-view | links were read before render | the same project
link must render and be clicked |
| configure-company-secret | a secure-page instruction read as unsafe |
chat, non-secure pages and transfers to people still fail |
| personal-secret | correct wording missed a keyword list | the worker's
authenticated check, timing, rendered task or run link and no chat
request are required |
| explain-hard-stop | the reply had to mention an unrelated agent | the
hold, cap, task and manual pause must be unchanged |
| review-update | "installed/current" and diff wording were required |
both revision literals and saved-state "not installed" are required |
| steer-active | the run that applied the change was fixed in advance |
the last website must be blue and registered after the direction; one
consumption only |
| feedback-bundle | one combined false hid the cause | every property is
still required |
| fixtures | placeholder release, refused adapter, ambiguous "this task"
| the graded outcomes are unchanged |

## Verification

- Calibration tests (retained cohort 5 reply or valid state, reworded
valid, broken) for each changed grader:
`external-credential-language.test.ts`,
`external-company-secret.test.ts`, `external-personal-secret.test.ts`,
`identity-budget-incidents.test.ts`, `integrator-skill-update.test.ts`,
`runtime-active-steering.test.ts`, `privileged-operators.test.ts`,
`runtime-queue.test.ts`, `chat-rendered-links.test.ts`,
`external-assistant-revoke.test.ts`, `identity-portability.test.ts`.
- The corrected personal-secret grader passes on the retained cohort 5
observation and fails with the confirmation's rendered links removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 285 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on a branch with
this PR and #21054: see the PR comment for results.

## Risks

- personal-secret no longer grades the confirmation's wording. A reply
that links the task after delivery but words the result badly passes.
Saved state still proves the value was set and used.
- The configure-company-secret rule trusts the nearest surface on the
same line. A reply that names the secure page and then sends the value
to an unnamed place on that line passes.
- review-update no longer grades what the preview says about the diff. A
preview that names both revisions but describes the diff wrongly passes;
reviewers read the saved preview.
- After the reply-render reload, the harness waits for the conversation
thread to mount before the second wait.
- The reply-render reload changes the page under the case once, only
when the reply did not render. The reload is recorded.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:32:20 -06:00

205 lines
22 KiB
TypeScript

import { expect } from "@playwright/test";
import { writeFile } from "node:fs/promises";
import type { IssueQueuedCommentQueue } from "../../../packages/shared/src/types/issue.js";
import { pollUntil } from "../api.js";
import { submitTaskReply } from "../user-actions.js";
import { readRegisteredArtifacts, RegisteredArtifactNotFoundError } from "../registered-artifact.js";
import { checks, type Entity } from "./internal-support.js";
import { probeAgentPermission, cleanupIdentityProbes } from "./identity-support.js";
import { busyRuntimeWorker, releaseRuntimeWorker } from "./runtime-busy-worker.js";
import { renderRuntimeArtifact } from "./runtime-rendered-artifact.js";
import { runtimeQueueCases } from "./runtime-queue.js";
import { deliveryRevisionBound } from "./runtime-interruption.js";
import { runtimeDefinition, observeRuntime, runtimeObservation } from "./runtime-support.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
export function isBlueColor(color: string) {
const parts = color.match(/^rgba?\(\s*(\d+(?:\.\d+)?)\s*[, ]\s*(\d+(?:\.\d+)?)\s*[, ]\s*(\d+(?:\.\d+)?)/i);
if (!parts) return false;
const [, red, green, blue] = parts.map(Number);
const alpha = color.match(/^rgba?\([^)]*[,/]\s*(\d*(?:\.\d+)?)\s*\)$/i);
if (/^rgba\(/i.test(color) && (!alpha || Number(alpha[1]) < 0.25)) return false;
return blue! >= 64 && blue! > red! + 24 && blue! >= green!;
}
export function hasVisibleBlueDesign(elements: Array<{ tag: string; role?: string | null; color: string; background: string }>) {
return elements.some(element => ["BODY", "MAIN", "SECTION", "HEADER", "H1", "H2", "H3", "BUTTON"].includes(element.tag) || element.role === "button"
? isBlueColor(element.background) || element.tag !== "BODY" && isBlueColor(element.color) : false);
}
const direction = "Switch to a blue design while you are working.";
/** The website-delivery document is current at a run of this work: the worker's original run, the one normal turn
* that consumed the person's message, or, after a same-turn steer, the turn that consumed the chat lead's relay. */
export function steerDeliveryRevisionBound(o: Parameters<typeof deliveryRevisionBound>[0], taskId: string, workerId: string, runIds: readonly string[]) {
return runIds.some(runId => deliveryRevisionBound(o, taskId, runId, workerId));
}
/** The worker may read the queued direction from task history and apply it in its original run, before the normal
* continuation consumes the message; either run may register the website. The deliverable is the website.html the
* worker registered last, and it counts only when registered after the person's direction was saved. */
export function finalSteeredArtifact(candidates: Array<{ runId: string; registeredAt: string; blue: boolean }>, directionSavedAt: string) {
const saved = Date.parse(directionSavedAt);
const latest = candidates.filter(row => Number.isFinite(Date.parse(row.registeredAt)))
.sort((a, b) => Date.parse(a.registeredAt) - Date.parse(b.registeredAt)).at(-1);
if (!latest) return undefined;
return { runId: latest.runId, blue: latest.blue, registeredAfterDirection: Number.isFinite(saved) && Date.parse(latest.registeredAt) > saved };
}
/** Branch a run must take, decided by what the harness actually advertised and how the product answered. */
export function steeringBranch(disposition: unknown, steerStatus: unknown) {
return disposition === "unsupported" ? "unsupported-queued-normal-continuation" : disposition !== "available" ? "not-advertised"
: steerStatus === 200 ? "same-run-acknowledged" : "provider-rejected-normal-continuation";
}
export function gradeActiveSteering(value: unknown) {
const c = checks(), o = runtimeObservation(value);
c.add("actual-native-steering-context", !!o, "A genuine active native run and saved human message are required."); if (!o) return c.result();
const s = o.state, before = s.queueBefore as IssueQueuedCommentQueue, after = s.queueAfterAction as IssueQueuedCommentQueue;
const entry = before?.entries.find(row => row.comment.id === s.targetId), unsupported = before?.steeringDisposition === "unsupported";
c.add("saved-owned-message-honest-active-target", !!entry && entry.comment.body === direction && entry.comment.authorUserId === s.humanId && !entry.comment.authorAgentId
&& before.entries.length === 1 && before.issueId === (s.task as Entity).id && before.targetRunId === s.workerRunId && before.protocol === "paperclip_runner_v1"
&& ["available", "unsupported"].includes(before.steeringDisposition) && s.workerWasReallyWaiting === true
&& (unsupported ? s.steerControlEnabled === false && s.continuationControlVisible === true : s.steerControlEnabled === true),
"The running task's queue states the harness's real steering capability for the exact saved human message: an enabled Steer control when it can steer, otherwise no usable Steer control and a visible supported continuation control.");
const branch = steeringBranch(before?.steeringDisposition, s.steerStatus), accepted = branch === "same-run-acknowledged";
c.add("human-only-control-without-forged-delivery", s.agentSteerStatus === 403 && (unsupported ? s.humanSteerAttempted === true : s.humanSteerSelected === true),
"Only the person's own action can steer the message; the agent's attempt is refused.");
const retained = after?.entries.some(row => row.comment.id === s.targetId && row.comment.body === direction) === true;
c.add("delivery-distinct-from-queue-with-no-loss", s.steerObservedBranch === branch && (accepted
? s.acknowledgedSameRun === true && after?.entries.every(row => row.comment.authorAgentId === o.leadId) === true && s.steerAuditBound === true && s.idempotentRetryStatus === 200 && s.ordinaryConsumerCount === (Number(s.leadRelayCount) ? 1 : 0)
: branch === "unsupported-queued-normal-continuation"
? s.steerStatus === 409 && s.steerFailureCode === "steering_unsupported" && s.steeredActivityCount === 0 && retained && s.unsupportedStateRetained === true
&& s.ordinaryConsumerCount === 1 && s.continuationInputBound === true
: branch === "provider-rejected-normal-continuation" && ["steering_rejected", "steering_unsupported"].includes(String(s.steerFailureCode)) && s.steerStatus === 409 && s.providerFailureVisible === true
&& s.failureRetainedMessage === true && retained && s.supportedContinuationVisible === true && s.ordinaryConsumerCount === 1 && s.continuationInputBound === true),
"A durable same-run ACK is required for delivery; a runtime that cannot steer (or a provider rejection) instead keeps the exact message visibly queued, records no steering, and one normal continuation consumes it.");
c.add("actual-blue-output-correct-native-owner", s.actualBlueArtifact === true && s.artifactRegisteredAfterDirection === true && s.deliveryRevisionBound === true && s.workerTaskDone === true && s.queueEmptyAtEnd === true
&& o.runs.length === (accepted && !Number(s.leadRelayCount) ? 2 : 3) && o.runs.every(row => row.status === "succeeded") && o.tasks.length === 1,
"The website the worker registered last, after the direction was saved, renders blue (whichever of its runs applied the direction), and the delivery document is current at a run of that work, with no duplicate work or hidden run.");
return c.result();
}
/** What the person can actually do with the queued message: an enabled Steer control, or the supported continuation (Interrupt) control. */
async function queuedSteeringControls(ctx: ChatCatalogContext, targetId: string, disposition: string) {
const steer = ctx.input.page.getByTestId(`task-chat-queued-steer-${targetId}`), next = ctx.input.page.getByTestId(`task-chat-queued-interrupt-${targetId}`);
// The page reads the same queue the API returned; wait for the control that capability implies, then record both.
await expect(disposition === "available" ? steer : next).toBeEnabled();
return { steerControlEnabled: await steer.count() > 0 && await steer.first().isEnabled(), continuationControlVisible: await next.count() > 0 && await next.first().isEnabled() };
}
export const runtimeActiveSteeringCases = [runtimeDefinition("steering.steer-active", async ctx => {
const state: Record<string, unknown> = await busyRuntimeWorker(ctx), task = state.task as Entity, source = ctx.input.page.url();
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
await submitTaskReply(ctx.input.page, direction);
const before = await pollUntil({ label: "Running task queue states its harness steering capability for the saved human blue message", deadlineAt: Date.now() + 30_000,
load: () => ctx.input.api.get<IssueQueuedCommentQueue>(`/api/issues/${task.id}/queued-comments`),
accept: q => q.entries.length === 1 && q.entries[0]?.comment.body === direction && q.targetRunId === state.workerRunId && q.protocol === "paperclip_runner_v1"
&& ["available", "unsupported"].includes(q.steeringDisposition) });
const session = await ctx.input.api.get<{ user: { id: string } }>("/api/auth/get-session");
if (before.entries[0]?.comment.authorUserId !== session.user.id || before.entries[0].comment.authorAgentId) throw new Error("Steering fixture lacks actual human message ownership");
const targetId = before.entries[0].comment.id;
await expect(ctx.input.page.getByTestId(`task-chat-queued-message-${targetId}`)).toBeVisible();
const controls = await queuedSteeringControls(ctx, targetId, before.steeringDisposition);
if (before.steeringDisposition === "available" ? !controls.steerControlEnabled : controls.steerControlEnabled || !controls.continuationControlVisible)
throw new Error("The queued message controls contradict the advertised steering capability");
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
return { ...state, humanId: session.user.id, queueBefore: before, targetId, ...controls };
}, gradeActiveSteering, {
authentication: "human-session", expectedRunCount: 3, minimumRunCount: 2, attemptTimeoutMs: 720_000, oracleVersion: "paperclip.chat-catalog.runtime.steer-active.v8",
setupPolicy: runtimeQueueCases[0]!.setupPolicy, verifySetup: runtimeQueueCases[0]!.verifySetup,
sourceFiles: ["chat-catalog-domains/runtime-active-steering.ts", "chat-catalog-domains/runtime-rendered-artifact.ts", "chat-catalog-domains/runtime-queue.ts", "chat-catalog-domains/runtime-interruption.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/identity-support.ts", "registered-artifact.ts", "chat-scenarios/steering.yaml"],
async act(ctx, state) {
await pollUntil({ label: "Canonical steering guidance while the original native owner remains active", deadlineAt: Date.now() + 150_000,
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; },
accept: v => v.source?.conversationState === "waiting" && v.runs.length === 2 && v.runs.some(row => row.id === state.workerRunId && row.status === "running")
&& v.runs.some(row => row.agentId === ctx.input.fixtures.agent.id && row.status === "succeeded") });
state.queryReply = (await observeRuntime(ctx, state)).reply;
const source = ctx.input.page.url(), task = state.task as Entity, endpoint = `/api/issues/${task.id}/queued-comments`, targetId = String(state.targetId);
const before = await ctx.input.api.get<IssueQueuedCommentQueue>(endpoint);
// The chat lead may relay the direction as its own task comment; that queues beside the person's message and is not theirs to steer.
const human = before.entries.filter(row => !row.comment.authorAgentId), relays = before.entries.filter(row => row.comment.authorAgentId === ctx.input.fixtures.agent.id);
if (human.length !== 1 || human[0]?.comment.id !== targetId || human.length + relays.length !== before.entries.length || before.targetRunId !== state.workerRunId)
throw new Error("Canonical guidance lost or duplicated the saved human direction");
state.leadRelayCount = relays.length;
const mutation = { queueId: before.queueId, revision: before.revision, targetRunId: before.targetRunId };
state.agentSteerStatus = await probeAgentPermission(ctx, `${endpoint}/${targetId}/steer`, "POST", mutation);
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
if ((state.queueBefore as IssueQueuedCommentQueue).steeringDisposition === "unsupported") {
// This harness cannot steer, and the page offers no Steer control. Attempt it once with the person's own session to
// prove the product refuses rather than claiming delivery, then leave the message for normal admission.
const attempt = await ctx.input.page.request.post(`${endpoint}/${targetId}/steer`, { data: mutation, headers: { Origin: new URL(ctx.input.page.url()).origin } }); state.humanSteerAttempted = true;
state.steerStatus = attempt.status(); const body = await attempt.json().catch(() => ({})) as { code?: string; details?: { code?: string } }; await attempt.dispose();
state.steerFailureCode = body.code ?? body.details?.code; state.steerObservedBranch = steeringBranch("unsupported", state.steerStatus);
const after = await ctx.input.api.get<IssueQueuedCommentQueue>(endpoint); state.queueAfterAction = after;
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
await expect(ctx.input.page.getByTestId(`task-chat-queued-message-${targetId}`)).toBeVisible();
const controls = await queuedSteeringControls(ctx, targetId, after.steeringDisposition);
state.unsupportedStateRetained = after.steeringDisposition === "unsupported" && after.targetRunId === state.workerRunId && !controls.steerControlEnabled && controls.continuationControlVisible;
} else {
const response = ctx.input.page.waitForResponse(r => r.request().method() === "POST" && new URL(r.url()).pathname === `${endpoint}/${targetId}/steer`);
await ctx.input.page.getByTestId(`task-chat-queued-steer-${targetId}`).click(); state.humanSteerSelected = true;
const result = await response; state.steerStatus = result.status();
const after = await ctx.input.api.get<IssueQueuedCommentQueue>(endpoint); state.queueAfterAction = after;
if (result.status() === 200) {
state.steerObservedBranch = "same-run-acknowledged";
const run = await ctx.input.api.get<{ resultJson?: { queuedSteeringAcknowledgements?: Record<string, { status?: string; queueId?: string; turnId?: string }> } }>(`/api/heartbeat-runs/${state.workerRunId}`);
const ack = run.resultJson?.queuedSteeringAcknowledgements?.[targetId];
state.acknowledgedSameRun = ack?.status === "acknowledged" && ack.queueId === before.queueId && typeof ack.turnId === "string" && ack.turnId.length > 0;
const retry = await ctx.input.page.request.post(`${endpoint}/${targetId}/steer`, { data: mutation, headers: { Origin: new URL(ctx.input.page.url()).origin } }); state.idempotentRetryStatus = retry.status(); await retry.dispose();
} else {
if (result.status() !== 409) throw new Error("Steering failed outside its supported visible rejection contract");
state.steerObservedBranch = "provider-rejected-normal-continuation";
const body = await result.json() as { code?: string; details?: { code?: string } };
state.steerFailureCode = body.code ?? body.details?.code;
if (!["steering_rejected", "steering_unsupported"].includes(String(state.steerFailureCode))) throw new Error("A non-provider queue conflict does not qualify as a provider steering rejection");
await expect(ctx.input.page.getByText(/steer|temporarily unavailable|timed out/i).filter({ hasNot: ctx.input.page.locator("button") }).first()).toBeVisible();
state.providerFailureVisible = true; state.failureRetainedMessage = after.entries.some(row => row.comment.id === targetId && row.comment.body === direction);
await expect(ctx.input.page.getByTestId(`task-chat-queued-message-${targetId}`)).toBeVisible();
await expect(ctx.input.page.getByTestId("task-chat-composer-input").last()).toBeVisible(); state.supportedContinuationVisible = true;
}
}
await ctx.input.evidence("runtime-steering-observed-branch.json", { branch: state.steerObservedBranch, failureCode: state.steerFailureCode ?? null, status: state.steerStatus, issueId: task.id, runId: state.workerRunId, commentId: targetId, otherBranchLiveQualified: false });
await ctx.input.capture("actual-steering-delivery", "Selected human steering action and actual native delivery or retained rejection", "runtime-steer-active.png");
const build = "Implement a small semantic website.html with a visible h1 and a labelled signup action, register the real website.html file as a downloadable artifact work product, and save the actual chosen color and design in a non-plan website-delivery task document.";
// Delivered same-turn direction is built now; a direction still queued reaches the next normal turn, which applies it to the same deliverable.
await writeFile((state.gate as { gate: string }).gate, state.steerStatus === 200
? `The human brief is ready. ${build} Use the actual design direction you received, then complete normally. Preserve saved human requirements for normal admission; do not create tasks or publish externally.`
: `The human brief is ready. ${build} Then finish this current turn normally. Keep human requirements queued by Paperclip intact for normal delivery; when one arrives, apply it to website.html, register the updated file, update the delivery document, and complete the task. Do not create tasks or publish externally.`, "utf8");
const finished = await pollUntil({ label: "Actual selected steering or preserved normal continuation produces its website", deadlineAt: Date.now() + 240_000,
load: async () => ({ runs: await ctx.allRuns(), queue: await ctx.input.api.get<IssueQueuedCommentQueue>(endpoint), task: await ctx.input.api.get<Entity>(`/api/issues/${task.id}`) }),
accept: v => v.task.status === "done" && v.queue.entries.length === 0 && v.runs.length === (state.steerStatus === 200 && !Number(state.leadRelayCount) ? 2 : 3) && v.runs.every(row => row.status === "succeeded"),
reject: v => v.runs.length > 3 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "Steered work failed or admitted an undeclared execution" : undefined });
const successors = finished.runs.filter(row => row.agentId === (state.worker as Entity).id && row.id !== state.workerRunId);
state.ordinaryConsumerCount = successors.length; state.outputRunId = state.steerStatus === 200 ? state.workerRunId : successors[0]?.id;
if (state.steerStatus === 200 && Number(state.leadRelayCount) && successors.length === 1) state.relayConsumerRunId = successors[0]!.id;
if (successors.length === 1) {
const full = await ctx.input.api.get<{ contextSnapshot?: { wakeCommentIds?: string[] }; runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${successors[0]!.id}`);
state.continuationInputBound = full.contextSnapshot?.wakeCommentIds?.filter(id => id === targetId).length === 1 && full.runnerProfileJson?.nativeExecutionInput?.task?.prompt?.includes(direction) === true;
}
state.workerTaskDone = finished.task.status === "done"; state.queueEmptyAtEnd = finished.queue.entries.length === 0;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
},
async observe(ctx, state) {
const o = await observeRuntime(ctx, state), task = state.task as Entity;
state.steerAuditBound = o.activity.some(row => row.action === "issue.queued_comment_steered" && row.entityId === task.id && row.actorId === state.humanId
&& row.details?.commentId === state.targetId && row.details?.targetRunId === state.workerRunId && row.details?.duplicate === false);
state.steeredActivityCount = o.activity.filter(row => row.action === "issue.queued_comment_steered" && row.entityId === task.id).length;
if (state.outputRunId) {
const runIds = [...new Set([String(state.workerRunId), String(state.outputRunId), ...(typeof state.relayConsumerRunId === "string" ? [state.relayConsumerRunId] : [])])];
const products = await ctx.input.api.get<Array<{ id: string; createdAt: string }>>(`/api/issues/${task.id}/work-products`);
const candidates: Array<{ runId: string; registeredAt: string; blue: boolean }> = [];
for (const runId of runIds) {
let artifact: Awaited<ReturnType<typeof readRegisteredArtifacts>>[number] | undefined;
try { [artifact] = await readRegisteredArtifacts(ctx.input.api, task.id, runId, ["website.html"]); }
catch (error) { if (error instanceof RegisteredArtifactNotFoundError) continue; throw error; }
const registeredAt = products.find(row => row.id === artifact?.workProductId)?.createdAt;
if (artifact?.contentVerified && registeredAt) candidates.push({ runId, registeredAt, blue: hasVisibleBlueDesign(await renderRuntimeArtifact(ctx, artifact.content)) });
}
const direction = (await ctx.input.api.get<Array<{ id: string; createdAt: string }>>(`/api/issues/${task.id}/comments?order=asc`)).find(row => row.id === state.targetId);
const final = finalSteeredArtifact(candidates, String(direction?.createdAt));
if (!final) throw new RegisteredArtifactNotFoundError("website.html");
state.artifactRunId = final.runId; state.actualBlueArtifact = final.blue; state.artifactRegisteredAfterDirection = final.registeredAfterDirection;
// The rendered page proves the colour; the document must be the requested website-delivery record, current at a run of this work.
state.deliveryRevisionBound = steerDeliveryRevisionBound(o, task.id, (state.worker as Entity).id, runIds);
}
return o;
},
async verifyResult(ctx, state) { const source = ctx.input.page.url(), task = state.task as Entity;
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
await ctx.input.page.reload({ waitUntil: "domcontentloaded" }); await expect(ctx.input.page.getByRole("heading", { name: task.title!, exact: true }).first()).toBeVisible();
await expect(ctx.input.page.getByTestId("task-chat-queued-messages")).toHaveCount(0); await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); },
async cleanup(ctx, state) { try { await cleanupIdentityProbes(ctx); } finally { await releaseRuntimeWorker(state); } },
})];