mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents/<id>/runtime`. The app then rewrote the route to `/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
186 lines
20 KiB
TypeScript
186 lines
20 KiB
TypeScript
import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely, chatCatalogNativePhaseRemainingBudget } from "../chat-catalog-contract.js";
|
|
import { expect } from "@playwright/test";
|
|
import { writeFile } from "node:fs/promises";
|
|
import type { IssueComment, IssueQueuedCommentQueue } from "../../../packages/shared/src/types/issue.js";
|
|
import { sendChatMessage } from "../chat-flow.js";
|
|
import { pollUntil } from "../api.js";
|
|
import { submitTaskReply } from "../user-actions.js";
|
|
import { checks, hasLink, type Entity } from "./internal-support.js";
|
|
import { probeAgentPermission, cleanupIdentityProbes } from "./identity-support.js";
|
|
import { releaseRuntimeWorker } from "./runtime-busy-worker.js";
|
|
import { humanQueuedRuntime, probeConsumedHumanQueue } from "./runtime-queue-controls.js";
|
|
import { handledInTurn, runCommentReceipts, runtimeQueueCases, settledQueue } from "./runtime-queue.js";
|
|
import { continuationCarriesOwed, owedAfterTurn, type CommentReceipts, type QueuedFollowUp } from "./runtime-cited-delivery.js";
|
|
import { runtimeDefinition, runtimeObservation, observeRuntime } from "./runtime-support.js";
|
|
|
|
const contactText = "Also add a contact section to the website.";
|
|
/** Native prompts embed task comments as JSON, so a multi-line body appears with escaped newlines. */
|
|
export function promptCarries(prompt: string | undefined, body: string) {
|
|
return !!prompt && body.trim().length > 0 && (prompt.includes(body) || prompt.includes(JSON.stringify(body).slice(1, -1)));
|
|
}
|
|
/** Links the reply saved in Markdown, plus links already rendered when the reply was observed. */
|
|
export function replyLinkHrefs(reply: string, rendered: readonly string[]) {
|
|
return [...new Set([...rendered, ...[...reply.matchAll(/\]\(([^)\s]+)\)/g)].map(match => match[1]!)])];
|
|
}
|
|
/**
|
|
* Contact details the later request asks for. Only these setup literals identify the contact section. The
|
|
* composer turns an email address into a mailto link, so the request names a contact person instead.
|
|
*/
|
|
export function contactDetails(nonce: string) {
|
|
return { name: "Harriet Quillfeather", phone: `+1-555-${nonce.replace(/\D/g, "").slice(-4).padStart(4, "0")}` };
|
|
}
|
|
/**
|
|
* Whole trimmed lines that belong only to the contact section of the revision that saved it: the body of a
|
|
* Markdown section whose heading carries a setup marker (label, contact name or phone), plus lines that carry a
|
|
* marker. Lines that also appear elsewhere in that revision, such as `---` or a shared bullet, are excluded.
|
|
*/
|
|
export function contactSectionLines(body: string, markers: readonly string[]): string[] {
|
|
const lines = body.split("\n").map(row => row.trim()), marked = (line: string) => markers.some(m => m.length > 0 && line.includes(m));
|
|
const inSection: string[] = []; let level = 0;
|
|
for (const line of lines) {
|
|
const heading = /^(#{1,6})\s/.exec(line)?.[1]?.length;
|
|
if (heading) level = marked(line) ? heading : heading <= level ? 0 : level;
|
|
if (line && (level > 0 || marked(line))) inSection.push(line);
|
|
}
|
|
const count = (rows: string[], line: string) => rows.filter(row => row === line).length;
|
|
return [...new Set(inSection)].filter(line => count(lines, line) === count(inSection, line));
|
|
}
|
|
/**
|
|
* None of the contact section's distinctive lines survives as a whole line in the final revision, and the
|
|
* requested contact name and phone appear nowhere in it, however reformatted. The label alone may still be quoted.
|
|
*/
|
|
export function documentContactRemoved(consumedBody: string, finalBody: string | undefined, label: string, details: readonly string[]) {
|
|
const section = contactSectionLines(consumedBody, [label, ...details]);
|
|
const kept = new Set((finalBody ?? "").split("\n").map(row => row.trim()));
|
|
return finalBody !== undefined && section.length > 0 && !section.some(line => kept.has(line))
|
|
&& !details.some(detail => detail.length > 0 && finalBody.includes(detail));
|
|
}
|
|
export function gradeDiscard(value: unknown) {
|
|
const c = checks(), o = runtimeObservation(value);
|
|
c.add("actual-pending-and-consumed-cancellation-evidence", !!o, "Both actual human discard and separately consumed input with later new direction are required."); if (!o) return c.result();
|
|
const s = o.state, before = s.queueBefore as IssueQueuedCommentQueue, after = s.queueAfterDiscard as IssueQueuedCommentQueue;
|
|
const target = before?.entries.find(row => row.comment.id === s.targetId);
|
|
c.add("only-owned-pending-message-discarded", !!target && target.comment.authorUserId === s.humanId && !target.comment.authorAgentId
|
|
&& target.comment.body === contactText && !!after && after.issueId === before.issueId && after.entries.length === 1
|
|
&& after.entries[0]?.comment.id === before.entries.find(row => row.comment.id !== s.targetId)?.comment.id
|
|
&& after.entries[0]?.comment.body === before.entries.find(row => row.comment.id !== s.targetId)?.comment.body,
|
|
"The supported human control removes precisely the eligible owned contact request and preserves its neighbour.");
|
|
c.add("history-and-ui-retain-truthful-discard", s.discardVisible === true && s.discardedCommentStatus === 404
|
|
&& o.activity.some(row => row.action === "issue.queued_comment_discarded" && row.actorType === "user" && row.actorId === s.humanId
|
|
&& row.entityId === (s.task as Entity).id && row.details?.commentId === s.targetId) && s.discardedNeverDelivered === true,
|
|
"Actual absent pending comment, retained human audit history and consuming-run wake IDs prove this pending instruction will not be delivered.");
|
|
c.add("chat-guides-supported-human-discard", String(s.queryReply ?? "").trim().length > 0 && s.queryLinkedTask === true
|
|
&& s.targetPendingAfterReply === true && s.agentDiscardStatus === 403,
|
|
"The reply links the task whose queue holds the request, the request is still pending after the reply, and the agent cannot impersonate the person's discard.");
|
|
const followUps = Array.isArray(s.queuedFollowUps) ? s.queuedFollowUps as QueuedFollowUp[] : [], receipts = s.workerCommentReceipts as CommentReceipts | undefined;
|
|
const second = followUps.find(row => row.id === s.secondContactId), owed = owedAfterTurn(followUps, receipts);
|
|
const firstDelivery = followUps.length === 2 && !!second && !receipts?.cited?.includes(String(s.targetId)) && (owed.length
|
|
? s.firstDeliveryRunCount === 3 && s.continuationConsumerCount === 1 && s.continuationInputBound === true
|
|
: s.firstDeliveryRunCount === 2 && s.continuationConsumerCount === 0);
|
|
c.add("later-input-delivered-once-before-cancellation", firstDelivery && (handledInTurn(receipts, second) ? s.consumerRunId === s.workerRunId : owed.includes(second!) && s.consumedBeforeCancellation === true),
|
|
"Each saved follow-up is delivered once before the cancellation: the running turn read it and cited it as handled, or one ordinary continuation receives every follow-up not handled in turn and none that was. The discarded request is never cited as handled.");
|
|
c.add("already-delivered-input-cannot-be-erased", s.consumedStillReadable === true
|
|
&& s.lateDiscardStatus === 409 && s.latePhaseAttributed === true && String(s.lateReply ?? "").trim().length > 0,
|
|
"The consumed request stays readable and cannot be discarded late; the later answer is attributed to its own run.");
|
|
c.add("new-direction-reaches-exactly-one-native-continuation", s.cancelDirectionRunBound === true && s.finalQueueEmpty === true
|
|
&& s.consumedLabelSaved === true && s.finalPlanWithoutContact === true
|
|
&& s.initialRevisionRetained === true && o.runs.length === Number(s.firstDeliveryRunCount) + 2 && o.runs.every(row => row.status === "succeeded") && o.tasks.length === 1
|
|
&& o.tasks[0]?.id === (s.task as Entity).id && o.tasks[0]?.status === "done",
|
|
"A distinct saved instruction reaches one real later worker, updates the saved plan and preserves prior work/history without replaying the discarded request.");
|
|
return c.result();
|
|
}
|
|
export const runtimeDiscardCases = [runtimeDefinition("steering.discard-queued-message", ctx => humanQueuedRuntime(ctx, contactText), gradeDiscard, {
|
|
authentication: "human-session", expectedHumanMessageCount: 2, expectedRunCount: 5, minimumRunCount: 4, attemptTimeoutMs: 1_020_000, oracleVersion: "paperclip.chat-catalog.runtime-discard.v8",
|
|
sourceFiles: ["chat-catalog-domains/runtime-discard.ts", "chat-catalog-domains/runtime-queue-controls.ts", "chat-catalog-domains/runtime-queue.ts", "chat-catalog-domains/runtime-cited-delivery.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/identity-support.ts", "chat-scenarios/steering.yaml"],
|
|
setupPolicy: runtimeQueueCases[0]!.setupPolicy, verifySetup: runtimeQueueCases[0]!.verifySetup,
|
|
async act(ctx, state) {
|
|
await pollUntil({ label: "Human discard guidance while Morgan is genuinely active", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
|
|
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; },
|
|
accept: v => chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && v.runs.length === 2 && v.runs.some(row => row.id === state.workerRunId && row.status === "running")
|
|
&& v.runs.some(row => row.agentId === ctx.input.fixtures.agent.id && row.status === "succeeded")) });
|
|
const answered = await observeRuntime(ctx, state);
|
|
const source = ctx.input.page.url(), task = state.task as Entity, root = `/api/issues/${task.id}`, queuePath = `${root}/queued-comments`;
|
|
const initial = await ctx.input.api.get<IssueQueuedCommentQueue>(queuePath), targetId = String(state.targetId);
|
|
state.queryReply = answered.reply; state.queryLinkedTask = hasLink({ ...answered, links: replyLinkHrefs(answered.reply, answered.links) }, "issues", task);
|
|
state.targetPendingAfterReply = initial.entries.some(row => row.comment.id === targetId && row.comment.body === contactText);
|
|
state.agentDiscardStatus = await probeAgentPermission(ctx, `${queuePath}/${targetId}`, "DELETE", { queueId: initial.queueId, revision: initial.revision });
|
|
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
|
|
await ctx.input.page.getByTestId(`task-chat-queued-discard-${targetId}`).click();
|
|
state.queueAfterDiscard = await pollUntil({ label: "Only the owned contact request is removed", deadlineAt: Date.now() + 20_000,
|
|
load: () => ctx.input.api.get<IssueQueuedCommentQueue>(queuePath), accept: q => q.entries.length === 1 && q.entries.every(row => row.comment.id !== targetId) });
|
|
await expect(ctx.input.page.getByTestId(`task-chat-queued-message-${targetId}`)).toHaveCount(0);
|
|
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
|
|
await expect(ctx.input.page.getByTestId(`task-chat-queued-message-${initial.entries.find(row => row.comment.id !== targetId)!.comment.id}`)).toBeVisible();
|
|
state.discardVisible = true; state.discardedCommentStatus = (await ctx.input.page.request.get(`${root}/comments/${targetId}`)).status();
|
|
await ctx.input.capture("actual-human-contact-discard", "Only the eligible contact request has left the real task queue", "runtime-discard.png");
|
|
// This new, explicit human request is a distinct later instruction. It
|
|
// deliberately reaches delivery before its cancellation.
|
|
const contact = contactDetails(ctx.input.nonce);
|
|
const secondText = `I have changed my mind: add a contact section to the saved website plan, labelled ${ctx.input.nonce}, listing the contact person ${contact.name} and the phone ${contact.phone}. Keep the accessible-font requirement and finish this task.`;
|
|
await submitTaskReply(ctx.input.page, secondText);
|
|
const secondQueue = await pollUntil({ label: "Distinct human contact request genuinely queued", deadlineAt: Date.now() + 20_000,
|
|
load: () => ctx.input.api.get<IssueQueuedCommentQueue>(queuePath), accept: q => q.entries.length === 2 && q.entries.some(row => row.comment.body === secondText) });
|
|
const second = secondQueue.entries.find(row => row.comment.body === secondText)!;
|
|
state.secondContactId = second.comment.id;
|
|
await writeFile((state.gate as { gate: string }).gate, "The human brief is ready: save an initial website-plan task document, retaining accessible font guidance, then finish this turn normally.", "utf8");
|
|
type Delivery = { runs: Array<{ status: string }>; task: Entity; queue: IssueQueuedCommentQueue };
|
|
const delivered = (v: Delivery) => v.runs.every(row => row.status === "succeeded") && v.task.status === "done" && v.queue.entries.length === 0;
|
|
const settledInTurn = settledQueue((v: Delivery) => delivered(v) && v.runs.length === 2, 30_000);
|
|
const consumed = await pollUntil({ label: "Real later contact request consumed before cancellation", deadlineAt: Date.now() + 240_000,
|
|
load: async () => ({ runs: await ctx.allRuns(), task: await ctx.input.api.get<Entity>(root), queue: await ctx.input.api.get<IssueQueuedCommentQueue>(queuePath) }),
|
|
accept: v => settledInTurn(v) || delivered(v) && v.runs.length === 3,
|
|
reject: v => v.runs.length > 3 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "The saved follow-ups failed or reached more than one continuation" : undefined });
|
|
const taskComments = await ctx.input.api.get<Entity[]>(`${root}/comments?order=asc`);
|
|
const followUps = secondQueue.entries.map(row => taskComments.find(comment => comment.id === row.comment.id)).filter((row): row is Entity => !!row)
|
|
.map(row => ({ id: String(row.id), updatedAt: row.updatedAt, body: String(row.body ?? "") }));
|
|
const receipts = await runCommentReceipts(ctx, String(state.workerRunId)); state.queuedFollowUps = followUps; state.workerCommentReceipts = receipts;
|
|
const successors = consumed.runs.filter(row => row.agentId === (state.worker as Entity).id && row.id !== state.workerRunId);
|
|
state.firstDeliveryRunCount = consumed.runs.length; state.continuationConsumerCount = successors.length;
|
|
const detail = successors[0] && await ctx.input.api.get<{ contextSnapshot?: { wakeCommentIds?: string[]; paperclipWake?: { comments?: Array<{ id?: string; readByEarlierTurn?: boolean }> } }; runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${successors[0].id}`);
|
|
const prompt = detail?.runnerProfileJson?.nativeExecutionInput?.task?.prompt;
|
|
state.continuationInputBound = !!detail && continuationCarriesOwed(followUps, receipts, { contextSnapshot: detail.contextSnapshot, prompt });
|
|
state.consumerRunId = handledInTurn(receipts, followUps.find(row => row.id === second.comment.id)) ? state.workerRunId : successors[0]?.id;
|
|
state.consumedBeforeCancellation = !!detail && detail.contextSnapshot?.wakeCommentIds?.includes(second.comment.id) === true && promptCarries(prompt, secondText);
|
|
state.discardedNeverDelivered = consumed.runs.every(row => !((row.contextSnapshot?.wakeCommentIds as string[] | undefined) ?? []).includes(targetId));
|
|
state.lateDiscardStatus = await probeConsumedHumanQueue(ctx, String(state.humanId), `${queuePath}/${second.comment.id}`, "DELETE", { queueId: secondQueue.queueId, revision: secondQueue.revision });
|
|
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
|
|
const message = `I later queued a different contact request on Website ${task.identifier ?? task.id}, comment ${second.comment.id}, and Morgan has already consumed it. Cancel that later contact requirement now by sending Morgan new direction to remove the contact section from the saved website plan while retaining the accessible-font requirement. Explain what happened to the already-delivered input; do not erase or rewrite it.`;
|
|
const priorIds = consumed.runs.map(row => row.id);
|
|
await sendChatMessage(ctx.input.page, message, chatCatalogNativePhaseRemainingBudget(ctx));
|
|
const final = await pollUntil({ label: "One real new-direction continuation after already-delivered cancellation", deadlineAt: Date.now() + 300_000,
|
|
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(), task: await ctx.input.api.get<Entity>(root), queue: await ctx.input.api.get<IssueQueuedCommentQueue>(queuePath) }; },
|
|
accept: v => v.source?.conversationState === "waiting" && v.runs.length === priorIds.length + 2 && v.runs.every(row => row.status === "succeeded") && v.task.status === "done" && v.queue.entries.length === 0,
|
|
reject: v => v.runs.length > priorIds.length + 2 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "The declared later cancellation failed or duplicated execution" : undefined });
|
|
const later = final.runs.filter(row => !priorIds.includes(row.id)), lead = later.find(row => row.agentId === ctx.input.fixtures.agent.id), worker = later.find(row => row.agentId === (state.worker as Entity).id);
|
|
const sourceComments = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`), human = sourceComments.filter(row => !row.authorAgentId && row.body === message);
|
|
const replies = sourceComments.filter(row => row.createdByRunId === lead?.id && row.authorAgentId === ctx.input.fixtures.agent.id);
|
|
const leadInput = lead && await ctx.input.api.get<{ runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${lead.id}`);
|
|
state.latePhaseAttributed = human.length === 1 && !!lead && replies.length > 0 && promptCarries(leadInput?.runnerProfileJson?.nativeExecutionInput?.task?.prompt, human[0]!.id);
|
|
state.lateReply = replies.map(row => row.body).join("\n");
|
|
const comments = await ctx.input.api.get<IssueComment[]>(`${root}/comments?order=asc`), direction = comments.filter(row => row.authorAgentId === ctx.input.fixtures.agent.id && row.createdByRunId === lead?.id);
|
|
const workerInput = worker && await ctx.input.api.get<{ contextSnapshot?: { wakeCommentIds?: string[]; wakeCommentId?: string }; runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${worker.id}`);
|
|
state.cancelDirectionRunBound = later.length === 2 && !!worker && direction.length === 1
|
|
&& [...(workerInput?.contextSnapshot?.wakeCommentIds ?? []), workerInput?.contextSnapshot?.wakeCommentId].includes(direction[0]!.id)
|
|
&& promptCarries(workerInput?.runnerProfileJson?.nativeExecutionInput?.task?.prompt, direction[0]!.body);
|
|
state.finalWorkerRunId = worker?.id; state.consumedStillReadable = comments.some(row => row.id === second.comment.id && row.body === secondText && !row.deletedAt);
|
|
state.finalQueueEmpty = final.queue.entries.length === 0;
|
|
},
|
|
async observe(ctx, state) {
|
|
const o = await observeRuntime(ctx, state);
|
|
state.initialRevisionRetained = o.revisions.some(row => row.createdByRunId === state.workerRunId && row.createdByAgentId === (state.worker as Entity).id);
|
|
// The consumed request asked for a contact section with the setup's contact name and phone. Its consuming run
|
|
// must have saved the name, and the cancellation run's latest plan must drop that section's distinctive lines.
|
|
const contact = contactDetails(ctx.input.nonce), worker = (state.worker as Entity).id;
|
|
const consumed = o.revisions.filter(row => row.createdByRunId === state.consumerRunId && row.createdByAgentId === worker && (row.body ?? "").includes(contact.name));
|
|
state.consumedLabelSaved = consumed.length > 0;
|
|
const latest = o.revisions.filter(row => row.createdByRunId === state.finalWorkerRunId && row.createdByAgentId === worker
|
|
&& o.documents.some(doc => doc.id === row.documentId && doc.latestRevisionId === row.id));
|
|
state.finalPlanWithoutContact = latest.length > 0 && consumed.every(row => {
|
|
const doc = o.documents.find(d => d.id === row.documentId);
|
|
return documentContactRemoved(row.body ?? "", o.revisions.find(r => r.id === doc?.latestRevisionId)?.body ?? undefined, ctx.input.nonce, [contact.name, contact.phone]);
|
|
});
|
|
return o;
|
|
},
|
|
async cleanup(ctx, state) { try { await cleanupIdentityProbes(ctx); } finally { await releaseRuntimeWorker(state); } },
|
|
})];
|