Files
PaperClipAI/tests/runner-e2e/chat-catalog-run-policy.test.ts
T
DottaandClaude Opus 5.5 74096c19f4 fix(evals): admit the in-flight chat completion follow-up on a settled source (#21313)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat request reaches the correct
saved outcome
> - The `source-settled-with-active-work` run policy checks that the CEO
chat is settled while delegated work can continue
> - After a delegated task finishes, the product starts its normal
`chat_task_completed` follow-up on the source chat
> - The policy required every source run to be `succeeded` and the
source to be `waiting`, so a re-grade during that follow-up failed
> - This pull request admits that one product follow-up while it is in
flight, and keeps every other source run strict
> - The benefit is that the eval grades the saved private-delegation
outcome, not the timing of the reload

## Linked Issues or Issue Description

**What happened?**
The OpenCode cell of
`chat-features.runner-opencode.local.tasks.private-task` failed with
"Independent oracle survives browser reload: expected true, received
false". The saved state was correct. The CEO answered, Morgan saved the
private draft in the private project, and the unrelated member could not
read it. The grade passed for its full 5-second stability window.
Morgan's run then finished, and 73 ms later the product started the CEO
`chat_task_completed` follow-up on the source chat. The harness reloaded
the page and re-graded while that follow-up was running. The only failed
check was `catalog-settled-source-separate-work`, because it required
all source runs to be `succeeded`.

**Expected behavior**
The case already counts this follow-up in its run bound
(`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running
product follow-up on an answered source must not fail the case.

**Steps to reproduce**
Run the private-task case. Let the worker finish within about 5 seconds
of the first passing grade. The post-reload re-grade sees the follow-up
as `running` and fails.

## What Changed

- Add `chatCatalogSourceCompletionFollowUp` to
`tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's
`chat_task_completed` run on the source whose every completion update
names separate work in the run ledger.
- In the `source-settled-with-active-work` check, admit at most one such
follow-up while it is `queued`, `running` or `succeeded`. Accept a
source `conversationState` of `active` only while that follow-up is in
flight.
- Keep the other rules strict. The original answer must exist and must
have succeeded. Any other active, failed or extra source run still
fails. The declared run-count bound still applies.
- Also admit the follow-up before its turn is prepared. The server
claims the run first and saves the completion updates after that, so a
queued run, or a running run in preparation, has only its delivery IDs.
It qualifies only while queued or running, when automation woke it, and
after a separate worker run in the ledger succeeded.
- Add a red/green test that reproduces the observed race and covers the
negative cases.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails
before the change (1 failed, 13 passed) and passes after it (14 passed).
The queued-snapshot cases were added after review.
- The related runner-e2e tests (`chat-catalog-run-policy`,
`chat-catalog-readiness`, `chat-catalog-failure-retirement`,
`runtime-dependency`, `internal`) pass: 200 tests.
- `pnpm test:e2e:runner:typecheck` passes.
- Paid run
[43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951)
on the first commit of this branch:
`chat-features.runner-opencode.local.tasks.private-task` passed.

## Risks

- Low risk. The change is in eval grading only. It does not change
product code.
- The check accepts a follow-up that is still running at grade time. A
later failure of that follow-up is not observed by this case. Completion
delivery is covered by its own completion cases.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:04:35 -06:00

221 lines
22 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { gradeChatCatalogRuns, chatCatalogAllowedRunStatuses, assertExpectedWorkerCancellation, chatCatalogRunStatusAllowed, createChatCatalogFinalRunVerifier, chatCatalogNeverStartedTerminalTaskRun, chatCatalogReassignedQueuedRun, chatCatalogSupersededQueuedRuns, chatCatalogProviderRuns } from "./chat-catalog-run-policy.js";
import type { ChatRun } from "./chat-flow.js";
const succeeded: ChatRun = { id: "source-run", companyId: "company", agentId: "ceo", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "source" } };
const worker: ChatRun = { id: "worker-run", companyId: "company", agentId: "worker", status: "running", runtimeMode: "native", contextSnapshot: { issueId: "task" } };
const scope = { companyId: "company", leadId: "ceo", source: { id: "source", companyId: "company", title: "Chat", status: "in_review", conversationState: "waiting" } };
it("requires successful native work by default and exact declared cancellation counts", () => {
expect(gradeChatCatalogRuns({ ...scope, implementation: { expectedRunCount: 1 }, runs: [succeeded] }).passed).toBe(true);
const cancelled = { ...succeeded, status: "cancelled" };
expect(gradeChatCatalogRuns({ ...scope, implementation: { expectedRunCount: 1 }, runs: [cancelled] }).passed).toBe(false);
const implementation = { expectedRunCount: 1, runPolicy: { kind: "terminal-counts" as const, terminalStatusCounts: { cancelled: 1 } } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded] }).passed).toBe(false);
});
it("admits declared separate active work while rejecting active CEO, wrong scope, unbound and excess work", () => {
const implementation = { expectedRunCount: 2, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, worker] }).passed).toBe(true);
for (const bad of [{ ...succeeded, status: "running" }, { ...worker, companyId: "other" }, { ...worker, contextSnapshot: {} }, { ...worker, agentId: "ceo" }])
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, bad] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, worker, { ...worker, id: "extra" }] }).passed).toBe(false);
});
it("admits a lead run outside the source only as a declared case handback", () => {
const handback: ChatRun = { id: "handback", companyId: "company", agentId: "ceo", status: "running", runtimeMode: "native", contextSnapshot: { issueId: "task", wakeReason: "case_handoff" } };
const done = { ...worker, status: "succeeded" };
const declared = { expectedRunCount: 3, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1, maximumLeadCaseHandbackRuns: 1 } };
expect(gradeChatCatalogRuns({ ...scope, implementation: declared, runs: [succeeded, done, handback] }).passed).toBe(true);
// Undeclared (default 0), a lead run woken another way, and a second handback all stay rejected.
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...declared, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } }, runs: [succeeded, done, handback] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: declared, runs: [succeeded, done, { ...handback, contextSnapshot: { issueId: "task", wakeReason: "issue_commented" } }] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...declared, expectedRunCount: 4 }, runs: [succeeded, done, handback, { ...handback, id: "again" }] }).passed).toBe(false);
});
it("admits at most one in-flight product completion follow-up on a settled source, and nothing else active there", () => {
const implementation = { expectedRunCount: 3, minimumRunCount: 2, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
const done = { ...worker, status: "succeeded" };
const followUp: ChatRun = { id: "follow-up", companyId: "company", agentId: "ceo", status: "running", runtimeMode: "native",
contextSnapshot: { issueId: "source", wakeReason: "chat_task_completed", chatCompletionUpdates: [{ id: "task", status: "done" }] } };
const active = { ...scope, source: { ...scope.source, conversationState: "active" as const } };
// The private-task race: Morgan finished, and the product's follow-up started on the answered source.
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, followUp] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, { ...followUp, status: "queued" }] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, done, { ...followUp, status: "succeeded" }] }).passed).toBe(true);
for (const bad of [
{ ...followUp, status: "failed" },
{ ...followUp, contextSnapshot: { ...followUp.contextSnapshot, chatCompletionUpdates: [{ id: "unrelated" }] } },
{ ...followUp, contextSnapshot: { ...followUp.contextSnapshot, chatCompletionUpdates: [] } },
{ ...followUp, contextSnapshot: { ...followUp.contextSnapshot, wakeReason: "issue_commented" } },
{ ...followUp, agentId: "worker" },
]) expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, bad] }).passed).toBe(false);
// Before the turn is prepared, the queued run carries only its saved delivery IDs (actual server snapshot shape).
const pending: ChatRun = { ...followUp, status: "queued",
contextSnapshot: { issueId: "source", wakeReason: "chat_task_completed", wakeSource: "automation", chatCompletionDeliveryIds: ["delivery"] } };
// The server claims the run (running) before it saves the updates, so a running run in preparation also qualifies.
for (const status of ["queued", "running"]) expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, { ...pending, status }] }).passed).toBe(true);
for (const bad of [{ ...pending, status: "succeeded" }, { ...pending, status: "failed" }, { ...pending, contextSnapshot: { ...pending.contextSnapshot, chatCompletionDeliveryIds: [] } },
{ ...pending, contextSnapshot: { ...pending.contextSnapshot, wakeSource: "manual" } }])
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, bad] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, worker, pending] }).passed).toBe(false);
// The original answer must exist and have succeeded; a second follow-up and an unexplained active source stay rejected.
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [done, followUp] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [{ ...succeeded, status: "running" }, done, followUp] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...active, implementation: { ...implementation, expectedRunCount: 4 }, runs: [succeeded, done, followUp, { ...followUp, id: "again" }] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...active, implementation, runs: [succeeded, done, { ...followUp, status: "succeeded" }] }).passed).toBe(false);
});
function reassignment() {
const receipt = { runId: "worker-run", taskId: "task", workerId: "worker", sourceRunId: "final-source" };
const stopped: ChatRun = { ...worker, status: "cancelled", errorCode: "issue_reassigned", resultJson: {
reassignmentStopConfirmed: true, nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId: "worker-run", issueId: "task", companyId: "company", dispatchState: "acknowledged", acknowledgedAt: "2026-10-27T10:00:00Z", intentAuditId: "intent", acknowledgementAuditId: "ack" },
} };
const implementation = { scenarioId: "tasks.edit-properties", expectedRunCount: 4, minimumRunCount: 3,
runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1, expectedWorkerCancellation: "owned-task-reassignment" as const },
expectedCancelledWorkerRun: (observation: unknown) => (observation as { proofComplete?: boolean } | undefined)?.proofComplete ? receipt : undefined };
const observation = { proofComplete: true, state: { heldRun: { ...worker }, heldTask: { id: "task", companyId: "company", assigneeAgentId: "worker", executionRunId: "worker-run" }, riley: { id: "worker" } } };
return { implementation, stopped, receipt, observation, runs: [succeeded, { ...succeeded, id: "final-source" }, stopped] };
}
it("defers only the declared worker cancellation during polling and requires final independent proof", () => {
const v = reassignment();
expect(chatCatalogAllowedRunStatuses(v.implementation)).toContain("cancelled");
expect(chatCatalogRunStatusAllowed({ ...scope, implementation: v.implementation, run: v.stopped, observation: { ...v.observation, proofComplete: false } })).toBe(true);
for (const run of [{ ...v.stopped, id: "other" }, { ...v.stopped, agentId: "ceo" }, { ...v.stopped, companyId: "foreign" }, { ...v.stopped, errorCode: "user_cancelled" }])
expect(chatCatalogRunStatusAllowed({ ...scope, implementation: v.implementation, run, observation: v.observation })).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: v.implementation, runs: v.runs }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: v.implementation, runs: v.runs, observation: v.observation }).passed).toBe(true);
// No cancellation remains a valid genuine ownership-denied branch; no proof is invented.
expect(gradeChatCatalogRuns({ ...scope, implementation: v.implementation, runs: [succeeded, { ...succeeded, id: "final-source" }, worker] }).passed).toBe(true);
});
it("rejects wrong cancellation identities, scope, reason, acknowledgement and unrelated cancelled work", () => {
const mutations = [
(v: ReturnType<typeof reassignment>) => { v.receipt.runId = "unrelated"; },
(v: ReturnType<typeof reassignment>) => { v.receipt.taskId = "other-task"; },
(v: ReturnType<typeof reassignment>) => { v.receipt.workerId = "other-worker"; },
(v: ReturnType<typeof reassignment>) => { v.receipt.sourceRunId = "unrelated-source"; },
(v: ReturnType<typeof reassignment>) => { v.stopped.companyId = "foreign"; },
(v: ReturnType<typeof reassignment>) => { v.stopped.agentId = "ceo"; },
(v: ReturnType<typeof reassignment>) => { v.stopped.errorCode = "user_cancelled"; },
(v: ReturnType<typeof reassignment>) => { v.stopped.resultJson!.reassignmentStopConfirmed = false; },
(v: ReturnType<typeof reassignment>) => { (v.stopped.resultJson!.nativeCancellation as Record<string, unknown>).dispatchState = "pending"; },
(v: ReturnType<typeof reassignment>) => { (v.stopped.resultJson!.nativeCancellation as Record<string, unknown>).issueId = "other-task"; },
(v: ReturnType<typeof reassignment>) => { v.runs.push({ ...v.stopped, id: "another-cancelled-run" }); },
];
for (const mutate of mutations) { const v = reassignment(); mutate(v); expect(gradeChatCatalogRuns({ ...scope, implementation: v.implementation, runs: v.runs, observation: v.observation }).passed).toBe(false); }
});
it("requires exact scenario/flag/callback and leaves every undeclared active-work case unchanged", () => {
const v = reassignment();
expect(() => assertExpectedWorkerCancellation(v.implementation)).not.toThrow();
for (const implementation of [
{ ...v.implementation, scenarioId: "tasks.assignment-conflict" },
{ ...v.implementation, expectedCancelledWorkerRun: undefined },
{ ...v.implementation, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } },
]) { expect(() => assertExpectedWorkerCancellation(implementation)).toThrow(); expect(gradeChatCatalogRuns({ ...scope, implementation, runs: v.runs, observation: v.observation }).passed).toBe(false); }
expect(chatCatalogAllowedRunStatuses({ runPolicy: { kind: "source-settled-with-active-work", maximumActiveWorkerRuns: 1 } })).not.toContain("cancelled");
});
it("binds cancellation proof presence through the actual lazy registry loader", async () => {
const { loadChatCatalogImplementation } = await import("./chat-catalog-registry.js");
const v = reassignment(); const { expectedCancelledWorkerRun, ...metadata } = v.implementation;
const descriptor = { ...metadata, modulePath: "./chat-catalog-domains/internal-tasks.js", exportName: "cases", sourceFiles: ["chat-catalog-domains/internal-edit-ownership.ts"], oracleVersion: "test", turnTimeoutMs: 120_000, attemptTimeoutMs: 600_000 };
const implementation = { ...descriptor, expectedCancelledWorkerRun, setup: async () => ({}), observe: async () => ({}), grade: () => ({ passed: true, checks: [{ id: "actual-proof", passed: true, detail: "Fixture-only registry control" }] }), verifyResult: async () => {} };
await expect(loadChatCatalogImplementation(descriptor, async () => ({ cases: [implementation] }))).resolves.toBe(implementation);
await expect(loadChatCatalogImplementation(descriptor, async () => ({ cases: [{ ...implementation, expectedCancelledWorkerRun: undefined }] }))).rejects.toThrow("cancellation proof");
const wrongDescriptor = { ...descriptor, scenarioId: "tasks.assignment-conflict" };
await expect(loadChatCatalogImplementation(wrongDescriptor, async () => ({ cases: [{ ...implementation, scenarioId: wrongDescriptor.scenarioId }] }))).rejects.toThrow("cancellation proof");
});
it("retains the executable cancellation proof at the outer final-run boundary", () => {
const v = reassignment();
const verify = createChatCatalogFinalRunVerifier({ implementation: v.implementation, companyId: scope.companyId,
leadId: scope.leadId, observation: v.observation });
expect(verify(scope.source, v.runs).passed).toBe(true);
// JSON declarations cannot replace the executable independent proof callback.
expect(() => createChatCatalogFinalRunVerifier({ implementation: JSON.parse(JSON.stringify(v.implementation)),
companyId: scope.companyId, leadId: scope.leadId, observation: v.observation })).toThrow("cancellation proof");
for (const run of [
{ ...v.stopped, errorCode: "user_cancelled" },
{ ...v.stopped, companyId: "foreign" },
{ ...v.stopped, resultJson: { ...v.stopped.resultJson, nativeCancellation: { ...(v.stopped.resultJson!.nativeCancellation as object), dispatchState: "pending" } } },
]) expect(verify(scope.source, v.runs.map(row => row.id === run.id ? run : row)).passed).toBe(false);
expect(verify(scope.source, [...v.runs, { ...v.stopped, id: "new-cancellation" }]).passed).toBe(false);
expect(verify({ ...scope.source, id: "different-source" }, v.runs).passed).toBe(false);
});
it("keeps an ordinary ownership-denial boundary strict without a cancellation declaration", () => {
const implementation = { expectedRunCount: 2,
runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
const verify = createChatCatalogFinalRunVerifier({ implementation, companyId: scope.companyId,
leadId: scope.leadId, observation: undefined });
expect(verify(scope.source, [succeeded, worker]).passed).toBe(true);
expect(verify(scope.source, [succeeded, { ...worker, status: "cancelled" }]).passed).toBe(false);
});
it("treats only a never-started run cancelled by its own task's terminal status as lifecycle, and still grades it", () => {
// Shape recorded for tasks.website-task: the agent cancelled its own stray task before the queued run started.
const queued: ChatRun = { id: "stray-run", companyId: "company", agentId: "ceo", status: "cancelled", runtimeMode: "legacy", startedAt: null,
errorCode: "issue_terminal_status", error: "Cancelled because issue reached terminal status (cancelled) before the queued run could start",
resultJson: { stopReason: "issue_terminal_status", timeoutFired: false }, contextSnapshot: { issueId: "stray-task", wakeReason: "issue_assigned" } };
expect(chatCatalogNeverStartedTerminalTaskRun(queued, scope.source)).toBe(true);
for (const run of [{ ...queued, startedAt: "2026-11-09T15:27:00Z" }, { ...queued, errorCode: "user_cancelled" }, { ...queued, status: "failed" },
{ ...queued, resultJson: {} }, { ...queued, contextSnapshot: { issueId: "source" } }, { ...queued, contextSnapshot: {} }])
expect(chatCatalogNeverStartedTerminalTaskRun(run, scope.source)).toBe(false);
const implementation = { expectedRunCount: 3, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, worker, queued] }).passed).toBe(false);
});
it("admits a declared role-bound range above an exact terminal count, and nothing beyond it", () => {
const cancelled = { ...succeeded, id: "stopped", status: "cancelled" };
const ok = (id: string) => ({ ...succeeded, id });
const implementation = { expectedRunCount: 6, minimumRunCount: 4, runPolicy: { kind: "terminal-counts" as const, terminalStatusCounts: { cancelled: 1, succeeded: 3 }, maximumTerminalStatusCounts: { succeeded: 5 } } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled, ok("a"), ok("b"), ok("c")] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled, ok("a"), ok("b"), ok("c"), ok("d"), ok("e")] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled, ok("a"), ok("b")] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...implementation, expectedRunCount: 7 }, runs: [cancelled, ok("a"), ok("b"), ok("c"), ok("d"), ok("e"), ok("f")] }).passed).toBe(false);
// A maximum may only widen a status the exact counts already declare.
const undeclared = { ...implementation, runPolicy: { ...implementation.runPolicy, maximumTerminalStatusCounts: { failed: 1 } } };
expect(gradeChatCatalogRuns({ ...scope, implementation: undeclared, runs: [cancelled, ok("a"), ok("b"), ok("c")] }).passed).toBe(false);
});
describe("queued run retired by reassignment before it started (full cohort 3)", () => {
// projects.project-work: the CEO created the task without an assignee (its own queued run), then
// reassigned it to Morgan; the reassignment stopped the run while still bootstrapping.
const reassigned: ChatRun = { id: "ceo-queued", companyId: "company", agentId: "ceo", status: "cancelled", runtimeMode: "legacy",
startedAt: null, logStore: null, logRef: null, nativeIssueId: null, errorCode: "issue_reassigned", createdAt: "2026-11-12T00:55:19.299Z",
contextSnapshot: { issueId: "task", wakeReason: "issue_assigned" },
resultJson: { stopReason: "cancelled", reassignmentStopConfirmed: true, executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
// tasks.private-task: the dispatch gate found the same kind of queued wake stale after the PATCH.
const staleGate: ChatRun = { ...reassigned, id: "ceo-stale", errorCode: "issue_assignee_changed",
resultJson: { stopReason: "issue_assignee_changed", timeoutFired: false, timeoutSource: "stale_queued_run_gate" } };
// Paid run 43143392360 (tasks.private-task): the new assignee's wake released the lock the queued run held.
const lockReleased: ChatRun = { ...reassigned, id: "ceo-lock", errorCode: "lock_released_on_reassignment", resultJson: undefined };
const morgan: ChatRun = { ...worker, id: "morgan-run", createdAt: "2026-11-12T00:55:56.398Z" };
const implementation = { expectedRunCount: 2, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
it("admits it, outside the provider-run bound, only when the new assignee ran on that task", () => {
for (const queued of [reassigned, staleGate, lockReleased]) {
expect(chatCatalogReassignedQueuedRun(queued, scope.source)).toBe(true);
const runs = [succeeded, queued, morgan];
expect(chatCatalogSupersededQueuedRuns(runs, scope.source).map(run => run.id)).toEqual([queued.id]);
expect(chatCatalogProviderRuns(runs, scope.source).map(run => run.id)).toEqual([succeeded.id, morgan.id]);
const grade = gradeChatCatalogRuns({ ...scope, implementation, runs });
expect(grade.passed).toBe(true);
expect(grade.checks.find(check => check.id === "catalog-superseded-queued-runs")?.passed).toBe(true);
}
});
it("still fails without a successor, for started or other cancellations, on the source, and for churn", () => {
// No new assignee run: the reassignment never produced work.
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, reassigned] }).passed).toBe(false);
// A successor must be another agent's later run on the same task.
for (const other of [{ ...morgan, agentId: "ceo" }, { ...morgan, contextSnapshot: { issueId: "elsewhere" } }, { ...morgan, createdAt: "2026-11-12T00:55:00.000Z" }])
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...implementation, expectedRunCount: 3 }, runs: [succeeded, reassigned, other] }).passed).toBe(false);
for (const run of [{ ...reassigned, startedAt: "2026-11-12T00:55:20.000Z" }, { ...reassigned, logRef: "log" }, { ...reassigned, errorCode: "cancelled" },
{ ...reassigned, resultJson: { ...reassigned.resultJson, executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } },
{ ...reassigned, resultJson: { stopReason: "cancelled" } }, { ...staleGate, resultJson: { stopReason: "issue_assignee_changed" } },
{ ...reassigned, contextSnapshot: { issueId: "source" } }, { ...reassigned, status: "failed" },
{ ...lockReleased, startedAt: "2026-11-12T00:55:20.000Z" }, { ...lockReleased, nativeIssueId: "task" }]) {
expect(chatCatalogReassignedQueuedRun(run, scope.source)).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...implementation, expectedRunCount: 3 }, runs: [succeeded, run, morgan] }).passed).toBe(false);
}
// At most one superseded queued run per case.
const second = { ...staleGate, contextSnapshot: { issueId: "task-2" } }, morgan2 = { ...morgan, id: "morgan-2", status: "succeeded", contextSnapshot: { issueId: "task-2" } };
const churn = gradeChatCatalogRuns({ ...scope, implementation: { ...implementation, expectedRunCount: 3 }, runs: [succeeded, reassigned, morgan, second, morgan2] });
expect(churn.checks.find(check => check.id === "catalog-superseded-queued-runs")?.passed).toBe(false);
});
});