mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-08 00:54:38 +02:00
## Thinking Path > - Paperclip manages AI agents and records their durable work outcomes. > - Product E2E checks those outcomes through the browser and public API. > - A successful long run can emit more than 1,000 durable events. > - The harness read one page and missed the later completion evidence. > - This pull request reads every page before it checks runtime invariants. > - Invalid or incomplete capture still fails. A missing page cannot produce a pass. ## Linked Issues or Issue Description Related: #13882 (Grok qualification). A duplicate search found no existing event-pagination fix. **What happened?** The local structured-question case in [campaign 36071063537](https://github.com/paperclipai/paperclip/actions/runs/36071063537) completed the task and passed its six outcome matchers. It failed native runtime invariants because the capture contained exactly 1,000 events. The last captured event preceded the run's completion by more than a minute. The API caps each response at 1,000 rows. The harness did not request the next page. **Expected behavior** Read the complete durable event stream through the public API before checking semantic-result and terminal-event counts. Reject incomplete or malformed evidence. **Steps to reproduce** 1. Complete a native task that emits more than 1,000 durable events. 2. Place the semantic-result and terminal events after row 1,000. 3. Capture the run with the Product E2E harness. 4. Before this fix, the invariant checker sees only the first page. **Paperclip version or commit** Observed at `4196a4cd76db434854b679035e4146c7f69689ce`. The same single-page capture exists on master. The original failed result remains unchanged; missing historical tail evidence is not reconstructed or graded as a pass. ## What Changed - Add a bounded event collector that advances through the public `afterSeq` cursor. - Use it for task success/failure evidence and shared chat run evidence. - Reject invalid pages, missing or non-increasing sequence numbers, repeated cursors, failed later requests, and an exhausted page limit. - Test completion events beyond the first page, exact page boundaries, and malformed evidence. - Correct the existing Everyday catalog test from 38 to the maintained 47 cells. The suite stays explicit-only. - Document the complete-capture requirement and its bound. ## Verification - Product harness typecheck passes. - The full credential-free harness suite passed 477 tests. After adding the chat integration regression, all 34 chat evidence tests pass. - Seventeen pagination tests cover the valid tail and malformed-evidence cases. - `git diff --check` passes. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/36080683422) passes at `aed6f79c089226be79e75dcf390969561ec4f787`: 52 successful checks and two intentional skips, including Rust, typecheck, build, server tests, and browser shards. Greptile gives this exact head 5/5 with no findings. No local Docker, Rust build, browser suite, or model invocation was used for this change. - [Live Grok requalification](https://github.com/paperclipai/paperclip/actions/runs/36080870743) is running on combined source `1b0551bb7c8de3c54f4bee64dbe2c88328b3645e`, with the runner, scheduler, and evidence fixes. Its full credential-free harness passes all 493 tests and typecheck. Live results are pending; the original campaign remains a failed measurement. ## Risks Long runs need more read-only API requests and larger private evidence files. Capture stops with an explicit error after 100 full pages. This changes neither production APIs nor provider behavior. It does not relax an invariant or change a historical grade. ## Model Used OpenAI GPT-6 through Codex, with repository tools and code execution. The exact serving identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
810 lines
26 KiB
TypeScript
810 lines
26 KiB
TypeScript
import { describe, it, expect } from "vitest";
|
|
import { runnerMatrix, runnerSuites } from "./catalog.js";
|
|
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
|
import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
|
|
import {
|
|
isStoryWorkspaceDeferral,
|
|
storyUnexpectedRunFailure,
|
|
storyUnexercisedReviewBoundary,
|
|
storyLifecycleChecks,
|
|
storyRepliesConsumed,
|
|
storyHasAgentReply,
|
|
storyHasPendingAgentReview,
|
|
storyHasPendingAgentWake,
|
|
storyHasPendingAgentWork,
|
|
storyHasPendingHumanInteraction,
|
|
storyHasStrandedBlockedLeaf,
|
|
storyHasDurableAgentReviewContinuation,
|
|
storyIssueHasBlockedTimeline,
|
|
storyIssueHasBlockedTimelineBefore,
|
|
storyIssueHasUnresolvedDependency,
|
|
storyRunReportsDependencyBlock,
|
|
storyParentFinishedAfterChildren,
|
|
artifactGradeModeForPhase,
|
|
storyParentCompletionPrecedesReview,
|
|
storyAcceptedAgentReview,
|
|
type StoryRun,
|
|
type StoryIssue,
|
|
} from "./everyday-observations.js";
|
|
import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interruption.js";
|
|
|
|
describe("everyday workflow grader and review timing", () => {
|
|
it("requires nonempty persisted source evidence", () => {
|
|
expect(hasPersistedSource(undefined)).toBe(false);
|
|
expect(hasPersistedSource(Buffer.alloc(0))).toBe(false);
|
|
expect(hasPersistedSource(Buffer.from("source"))).toBe(true);
|
|
});
|
|
|
|
it("requires an active run and saved source at the same Stop checkpoint", () => {
|
|
expect(isSavedSourceCheckpoint(false, Buffer.from("source"))).toBe(false);
|
|
expect(isSavedSourceCheckpoint(true, Buffer.alloc(0))).toBe(false);
|
|
expect(isSavedSourceCheckpoint(true, Buffer.from("source"))).toBe(true);
|
|
});
|
|
it("uses base grading for review delivery and preserves late max-length cases", () => {
|
|
expect(artifactGradeModeForPhase("reviewed-delivery")).toBe("base");
|
|
expect(artifactGradeModeForPhase("delegated-delivery")).toBe("max-length");
|
|
expect(artifactGradeModeForPhase("recovered-delivery")).toBe("max-length");
|
|
});
|
|
|
|
it("compares parent completion with review start rather than card creation", () => {
|
|
const interactionCreatedAt = "2026-09-17T10:00:00.000Z";
|
|
const parentFinishedAt = "2026-09-17T10:05:00.000Z";
|
|
const reviewStartedAt = "2026-09-17T10:06:00.000Z";
|
|
expect(interactionCreatedAt < parentFinishedAt).toBe(true);
|
|
expect(
|
|
storyParentCompletionPrecedesReview(
|
|
parentFinishedAt,
|
|
reviewStartedAt,
|
|
false,
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyParentCompletionPrecedesReview(
|
|
"2026-09-17T10:07:00.000Z",
|
|
reviewStartedAt,
|
|
false,
|
|
),
|
|
).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("unexercised review boundary diagnostics", () => {
|
|
const done = { id: "parent", companyId: "company", title: "task", status: "done" };
|
|
const completed = {
|
|
id: "run", companyId: "company", agentId: "lead", status: "succeeded",
|
|
nativeIssueId: "parent", finishedAt: "2026-09-19T14:47:48.134Z",
|
|
};
|
|
const child = {
|
|
...done, id: "child", parentId: "parent", interactions: [{
|
|
id: "card", issueId: "child", kind: "request_confirmation", status: "accepted",
|
|
addresseeAgentId: "lead", resolvedByAgentId: "lead", resolvedByRunId: "review",
|
|
createdAt: "2026-09-19T14:47:04.000Z", resolvedAt: "2026-09-19T14:47:21.876Z",
|
|
payload: { target: { type: "custom", key: "native_completion_review", revisionId: "decision" } },
|
|
result: { version: 1, outcome: "accepted" },
|
|
}],
|
|
};
|
|
const review = {
|
|
...completed, id: "review", nativeIssueId: "child", startedAt: "2026-09-19T14:47:05.802Z",
|
|
contextSnapshot: { nativeReviewInteractionId: "card", nativeReviewDecisionId: "decision" },
|
|
};
|
|
const diagnose = (issues: StoryIssue[] = [done, child], runs: StoryRun[] = [completed, review]) =>
|
|
storyUnexercisedReviewBoundary(issues, runs, "parent", "lead");
|
|
it("fails promptly with persisted proof that review preceded parent completion", () => {
|
|
expect(diagnose()).toContain("not exercised");
|
|
});
|
|
it("does not preempt in-flight finalization, recovery, or unfinished work", () => {
|
|
expect(diagnose([done, child], [{ ...completed, status: "running" }, review])).toBeUndefined();
|
|
expect(storyUnexercisedReviewBoundary([{ ...done, scheduledRetry: {} }, child], [completed, review], "parent", "lead")).toBeUndefined();
|
|
expect(storyUnexercisedReviewBoundary([{ ...done, activeRecoveryAction: {} }, child], [completed, review], "parent", "lead")).toBeUndefined();
|
|
expect(diagnose([{ ...done, status: "blocked" }, child])).toBeUndefined();
|
|
expect(diagnose([], [])).toBeUndefined();
|
|
});
|
|
it("waits when separately fetched snapshots lack review evidence or timestamps", () => {
|
|
expect(diagnose([done, { ...child, interactions: [] }])).toBeUndefined();
|
|
expect(diagnose([done, child], [completed])).toBeUndefined();
|
|
expect(diagnose([done, child], [{ ...completed, finishedAt: "" }, review])).toBeUndefined();
|
|
expect(diagnose([done, child], [completed, { ...review, startedAt: "" }])).toBeUndefined();
|
|
expect(diagnose([done, child], [review])).toBeUndefined();
|
|
});
|
|
it("does not reject a completed workflow with the required timing", () => {
|
|
expect(diagnose([done, child], [{ ...completed, finishedAt: review.startedAt }, review])).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
describe("multi-round agent review handoff", () => {
|
|
const child = {
|
|
id: "child",
|
|
companyId: "company",
|
|
title: "child",
|
|
status: "done",
|
|
interactions: [
|
|
{
|
|
id: "first",
|
|
issueId: "child",
|
|
kind: "request_confirmation",
|
|
status: "rejected",
|
|
addresseeAgentId: "lead",
|
|
createdAt: "2026-09-17T19:07:19.277Z",
|
|
payload: { target: { type: "custom", key: "native_completion_review", revisionId: "decision-1" } },
|
|
result: { version: 1, outcome: "rejected" },
|
|
},
|
|
{
|
|
id: "second",
|
|
issueId: "child",
|
|
kind: "request_confirmation",
|
|
status: "accepted",
|
|
addresseeAgentId: "lead",
|
|
resolvedByAgentId: "lead",
|
|
resolvedByRunId: "review-run",
|
|
createdAt: "2026-09-17T19:08:54.510Z",
|
|
resolvedAt: "2026-09-17T19:09:33.722Z",
|
|
payload: { target: { type: "custom", key: "native_completion_review", revisionId: "decision-2" } },
|
|
result: {
|
|
version: 1,
|
|
outcome: "accepted",
|
|
},
|
|
},
|
|
],
|
|
};
|
|
const runs = [{
|
|
id: "review-run",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "succeeded",
|
|
contextSnapshot: {
|
|
nativeReviewInteractionId: "second",
|
|
nativeReviewDecisionId: "decision-2",
|
|
},
|
|
}];
|
|
|
|
it("follows a later accepted repair review for the same child and lead", () => {
|
|
expect(storyAcceptedAgentReview(child, "first", "lead", runs)?.id).toBe("second");
|
|
});
|
|
|
|
it("accepts a single first review when it is already accepted", () => {
|
|
expect(
|
|
storyAcceptedAgentReview(
|
|
{ ...child, interactions: [child.interactions[1]] },
|
|
"second",
|
|
"lead",
|
|
runs,
|
|
)?.id,
|
|
).toBe("second");
|
|
});
|
|
|
|
it("does not pass with only a rejected review", () => {
|
|
expect(
|
|
storyAcceptedAgentReview(
|
|
{ ...child, interactions: [child.interactions[0]] },
|
|
"first",
|
|
"lead",
|
|
runs,
|
|
),
|
|
).toBeUndefined();
|
|
});
|
|
|
|
it("rejects accepted cards for another child, agent, or interaction kind", () => {
|
|
const unrelated = [
|
|
{ ...child.interactions[1], issueId: "other" },
|
|
{ ...child.interactions[1], addresseeAgentId: "other", resolvedByAgentId: "other" },
|
|
{ ...child.interactions[1], payload: { target: { type: "custom", key: "other", revisionId: "decision-2" } } },
|
|
{ ...child.interactions[1], resolvedByRunId: "wrong-run" },
|
|
];
|
|
for (const interaction of unrelated)
|
|
expect(
|
|
storyAcceptedAgentReview({ ...child, interactions: [interaction] }, "first", "lead", runs),
|
|
).toBeUndefined();
|
|
});
|
|
|
|
it("rejects a review whose decision revision does not match the reviewer run", () => {
|
|
const bad = {
|
|
...child.interactions[1],
|
|
payload: { target: { type: "custom", key: "native_completion_review", revisionId: "wrong" } },
|
|
};
|
|
expect(storyAcceptedAgentReview({ ...child, interactions: [bad] }, "second", "lead", runs)).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
describe("manual everyday workflow catalog", () => {
|
|
it("is discoverable and explicitly selected without changing scheduled --all", () => {
|
|
const all = selectRunnerExecutions(parseRunnerSelectors(["--all"]));
|
|
expect(all.some((e) => e.suite.id === "everyday-workflows")).toBe(false);
|
|
const selected = selectRunnerExecutions(
|
|
parseRunnerSelectors(["--suite", "everyday-workflows"]),
|
|
);
|
|
expect(selected).toHaveLength(47);
|
|
expect(selected.every((e) => e.profile.generation === "native")).toBe(true);
|
|
expect(
|
|
selectRunnerExecutions(
|
|
parseRunnerSelectors(["--profile", "runner-codex"]),
|
|
).some((e) => e.suite.id === "everyday-workflows"),
|
|
).toBe(false);
|
|
const listed = selectRunnerExecutions(parseRunnerSelectors(["--list"]));
|
|
expect(listed.some((e) => e.suite.id === "everyday-workflows")).toBe(true);
|
|
});
|
|
it("does not schedule arbitrary runner-crash probes as model evals", () => {
|
|
const selected = selectRunnerExecutions(
|
|
parseRunnerSelectors(["--suite", "everyday-workflows"]),
|
|
);
|
|
expect(selected.some((e) => e.task.id.startsWith("recover-runner"))).toBe(false);
|
|
for (const id of ["recover-runner", "recover-runner-safe", "recover-runner-uncertain"])
|
|
expect(() => selectRunnerExecutions(parseRunnerSelectors([
|
|
"--id", `everyday-workflows.runner-codex.local.${id}`,
|
|
]))).toThrow();
|
|
expect(selected.some((e) => e.task.id === "recover-controller")).toBe(true);
|
|
expect(selected.some((e) => e.task.id === "stop-redirect")).toBe(true);
|
|
});
|
|
it("keeps ordinary prompts free of completion/API instructions", () => {
|
|
for (const task of everydayTasks.filter(
|
|
(candidate) => candidate.id !== "agent-review-handoff",
|
|
))
|
|
expect(task.buildPrompt("sample")).not.toMatch(
|
|
/finish_task|paperclip_finish|PATCH|mark .*done|idempotencyKey/i,
|
|
);
|
|
expect(
|
|
everydayTasks
|
|
.find((task) => task.id === "agent-review-handoff")!
|
|
.buildPrompt("sample"),
|
|
).toContain("resolve_review");
|
|
const profile = runnerSuites.find((s) => s.id === "everyday-workflows")!
|
|
.profiles[0]!;
|
|
const value = productionStoryProfile(profile).buildAgent({
|
|
environmentId: "env",
|
|
environmentFixtureId: "local",
|
|
workspacePath: "/tmp/test",
|
|
secretRefs: {
|
|
OPENAI_API_KEY: {
|
|
type: "secret_ref",
|
|
secretId: "secret",
|
|
version: "latest",
|
|
},
|
|
},
|
|
executionId: "case",
|
|
});
|
|
expect(JSON.stringify(value.instructionsBundle)).not.toMatch(
|
|
/fixture|mark .*done|finish_task|api\/issues/i,
|
|
);
|
|
});
|
|
it("does not claim unsupported remote crash/hiring coverage", () => {
|
|
const remote = runnerMatrix.filter(
|
|
(e) =>
|
|
e.suite.id === "everyday-workflows" && e.environment.id === "daytona",
|
|
);
|
|
expect(remote).toHaveLength(8);
|
|
expect(new Set(remote.map((e) => e.task.id))).toEqual(
|
|
new Set(["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"]),
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("lifecycle oracle calibrated failures", () => {
|
|
const parent = {
|
|
id: "parent",
|
|
companyId: "company",
|
|
title: "Project",
|
|
status: "done",
|
|
assigneeAgentId: "lead",
|
|
};
|
|
const run: StoryRun = {
|
|
id: "run",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "succeeded",
|
|
runtimeMode: "native",
|
|
runnerInstanceId: "runner",
|
|
contextSnapshot: { issueId: "parent" },
|
|
};
|
|
const score = (
|
|
runs: StoryRun[],
|
|
issues = [parent],
|
|
allowedInterruptedRuns: string[] = [],
|
|
) =>
|
|
storyLifecycleChecks({
|
|
issues,
|
|
runs,
|
|
parentId: parent.id,
|
|
leadId: "lead",
|
|
allowedInterruptedRuns,
|
|
});
|
|
it("accepts successful owned native work", () =>
|
|
expect(score([run]).every((c) => c.passed)).toBe(true));
|
|
it.each([
|
|
["legacy execution", { ...run, runtimeMode: "legacy" }, "native-runtime"],
|
|
[
|
|
"no runner identity",
|
|
{ ...run, runnerInstanceId: null },
|
|
"native-runtime",
|
|
],
|
|
["worker on parent", { ...run, agentId: "worker" }, "parent-owned-by-lead"],
|
|
["crash without recovery", { ...run, status: "failed" }, "successful-runs"],
|
|
["unsettled run", { ...run, status: "running" }, "settled"],
|
|
] as const)("rejects %s", (_label, bad, id) =>
|
|
expect(score([bad]).find((c) => c.id === id)?.passed).toBe(false),
|
|
);
|
|
it("does not exempt unrelated failures because another run was intentionally stopped", () => {
|
|
const checks = score(
|
|
[
|
|
{ ...run, status: "cancelled" },
|
|
{ ...run, id: "other", status: "failed" },
|
|
],
|
|
[parent],
|
|
["run"],
|
|
);
|
|
expect(checks.find((c) => c.id === "successful-runs")?.passed).toBe(false);
|
|
});
|
|
it.each(["adapter_failed", "tool_validation_error", "timed_out"])(
|
|
"does not excuse a later %s on the same deliberately interrupted run",
|
|
(errorCode) => {
|
|
const checks = score(
|
|
[{ ...run, status: "failed", errorCode }],
|
|
[parent],
|
|
[run.id],
|
|
);
|
|
expect(checks.find((c) => c.id === "successful-runs")?.passed).toBe(false);
|
|
const failed = { ...run, status: "failed", errorCode };
|
|
expect(storyUnexpectedRunFailure([failed], [run.id])).toBe(failed);
|
|
},
|
|
);
|
|
it.each([
|
|
["cancelled", "cancelled"],
|
|
["interrupted", "server_shutdown_interrupted"],
|
|
["failed", "process_lost"],
|
|
])("accepts an injected interruption with %s / %s", (status, errorCode) => {
|
|
const checks = score([{ ...run, status, errorCode }], [parent], [run.id]);
|
|
expect(checks.find((c) => c.id === "successful-runs")?.passed).toBe(true);
|
|
expect(storyUnexpectedRunFailure([{ ...run, status, errorCode }], [run.id])).toBeUndefined();
|
|
});
|
|
it("excludes a proven pre-dispatch workspace deferral without hiding executed failures", () => {
|
|
const deferred: StoryRun = {
|
|
id: "deferred",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "cancelled",
|
|
errorCode: "workspace_busy",
|
|
resultJson: {
|
|
executionRecovery: {
|
|
kind: "workspace_wait",
|
|
providerWorkStarted: false,
|
|
},
|
|
},
|
|
};
|
|
expect(score([run, deferred]).every((c) => c.passed)).toBe(true);
|
|
expect(
|
|
score([run, { ...deferred, processPid: 123 }]).every((c) => c.passed),
|
|
).toBe(false);
|
|
expect(
|
|
score([run, { ...deferred, resultJson: {} }]).every((c) => c.passed),
|
|
).toBe(false);
|
|
});
|
|
it("rejects a finished answer left in review", () =>
|
|
expect(
|
|
score([run], [{ ...parent, status: "in_review" }]).find(
|
|
(c) => c.id === "tasks-done",
|
|
)?.passed,
|
|
).toBe(false));
|
|
});
|
|
|
|
describe("reply completion boundary", () => {
|
|
const run: StoryRun = {
|
|
id: "old",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "succeeded",
|
|
runnerProfileJson: {
|
|
nativeExecutionInput: { task: { prompt: "Original task" } },
|
|
},
|
|
};
|
|
it("does not accept Done from the first run while a later request is still queued", () => {
|
|
expect(storyRepliesConsumed([run], ["later-comment"])).toBe(false);
|
|
const next = {
|
|
...run,
|
|
id: "new",
|
|
runnerProfileJson: {
|
|
nativeExecutionInput: { task: { prompt: "User reply later-comment" } },
|
|
},
|
|
};
|
|
expect(
|
|
storyRepliesConsumed(
|
|
[run, { ...next, status: "running" }],
|
|
["later-comment"],
|
|
),
|
|
).toBe(false);
|
|
expect(storyRepliesConsumed([run, next], ["later-comment"])).toBe(true);
|
|
});
|
|
|
|
it("waits for the expected agent reply after terminal state is visible", () => {
|
|
const issue = {
|
|
id: "parent",
|
|
companyId: "company",
|
|
title: "Stop and redirect",
|
|
status: "done",
|
|
comments: [
|
|
{
|
|
id: "user-reply",
|
|
authorAgentId: null,
|
|
body: "Change direction. Reference nonce.",
|
|
},
|
|
],
|
|
};
|
|
expect(storyHasAgentReply(issue, "lead", "Reference nonce")).toBe(false);
|
|
expect(
|
|
storyHasAgentReply(
|
|
{
|
|
...issue,
|
|
comments: [
|
|
...issue.comments,
|
|
{
|
|
id: "agent-reply",
|
|
authorAgentId: "lead",
|
|
body: "The studio is ready. Reference nonce.",
|
|
},
|
|
],
|
|
},
|
|
"lead",
|
|
"Reference nonce",
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyHasAgentReply(
|
|
{
|
|
...issue,
|
|
comments: [
|
|
...issue.comments,
|
|
{
|
|
id: "other-agent-reply",
|
|
authorAgentId: "worker",
|
|
body: "The studio is ready. Reference nonce.",
|
|
},
|
|
],
|
|
},
|
|
"lead",
|
|
"Reference nonce",
|
|
),
|
|
).toBe(false);
|
|
});
|
|
|
|
it("keeps a blocked lead alive when its child has a durable agent wake", () => {
|
|
const child = {
|
|
id: "child",
|
|
companyId: "company",
|
|
title: "Worker delivery",
|
|
status: "in_progress",
|
|
interactions: [
|
|
{
|
|
id: "review",
|
|
status: "pending",
|
|
continuationPolicy: "wake_assignee",
|
|
addresseeAgentId: "lead",
|
|
},
|
|
],
|
|
wakeDiagnostics: {
|
|
events: [
|
|
{
|
|
kind: "wake_request",
|
|
agentId: "lead",
|
|
status: "queued",
|
|
},
|
|
],
|
|
},
|
|
};
|
|
expect(storyHasPendingAgentWake(child, "lead")).toBe(true);
|
|
expect(
|
|
storyHasPendingAgentWork(
|
|
[
|
|
{ ...child, status: "blocked", wakeDiagnostics: { events: [] } },
|
|
child,
|
|
],
|
|
"lead",
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyHasPendingAgentWork(
|
|
[{ ...child, status: "blocked", wakeDiagnostics: { events: [] } }],
|
|
"lead",
|
|
),
|
|
).toBe(false);
|
|
expect(
|
|
storyHasStrandedBlockedLeaf(
|
|
[{ ...child, status: "blocked", wakeDiagnostics: { events: [] } }],
|
|
"lead",
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyHasStrandedBlockedLeaf(
|
|
[
|
|
{ ...child, status: "blocked", wakeDiagnostics: { events: [] } },
|
|
child,
|
|
],
|
|
"lead",
|
|
),
|
|
).toBe(false);
|
|
expect(storyHasPendingHumanInteraction(child, "lead")).toBe(false);
|
|
expect(storyHasPendingAgentReview(child, "lead")).toBe(true);
|
|
expect(storyHasPendingAgentWake(child, "worker")).toBe(false);
|
|
expect(
|
|
storyHasPendingHumanInteraction(
|
|
{
|
|
...child,
|
|
wakeDiagnostics: { events: [] },
|
|
},
|
|
"lead",
|
|
),
|
|
).toBe(true);
|
|
const sameAgentCards = {
|
|
...child,
|
|
interactions: [
|
|
{
|
|
id: "review-a",
|
|
status: "pending",
|
|
continuationPolicy: "wake_assignee",
|
|
addresseeAgentId: "lead",
|
|
},
|
|
{
|
|
id: "review-b",
|
|
status: "pending",
|
|
continuationPolicy: "wake_assignee",
|
|
addresseeAgentId: "lead",
|
|
},
|
|
],
|
|
wakeDiagnostics: {
|
|
events: [
|
|
{
|
|
kind: "wake_request",
|
|
agentId: "lead",
|
|
status: "queued",
|
|
payload: { nativeReviewInteractionId: "review-a" },
|
|
},
|
|
],
|
|
},
|
|
};
|
|
expect(storyHasPendingAgentReview(sameAgentCards, "lead")).toBe(true);
|
|
expect(storyHasPendingHumanInteraction(sameAgentCards, "lead")).toBe(
|
|
true,
|
|
);
|
|
expect(
|
|
storyHasPendingHumanInteraction(
|
|
{
|
|
...child,
|
|
interactions: [
|
|
{
|
|
id: "review",
|
|
status: "pending",
|
|
continuationPolicy: "wake_assignee",
|
|
effectiveResolverPolicy: "human_only",
|
|
addresseeAgentId: "lead",
|
|
},
|
|
],
|
|
},
|
|
"lead",
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyHasPendingHumanInteraction(
|
|
{
|
|
...child,
|
|
interactions: [
|
|
...(child.interactions ?? []),
|
|
{
|
|
id: "human-review",
|
|
status: "pending",
|
|
continuationPolicy: "wake_assignee",
|
|
resolverPolicy: "human_only",
|
|
addresseeAgentId: null,
|
|
},
|
|
],
|
|
},
|
|
["lead", "hired-worker"],
|
|
),
|
|
).toBe(true);
|
|
expect(storyHasPendingAgentWake(child, ["hired-worker", "lead"])).toBe(
|
|
true,
|
|
);
|
|
expect(
|
|
storyIssueHasBlockedTimeline({
|
|
...child,
|
|
status: "done",
|
|
activity: [
|
|
{
|
|
action: "issue.updated",
|
|
details: { fromStatus: "in_progress", toStatus: "blocked" },
|
|
createdAt: "2026-09-17T08:00:00.000Z",
|
|
},
|
|
],
|
|
}),
|
|
).toBe(true);
|
|
expect(
|
|
storyIssueHasBlockedTimelineBefore(
|
|
{
|
|
...child,
|
|
status: "done",
|
|
activity: [
|
|
{
|
|
action: "issue.updated",
|
|
details: { status: "blocked" },
|
|
createdAt: "2026-09-17T08:00:00.000Z",
|
|
},
|
|
],
|
|
},
|
|
"2026-09-17T08:01:00.000Z",
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyIssueHasUnresolvedDependency({
|
|
...child,
|
|
status: "blocked",
|
|
wakeDiagnostics: {
|
|
events: [],
|
|
blockerDiagnostics: {
|
|
readiness: { unresolvedBlockerCount: 1 },
|
|
},
|
|
},
|
|
}),
|
|
).toBe(true);
|
|
expect(
|
|
storyRunReportsDependencyBlock({
|
|
id: "parent-run",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "succeeded",
|
|
resultJson: {
|
|
nativeResult: {
|
|
reportedWorkDisposition: "blocked",
|
|
blocker: { reasonCode: "dependency_unresolved" },
|
|
},
|
|
},
|
|
}),
|
|
).toBe(true);
|
|
});
|
|
|
|
it("keeps polling when review acceptance is durable but the parent wake is not projected yet", () => {
|
|
const issues = [
|
|
{
|
|
id: "parent", companyId: "company", title: "parent", status: "blocked",
|
|
blockedTransitionAt: "2026-09-17T19:07:30.000Z",
|
|
},
|
|
{
|
|
id: "child",
|
|
parentId: "parent",
|
|
companyId: "company",
|
|
title: "child",
|
|
status: "done",
|
|
interactions: [{
|
|
id: "review",
|
|
issueId: "child",
|
|
kind: "request_confirmation",
|
|
status: "accepted",
|
|
addresseeAgentId: "lead",
|
|
resolvedByAgentId: "lead",
|
|
resolvedByRunId: "review-run",
|
|
resolvedAt: "2026-09-17T19:08:10.000Z",
|
|
payload: { target: { type: "custom", key: "native_completion_review", revisionId: "decision" } },
|
|
result: { version: 1, outcome: "accepted" },
|
|
}],
|
|
},
|
|
];
|
|
const runs = [{
|
|
id: "review-run",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "succeeded",
|
|
finishedAt: "2026-09-17T19:08:11.000Z",
|
|
}];
|
|
expect(storyHasStrandedBlockedLeaf(issues, "lead")).toBe(true);
|
|
expect(storyHasDurableAgentReviewContinuation(
|
|
issues, "parent", "lead", runs,
|
|
)).toBe(true);
|
|
for (const status of ["completed", "failed", "cancelled"]) {
|
|
expect(storyHasDurableAgentReviewContinuation([
|
|
{ ...issues[0]!, wakeDiagnostics: { events: [{
|
|
kind: "wake_request", agentId: "lead", reason: "issue_blockers_resolved", status,
|
|
}] } },
|
|
issues[1]!,
|
|
], "parent", "lead", runs)).toBe(false);
|
|
}
|
|
expect(storyHasDurableAgentReviewContinuation([
|
|
{ ...issues[0]!, blockedTransitionAt: "2026-09-17T19:09:00.000Z" }, issues[1]!,
|
|
], "parent", "lead", runs)).toBe(false);
|
|
});
|
|
|
|
it("still identifies a blocked parent as stranded without accepted review evidence", () => {
|
|
const issues = [
|
|
{ id: "parent", companyId: "company", title: "parent", status: "blocked" },
|
|
{ id: "child", parentId: "parent", companyId: "company", title: "child", status: "done", interactions: [] },
|
|
];
|
|
expect(storyHasDurableAgentReviewContinuation(issues, "parent", "lead", [])).toBe(false);
|
|
});
|
|
|
|
it.each(["deferred_issue_execution", "claimed"])(
|
|
"keeps a %s agent wake actionable",
|
|
(status) =>
|
|
expect(
|
|
storyHasPendingAgentWake(
|
|
{
|
|
id: "child",
|
|
companyId: "company",
|
|
title: "Worker delivery",
|
|
status: "in_progress",
|
|
wakeDiagnostics: {
|
|
events: [{ kind: "wake_request", agentId: "lead", status }],
|
|
},
|
|
},
|
|
"lead",
|
|
),
|
|
).toBe(true),
|
|
);
|
|
});
|
|
|
|
describe("delegation completion order", () => {
|
|
it("rejects a parent closed before its worker finishes", () => {
|
|
const base: StoryRun = {
|
|
id: "lead-run",
|
|
companyId: "company",
|
|
agentId: "lead",
|
|
status: "succeeded",
|
|
nativeIssueId: "parent",
|
|
finishedAt: "2026-09-14T12:00:00Z",
|
|
};
|
|
const child = {
|
|
...base,
|
|
id: "worker-run",
|
|
agentId: "worker",
|
|
nativeIssueId: "child",
|
|
finishedAt: "2026-09-14T12:01:00Z",
|
|
};
|
|
expect(
|
|
storyParentFinishedAfterChildren([base, child], "parent", "lead", [
|
|
"child",
|
|
]),
|
|
).toBe(false);
|
|
expect(
|
|
storyParentFinishedAfterChildren(
|
|
[{ ...base, finishedAt: "2026-09-14T12:02:00Z" }, child],
|
|
"parent",
|
|
"lead",
|
|
["child"],
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
storyParentFinishedAfterChildren([base], "parent", "lead", ["child"]),
|
|
).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("cancelled queued workspace retry", () => {
|
|
const queued: StoryRun = {
|
|
id: "retry",
|
|
companyId: "company",
|
|
agentId: "agent",
|
|
status: "cancelled",
|
|
runtimeMode: "legacy",
|
|
runtimeModeResolvedAt: null,
|
|
startedAt: null,
|
|
retryOfRunId: "workspace-deferral",
|
|
scheduledRetryReason: "workspace_busy",
|
|
errorCode: "cancelled",
|
|
lastOutputSeq: 0,
|
|
resultJson: {
|
|
startupCancellation: {
|
|
requestedAt: "2026-09-14T19:10:23.057Z",
|
|
beforeNativeSelection: false,
|
|
},
|
|
},
|
|
};
|
|
it("does not mistake an unstarted cancelled retry for provider execution", () => {
|
|
expect(isStoryWorkspaceDeferral(queued)).toBe(true);
|
|
});
|
|
it.each([
|
|
{ startedAt: "2026-09-14T19:10:22Z" },
|
|
{ processPid: 42 },
|
|
{ runnerInstanceId: "runner" },
|
|
{ nativeSessionId: "session" },
|
|
{ lastOutputSeq: 1 },
|
|
{ usageJson: { outputTokens: 1 } },
|
|
{ runtimeModeResolvedAt: "2026-09-14T19:10:22Z" },
|
|
{ scheduledRetryReason: "other" },
|
|
{ resultJson: null },
|
|
])("retains a contradictory or unproven cancellation %j", (change) => {
|
|
expect(isStoryWorkspaceDeferral({ ...queued, ...change })).toBe(false);
|
|
});
|
|
});
|