From f56aee54236f653ad191d82e70072a3ebc8894af Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 25 Sep 2026 08:06:33 -0700 Subject: [PATCH] fix(evals): capture complete durable run event streams (#13977) ## Thinking Path > - Paperclip manages AI agents and records their durable work outcomes. > - Product E2E checks those outcomes through the browser and public API. > - A successful long run can emit more than 1,000 durable events. > - The harness read one page and missed the later completion evidence. > - This pull request reads every page before it checks runtime invariants. > - Invalid or incomplete capture still fails. A missing page cannot produce a pass. ## Linked Issues or Issue Description Related: #13882 (Grok qualification). A duplicate search found no existing event-pagination fix. **What happened?** The local structured-question case in [campaign 36071063537](https://github.com/paperclipai/paperclip/actions/runs/36071063537) completed the task and passed its six outcome matchers. It failed native runtime invariants because the capture contained exactly 1,000 events. The last captured event preceded the run's completion by more than a minute. The API caps each response at 1,000 rows. The harness did not request the next page. **Expected behavior** Read the complete durable event stream through the public API before checking semantic-result and terminal-event counts. Reject incomplete or malformed evidence. **Steps to reproduce** 1. Complete a native task that emits more than 1,000 durable events. 2. Place the semantic-result and terminal events after row 1,000. 3. Capture the run with the Product E2E harness. 4. Before this fix, the invariant checker sees only the first page. **Paperclip version or commit** Observed at `4196a4cd76db434854b679035e4146c7f69689ce`. The same single-page capture exists on master. The original failed result remains unchanged; missing historical tail evidence is not reconstructed or graded as a pass. ## What Changed - Add a bounded event collector that advances through the public `afterSeq` cursor. - Use it for task success/failure evidence and shared chat run evidence. - Reject invalid pages, missing or non-increasing sequence numbers, repeated cursors, failed later requests, and an exhausted page limit. - Test completion events beyond the first page, exact page boundaries, and malformed evidence. - Correct the existing Everyday catalog test from 38 to the maintained 47 cells. The suite stays explicit-only. - Document the complete-capture requirement and its bound. ## Verification - Product harness typecheck passes. - The full credential-free harness suite passed 477 tests. After adding the chat integration regression, all 34 chat evidence tests pass. - Seventeen pagination tests cover the valid tail and malformed-evidence cases. - `git diff --check` passes. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/36080683422) passes at `aed6f79c089226be79e75dcf390969561ec4f787`: 52 successful checks and two intentional skips, including Rust, typecheck, build, server tests, and browser shards. Greptile gives this exact head 5/5 with no findings. No local Docker, Rust build, browser suite, or model invocation was used for this change. - [Live Grok requalification](https://github.com/paperclipai/paperclip/actions/runs/36080870743) is running on combined source `1b0551bb7c8de3c54f4bee64dbe2c88328b3645e`, with the runner, scheduler, and evidence fixes. Its full credential-free harness passes all 493 tests and typecheck. Live results are pending; the original campaign remains a failed measurement. ## Risks Long runs need more read-only API requests and larger private evidence files. Capture stops with an explicit error after 100 full pages. This changes neither production APIs nor provider behavior. It does not relax an invariant or change a historical grade. ## Model Used OpenAI GPT-6 through Codex, with repository tools and code execution. The exact serving identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- tests/runner-e2e/README.md | 6 ++++ tests/runner-e2e/chat-flow.test.ts | 19 +++++++++++-- tests/runner-e2e/chat-flow.ts | 5 +++- tests/runner-e2e/everyday.test.ts | 2 +- tests/runner-e2e/run-observations.ts | 25 +++++++++++++++++ tests/runner-e2e/runner.spec.ts | 9 +++--- tests/runner-e2e/support.test.ts | 42 ++++++++++++++++++++++++++++ 7 files changed, 99 insertions(+), 9 deletions(-) diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index ffe53d181b0..db28e68025a 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -381,6 +381,12 @@ published with permanent campaign dashboards; fixture authors must therefore keep credentials and other private data out of every captured UI state. SVG is active content and is rejected from the packaged evidence entirely. +The standard task and chat evidence collectors read durable run events through the paginated +public API, including completion events beyond the first 1,000 rows. It rejects +missing, repeated, or out-of-order sequence numbers and fails capture after +100 full pages instead of grading a truncated stream. Original incomplete +captures remain failed evidence; qualifying a fix requires a new live attempt. + Every completed local campaign also writes `tests/runner-e2e/results//dashboard.html`. The self-contained page shows the complete profile/environment grid with screenshot thumbnails. diff --git a/tests/runner-e2e/chat-flow.test.ts b/tests/runner-e2e/chat-flow.test.ts index c0a1a174fa8..f69ff74cd3e 100644 --- a/tests/runner-e2e/chat-flow.test.ts +++ b/tests/runner-e2e/chat-flow.test.ts @@ -224,21 +224,34 @@ describe("chat acceptance contracts", () => { ); }); it("retains reset events without requesting a provider log, and does not hide missing real logs", async () => { - const get = vi.fn().mockResolvedValue([{ type: "session_reset" }]); + const get = vi.fn().mockResolvedValue([{ seq: 1, type: "session_reset" }]); const reset = { ...run, resultJson: { conversationReset: true } }; await expect(collectChatRunEvidence({ get }, reset)).resolves.toEqual({ runId: run.id, log: null, - events: [{ type: "session_reset" }], + events: [{ seq: 1, type: "session_reset" }], }); expect(get.mock.calls).toEqual([ - [`/api/heartbeat-runs/${run.id}/events?limit=1000`], + [`/api/heartbeat-runs/${run.id}/events?afterSeq=0&limit=1000`], ]); get.mockRejectedValue(new Error("Run log not found")); await expect(collectChatRunEvidence({ get }, run)).rejects.toThrow( "Run log not found", ); }); + it("retains the tail of a long chat run for invariant checks", async () => { + const firstPage = Array.from({ length: 1000 }, (_, i) => ({ seq: i + 1, eventType: "item.delta" })); + const tail = { seq: 1001, eventType: "run.terminal" }; + const get = vi.fn().mockResolvedValueOnce({ content: "fixture log" }) + .mockResolvedValueOnce(firstPage).mockResolvedValueOnce([tail]); + const evidence = await collectChatRunEvidence({ get }, run); + expect(evidence.events).toEqual([...firstPage, tail]); + expect(get.mock.calls).toEqual([ + [`/api/heartbeat-runs/${run.id}/log?limitBytes=1048576`], + [`/api/heartbeat-runs/${run.id}/events?afterSeq=0&limit=1000`], + [`/api/heartbeat-runs/${run.id}/events?afterSeq=1000&limit=1000`], + ]); + }); it("retains events for an unstarted dependency-blocked wake without asking for a nonexistent log", async () => { const get = vi.fn().mockResolvedValue([]); const suppressed = { ...run, status: "cancelled", errorCode: "issue_dependencies_blocked", startedAt: null }; diff --git a/tests/runner-e2e/chat-flow.ts b/tests/runner-e2e/chat-flow.ts index db36a37124c..a77712a674d 100644 --- a/tests/runner-e2e/chat-flow.ts +++ b/tests/runner-e2e/chat-flow.ts @@ -7,6 +7,7 @@ import type { import type { LiveFixtureValues } from "./live-fixtures.js"; import type { MatrixExecution } from "./types.js"; import { isBlockedUnstartedWake } from "./non-execution-wake.js"; +import { collectRunEvents } from "./run-observations.js"; import { chatMarker } from "./chat-cases.js"; import { assertChatRememberedAfterRestart, assertChatStartupStopped, isChatStopReady, runChatHardeningFlow } from "./chat-hardening.js"; import { enableChatThroughSettings, runChatInterruption, runChatSettingsLifecycle } from "./chat-stories.js"; @@ -239,7 +240,9 @@ export async function collectChatRunEvidence( log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) ? null : await api.get(`/api/heartbeat-runs/${run.id}/log?limitBytes=1048576`), - events: await api.get(`/api/heartbeat-runs/${run.id}/events?limit=1000`), + events: await collectRunEvents((afterSeq, limit) => + api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`), + ), }; } diff --git a/tests/runner-e2e/everyday.test.ts b/tests/runner-e2e/everyday.test.ts index 5016adf2688..c1322eb7943 100644 --- a/tests/runner-e2e/everyday.test.ts +++ b/tests/runner-e2e/everyday.test.ts @@ -212,7 +212,7 @@ describe("manual everyday workflow catalog", () => { const selected = selectRunnerExecutions( parseRunnerSelectors(["--suite", "everyday-workflows"]), ); - expect(selected).toHaveLength(38); + expect(selected).toHaveLength(47); expect(selected.every((e) => e.profile.generation === "native")).toBe(true); expect( selectRunnerExecutions( diff --git a/tests/runner-e2e/run-observations.ts b/tests/runner-e2e/run-observations.ts index e0f18d49cef..1e5b1949e22 100644 --- a/tests/runner-e2e/run-observations.ts +++ b/tests/runner-e2e/run-observations.ts @@ -1,3 +1,28 @@ +/** Read the complete durable stream; never grade a silently truncated page. */ +export async function collectRunEvents( + loadPage: (afterSeq: number, limit: number) => Promise, +): Promise { + const pageSize = 1000; + const events: T[] = []; + let afterSeq = 0; + for (let pageNumber = 0; pageNumber < 100; pageNumber += 1) { + const page = await loadPage(afterSeq, pageSize); + if (!Array.isArray(page) || page.length > pageSize) { + throw new Error("Run event evidence returned an invalid page"); + } + for (const event of page) { + const seq = event?.seq; + if (!Number.isSafeInteger(seq) || seq <= afterSeq) { + throw new Error("Run event evidence has a missing or non-increasing sequence"); + } + afterSeq = seq; + events.push(event as T); + } + if (page.length < pageSize) return events; + } + throw new Error("Run event evidence exceeded 100 pages; refusing incomplete evidence"); +} + export interface ObservableRunState { status?: string | null; errorCode?: string | null; diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 9649bf1da52..5e8392c529a 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -21,6 +21,7 @@ import { setupLiveFixtures, type LiveFixtureValues } from "./live-fixtures.js"; import { evaluateMatcher, persistedFinalRunMessage, type MatcherResult } from "./matchers.js"; import { acceptedPlanSessionResetFailures, + collectRunEvents, hasTerminalMalformedPlanConfirmation, isControlPlaneGovernedResponseWait, isNonExecutingReviewFenceRun, @@ -696,8 +697,8 @@ for (const execution of executions) { ), ), events: await capture(() => - api.get( - `/api/heartbeat-runs/${candidate.id}/events?limit=1000`, + collectRunEvents((afterSeq, limit) => + api.get(`/api/heartbeat-runs/${candidate.id}/events?afterSeq=${afterSeq}&limit=${limit}`), ), ), })), @@ -1694,8 +1695,8 @@ for (const execution of executions) { try { return { runId: candidate.id, - events: await api.get( - `/api/heartbeat-runs/${candidate.id}/events?limit=1000`, + events: await collectRunEvents((afterSeq, limit) => + api.get(`/api/heartbeat-runs/${candidate.id}/events?afterSeq=${afterSeq}&limit=${limit}`), ), error: null, }; diff --git a/tests/runner-e2e/support.test.ts b/tests/runner-e2e/support.test.ts index e7c4cb95fff..9c71367fa0b 100644 --- a/tests/runner-e2e/support.test.ts +++ b/tests/runner-e2e/support.test.ts @@ -38,6 +38,7 @@ import { } from "./ports.js"; import { acceptedPlanSessionResetFailures, + collectRunEvents, hasTerminalMalformedPlanConfirmation, isControlPlaneGovernedResponseWait, isNonExecutingReviewFenceRun, @@ -48,6 +49,47 @@ import { import { runnerE2EWebServerCommand } from "./web-server-command.js"; const cleanupDirectories: string[] = []; + +describe("complete run event evidence", () => { + const page = Array.from({ length: 1000 }, (_, i) => ({ seq: i + 1, eventType: "item.delta" })); + it("reads completion events beyond the first 1000 rows", async () => { + const terminal = ["run.result.proposed", "run.result.accepted", "run.terminal"] + .map((eventType, i) => ({ seq: 1001 + i, eventType })); + const load = vi.fn().mockResolvedValueOnce(page).mockResolvedValueOnce(terminal); + const events = await collectRunEvents(load); + expect(events).toEqual([...page, ...terminal]); + expect(load.mock.calls).toEqual([[0, 1000], [1000, 1000]]); + }); + it("checks for another page even at an exact page boundary", async () => { + const load = vi.fn().mockResolvedValueOnce(page).mockResolvedValueOnce([]); + expect(await collectRunEvents(load)).toEqual(page); + expect(load).toHaveBeenCalledTimes(2); + }); + it.each([ + null, {}, [{ eventType: "run.terminal" }], [{ seq: 0 }], [{ seq: -1 }], + [{ seq: 1.5 }], [{ seq: "1" }], [{ seq: NaN }], [{ seq: Infinity }], + [{ seq: 2 }, { seq: 1 }], [{ seq: 1 }, { seq: 1 }], [...page, { seq: 1001 }], + ])("rejects malformed evidence page %#", async (malformed) => { + await expect(collectRunEvents(async () => malformed)).rejects.toThrow("Run event evidence"); + }); + it("rejects a repeated cursor instead of accepting duplicate events", async () => { + const load = vi.fn().mockResolvedValue(page); + await expect(collectRunEvents(load)).rejects.toThrow("non-increasing sequence"); + expect(load).toHaveBeenCalledTimes(2); + }); + it("propagates a missing later page without returning partial evidence", async () => { + const load = vi.fn().mockResolvedValueOnce(page).mockRejectedValueOnce(new Error("Unavailable")); + await expect(collectRunEvents(load)).rejects.toThrow("Unavailable"); + }); + it("fails closed when a stream never ends within the bounded capture", async () => { + const load = vi.fn(async (afterSeq: number) => page.map((event) => ({ ...event, seq: event.seq + afterSeq }))); + await expect(collectRunEvents(load)).rejects.toThrow("refusing incomplete evidence"); + expect(load).toHaveBeenCalledTimes(100); + }); +}); + + + afterEach(async () => { vi.unstubAllEnvs(); vi.unstubAllGlobals();