mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(evals): keep evidence for wakes retired by a terminal issue before start (#19258)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Product regression TestOps plan runs 20 Product E2E cells, and each cell saves run evidence after grading > - In the final full run, `first-task/clarify-propose-accept` passed every journey check > - The cell still failed while saving evidence: it requested a provider log for a wake that never started > - That wake was retired with `issue_terminal_status` because its child task was already done, so no log exists > - The settle oracle already accepts such a wake (`isTerminalUnstartedWake`), but the evidence collector did not > - This pull request records that wake's log as omitted, with a reason, and keeps its events > - The benefit is that a passing journey is no longer reported as failed because of a log that never existed ## Linked Issues or Issue Description Refs #19073, #19243 **What happened?** In GitHub run https://github.com/paperclipai/paperclip/actions/runs/42046147744, `first-task.runner-codex.local.clarify-propose-accept` had no failed checks in `first-task.json`. The conversational acceptance was recorded through `resolve-from-comment` (200). The `finally` evidence step then threw `GET /api/heartbeat-runs/ef2295c4…/log returned 404: Run log not found`. Run `ef2295c4` is an `issue_status_changed` wake on the completed child task. It is `cancelled` with `issue_terminal_status`, and `startedAt`, `logStore` and `logRef` are all null. **Expected behavior** Evidence capture keeps that wake's events and records why it has no log, as it already does for reset, blocked, reassigned and pre-provider runs. **Steps to reproduce** Run the cell. A child status-change wake may be queued just before the child becomes `done`, and is then retired before admission. ## What Changed - `collectChatRunEvidence` (`tests/runner-e2e/chat-flow.ts`): a run that is `cancelled` with `errorCode: issue_terminal_status` and explicit null `startedAt`, `logStore` and `logRef` gets `log: null` and `logOmissionReason: "issue_terminal_before_start"`. Its events are still collected. - `chat-flow.test.ts`: a test for that case, plus negative controls. Each of the following still fails on a missing log: started, `startedAt` undefined, `failed` status, a different error code, a log store, a log ref. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-flow.test.ts tests/runner-e2e/first-task.test.ts tests/runner-e2e/continuation.test.ts`: 378 pass. - `tsc -p tests/runner-e2e/tsconfig.json`: no errors in `tests/runner-e2e`. - Retained artifact from run 42046147744 (cell `clarify-propose-accept`): there are no failed checks, and the run ledger shows the retired wake with null `startedAt`, `logStore` and `logRef`. ## Risks - Low. This is an evidence-capture change in the eval harness, and the journey's grading is unchanged. The omission needs explicit positive evidence that the run never started and never had a log store. Any run that did provider work still fails on a missing log. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
01dd6ab336
commit
45073d97ae
2 files changed
+17
No files matched your search
@@ -307,6 +307,19 @@ If you only have the club name, audience, and tone, that is enough to begin; I c
|
||||
await expect(collectChatRunEvidence({ get }, { ...suppressed, ...patch })).rejects.toThrow("Run log not found");
|
||||
}
|
||||
});
|
||||
it("retains a wake retired by a terminal issue before admission without requesting a log it never had", async () => {
|
||||
const get = vi.fn().mockResolvedValue([{ seq: 1, eventType: "lifecycle" }]);
|
||||
const retired = { ...run, status: "cancelled", errorCode: "issue_terminal_status", startedAt: null, logStore: null, logRef: null };
|
||||
expect(await collectChatRunEvidence({ get }, retired)).toMatchObject({
|
||||
runId: run.id, log: null, logOmissionReason: "issue_terminal_before_start", events: [{ seq: 1, eventType: "lifecycle" }],
|
||||
});
|
||||
expect(get.mock.calls).toEqual([[`/api/heartbeat-runs/${run.id}/events?afterSeq=0&limit=1000`]]);
|
||||
get.mockRejectedValue(new Error("Run log not found"));
|
||||
for (const patch of [{ startedAt: "2026-10-26T10:00:15Z" }, { startedAt: undefined }, { status: "failed" },
|
||||
{ errorCode: "cancelled" }, { logStore: "local_file" }, { logRef: "recorded-log" }]) {
|
||||
await expect(collectChatRunEvidence({ get }, { ...retired, ...patch })).rejects.toThrow("Run log not found");
|
||||
}
|
||||
});
|
||||
it("records legitimate pre-provider log absence only with explicit bootstrap and storage evidence", async () => {
|
||||
const stopped = { ...run, status: "interrupted", runtimeMode: "legacy", runtimeModeResolvedAt: null, logStore: null, logRef: null,
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
|
||||
|
||||
@@ -319,12 +319,16 @@ export async function collectChatRunEvidence(
|
||||
run.logStore === null && run.logRef === null && ["failed", "interrupted"].includes(run.status) &&
|
||||
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
|
||||
const reassignedBeforeStart = isReassignedUnstartedWake(run);
|
||||
const terminalBeforeStart = run.status === "cancelled" && run.errorCode === "issue_terminal_status" &&
|
||||
run.startedAt === null && run.logStore === null && run.logRef === null;
|
||||
return {
|
||||
runId: run.id,
|
||||
...(providerNeverStarted ? { logOmissionReason: "provider_not_started" } : {}),
|
||||
...(admissionWait ? { logOmissionReason: "pre_dispatch_execution_wait" } : {}),
|
||||
...(reassignedBeforeStart ? { logOmissionReason: "assignee_changed_before_start" } : {}),
|
||||
...(terminalBeforeStart ? { logOmissionReason: "issue_terminal_before_start" } : {}),
|
||||
log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) || providerNeverStarted || admissionWait || reassignedBeforeStart
|
||||
|| terminalBeforeStart
|
||||
? null
|
||||
: await api.get(`/api/heartbeat-runs/${run.id}/log?limitBytes=1048576`),
|
||||
events: admissionEvents ?? await collectRunEvents((afterSeq, limit) =>
|
||||
|
||||
Reference in new issue
Block a user