fix(evals): keep evidence for wakes retired by a terminal issue before start (#19258)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Product regression TestOps plan runs 20 Product E2E cells, and
each cell saves run evidence after grading
> - In the final full run, `first-task/clarify-propose-accept` passed
every journey check
> - The cell still failed while saving evidence: it requested a provider
log for a wake that never started
> - That wake was retired with `issue_terminal_status` because its child
task was already done, so no log exists
> - The settle oracle already accepts such a wake
(`isTerminalUnstartedWake`), but the evidence collector did not
> - This pull request records that wake's log as omitted, with a reason,
and keeps its events
> - The benefit is that a passing journey is no longer reported as
failed because of a log that never existed

## Linked Issues or Issue Description

Refs #19073, #19243

**What happened?**
In GitHub run
https://github.com/paperclipai/paperclip/actions/runs/42046147744,
`first-task.runner-codex.local.clarify-propose-accept` had no failed
checks in `first-task.json`. The conversational acceptance was recorded
through `resolve-from-comment` (200). The `finally` evidence step then
threw `GET /api/heartbeat-runs/ef2295c4…/log returned 404: Run log not
found`. Run `ef2295c4` is an `issue_status_changed` wake on the
completed child task. It is `cancelled` with `issue_terminal_status`,
and `startedAt`, `logStore` and `logRef` are all null.

**Expected behavior**
Evidence capture keeps that wake's events and records why it has no log,
as it already does for reset, blocked, reassigned and pre-provider runs.

**Steps to reproduce**
Run the cell. A child status-change wake may be queued just before the
child becomes `done`, and is then retired before admission.

## What Changed

- `collectChatRunEvidence` (`tests/runner-e2e/chat-flow.ts`): a run that
is `cancelled` with `errorCode: issue_terminal_status` and explicit null
`startedAt`, `logStore` and `logRef` gets `log: null` and
`logOmissionReason: "issue_terminal_before_start"`. Its events are still
collected.
- `chat-flow.test.ts`: a test for that case, plus negative controls.
Each of the following still fails on a missing log: started, `startedAt`
undefined, `failed` status, a different error code, a log store, a log
ref.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-flow.test.ts tests/runner-e2e/first-task.test.ts
tests/runner-e2e/continuation.test.ts`: 378 pass.
- `tsc -p tests/runner-e2e/tsconfig.json`: no errors in
`tests/runner-e2e`.
- Retained artifact from run 42046147744 (cell
`clarify-propose-accept`): there are no failed checks, and the run
ledger shows the retired wake with null `startedAt`, `logStore` and
`logRef`.

## Risks

- Low. This is an evidence-capture change in the eval harness, and the
journey's grading is unchanged. The omission needs explicit positive
evidence that the run never started and never had a log store. Any run
that did provider work still fails on a missing log.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-05 10:07:05 -06:00
1 parent 01dd6ab336
commit 45073d97ae
2 files changed
+17

No files matched your search

+13
View File
@@ -307,6 +307,19 @@ If you only have the club name, audience, and tone, that is enough to begin; I c
await expect(collectChatRunEvidence({ get }, { ...suppressed, ...patch })).rejects.toThrow("Run log not found");
}
});
it("retains a wake retired by a terminal issue before admission without requesting a log it never had", async () => {
const get = vi.fn().mockResolvedValue([{ seq: 1, eventType: "lifecycle" }]);
const retired = { ...run, status: "cancelled", errorCode: "issue_terminal_status", startedAt: null, logStore: null, logRef: null };
expect(await collectChatRunEvidence({ get }, retired)).toMatchObject({
runId: run.id, log: null, logOmissionReason: "issue_terminal_before_start", events: [{ seq: 1, eventType: "lifecycle" }],
});
expect(get.mock.calls).toEqual([[`/api/heartbeat-runs/${run.id}/events?afterSeq=0&limit=1000`]]);
get.mockRejectedValue(new Error("Run log not found"));
for (const patch of [{ startedAt: "2026-10-26T10:00:15Z" }, { startedAt: undefined }, { status: "failed" },
{ errorCode: "cancelled" }, { logStore: "local_file" }, { logRef: "recorded-log" }]) {
await expect(collectChatRunEvidence({ get }, { ...retired, ...patch })).rejects.toThrow("Run log not found");
}
});
it("records legitimate pre-provider log absence only with explicit bootstrap and storage evidence", async () => {
const stopped = { ...run, status: "interrupted", runtimeMode: "legacy", runtimeModeResolvedAt: null, logStore: null, logRef: null,
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
+4
View File
@@ -319,12 +319,16 @@ export async function collectChatRunEvidence(
run.logStore === null && run.logRef === null && ["failed", "interrupted"].includes(run.status) &&
recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false;
const reassignedBeforeStart = isReassignedUnstartedWake(run);
const terminalBeforeStart = run.status === "cancelled" && run.errorCode === "issue_terminal_status" &&
run.startedAt === null && run.logStore === null && run.logRef === null;
return {
runId: run.id,
...(providerNeverStarted ? { logOmissionReason: "provider_not_started" } : {}),
...(admissionWait ? { logOmissionReason: "pre_dispatch_execution_wait" } : {}),
...(reassignedBeforeStart ? { logOmissionReason: "assignee_changed_before_start" } : {}),
...(terminalBeforeStart ? { logOmissionReason: "issue_terminal_before_start" } : {}),
log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) || providerNeverStarted || admissionWait || reassignedBeforeStart
|| terminalBeforeStart
? null
: await api.get(`/api/heartbeat-runs/${run.id}/log?limitBytes=1048576`),
events: admissionEvents ?? await collectRunEvents((afterSeq, limit) =>