Files
PaperClipAI/tests
DottaandPaperclip 59d1b7b43d fix(evals): collect usage when GitHub skips jobs without a runner (#17352)
## Thinking Path

> - Paperclip manages AI agents and their operating budgets.
> - Product E2E runs retain accounting for each campaign.
> - The usage collector reads completed GitHub jobs without rerunning
paid work.
> - GitHub returns a null runner ID for jobs that it skips before
assigning a runner.
> - The collector rejected those jobs and could not retain the completed
campaign inventory.
> - This change accepts that skipped-job representation while preserving
started-job checks.

## Linked Issues or Issue Description

Refs #17009.

**What happened?**

The usage collector rejected completed qualification run 39971835384.
GitHub returned skipped jobs with null runner IDs. The collector
accepted only zero for jobs without a runner.

**Expected behavior**

Retain completed run usage when skipped jobs have a null or zero runner
ID. Do not count VM time for those jobs. Continue to reject missing
runner identity for started jobs.

**Steps to reproduce**

Read the completed first-attempt Product E2E run with
operating-receipts.ts github. Before this fix, a skipped job with
runner_id set to null throws Invalid workflow job identity.

**Paperclip version or commit**

ed3f067dbd.

## What Changed

- Accept a null or zero runner ID only for completed skipped jobs.
- Keep run, attempt, labels and started-job identity validation.
- Test both skipped-job forms and reject absent runner IDs.
- Document the treatment of skipped jobs.

## Verification

- Reproduced the observed null-runner failure: 1 failed, 4 passed.
- Usage collector and operating receipt tests: 17 passed.
- Eval TypeScript check passed.
- Read the same completed hosted run successfully without dispatching or
retrying tests. The inventory retained 1,303 job seconds and 290,156,153
artifact bytes.
- Full hosted CI passed at fb57826c22,
including typecheck, tests, build and browser shards.
- Greptile and Paperclip reviews rated the exact commit 5/5 with no
actionable findings.

## Risks

- The exception applies only when GitHub reports a completed skipped job
and an explicit null or zero runner ID.
- Current usage remains an inventory. This change does not convert it to
a bill, release cost holds, or enable recurrence.

## Model Used

- OpenAI GPT-6 in Codex, with reasoning, repository inspection, code
execution and test tools. The exact backend model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 11:35:35 -07:00
..