Files
PaperClipAI/tests
DottaandClaude Opus 5.5 558dbb0132 test(restart-chaos): grade hosted runs with R1 and a strict release-swap gate (#22490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) proves that agent
work survives a control-plane release swap
> - Local waves grade each task with rules R1–R6. R1 is the time budget,
with credit for a provably fair queue wait (#22446, #22452)
> - Hosted mode (`hosted.ts`) runs the same proof on a real test staging
instance that takes automatic releases
> - Hosted mode passed a run when every task finished hands-off. It did
not apply R1, and banners, failed health probes, duplicates and the swap
shape were only reported
> - The next acceptance gate needs 10 consecutive hosted releases with
100% hands-off, 0 Retry banners and 0 failed health probes, so the exit
code must enforce these rules
> - This pull request adds hosted R1 and a strict "Gate D" verdict to
the hosted grade
> - The benefit is that one hosted run gives a single pass or fail with
the same R1 rule as the local waves

## Linked Issues or Issue Description

Refs #22446
Refs #22452
Refs #21153

**What happened?**

A hosted run (`hosted.ts --hosted-releases N`) exited 0 when every
started task finished without a person. The run did not grade R1. A
Retry banner that cleared by itself, a failed `/api/health` probe, a
duplicated agent comment, a stop-then-start release or a release with no
live run did not fail the run. These were only lines in `SUMMARY.md`.

**Expected behavior**

A hosted run passes only when every task finishes hands-off and inside
the R1 budget (with the same fair queue-wait credit as the local waves),
no Retry banner shows, no health probe fails, no work is duplicated, no
release is stop-then-start, and each release carries at least one live
run.

**Steps to reproduce**

1. Run `hosted.ts --hosted-releases 1` against a test staging instance
whose deploy is stop-then-start.
2. Let all tasks finish by themselves after the outage.
3. See exit 0, although health probes failed during the release.

**Paperclip version or commit**

`95ba0a7de6` (master)

## What Changed

- `hosted-gate.ts` (new): `hostedR1` builds the R1 input from API rows
and calls `gradeR1` from `grader.ts`. It uses the staging-mix budget
(`stagingMixBudgetMs`), outage time only for a release whose health
probes failed before the new build answered, and the fair queue-wait
proof (`queueWaitFairnessFrom`) from the company run list and each
agent's `maxConcurrentRuns`. A follow-up run's wait is not credited,
because the API has no wake evidence. `hostedGate` lists seven checks
and passes only when all hold.
- `hosted-readers.ts`: run rows keep `invocationSource`, `retryOfRunId`
and the wake comment ids.
- `hosted-grade.ts`: grades every started task with R1, adds the `Gate
D` block to `SUMMARY.md`, adds `gate` and `r1` to
`hosted-release-swap.json`, and returns the gate as the pass.
- `hosted-main.ts`: passes the task records to the grade.
- `README.md`: documents the gate and the API proxies for per-release
hand-off and reach.

## Verification

- `npx vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/hosted` passes (8 files, 56 tests). The new
`hosted-gate.test.ts` covers R1 pass and fail, a fair and an unfair
queue wait, outage credit, Retry and open tasks, and each gate check.
- `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit` is clean.
- No hosted run was started for this change. It changes only how a
finished run is graded.

## Risks

- Low risk. The change is in the test harness only.
- Hosted runs that passed before can now fail on R1, a banner, a failed
probe, a duplicate, a stop-then-start release or an unloaded release.
This is intended.
- A follow-up run's queue wait gets no credit on hosted runs, so R1 can
be stricter than in local waves when follow-ups queue for a long time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 07:33:25 -06:00
..