Files
PaperClipAI/tests
DottaandClaude Opus 5.5 6a24b417c0 test(restart-chaos): add hosted mode for the release-swap proof on a real staging instance (#21153)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted instances take an automatic release about every 40 minutes as
an overlap swap: a new server starts beside the old one, both serve for
a while, then the old one stops
> - The restart-chaos harness (`tests/restart-chaos/`) proves release
swaps only on servers it starts locally, with the release-swap cell RS1
(#21126) and the staging-mix workload (#21125)
> - The zero-downtime finish line needs the same proof on a real hosted
staging instance across real releases, with the same per-task grade
> - This pull request adds a hosted mode: the harness starts no server
and injects no fault; it runs the staging mix against one allowed hosted
instance through its API, detects each real release from `/api/health`,
and grades every started task with the RS1 grade module
> - The benefit is one comparable number per hosted run (`tasks
completed without human action / tasks started`) plus per-release
continuity, with no database access and no risk to other instances

## Linked Issues or Issue Description

Refs #21126
Refs #21125
Refs #21139

**Is your feature request related to a problem? Please describe.**
The release-swap proof runs only against local servers. It cannot show
what a real hosted instance does across its own automatic releases.

**Describe the solution you'd like**
A hosted mode in the harness. It takes one explicitly allowed staging or
test host, signs in with an existing board token, creates labelled test
agents and tasks, watches real releases through `/api/health`, grades
every task with the RS1 grade, reports web and API continuity, and
removes what it created.

**Describe alternatives you've considered**
Reading the hosted database directly. That needs production-grade
database credentials and gives a second, different collector. The API
path uses the same access a person has and reports the fields the API
cannot supply as unavailable.

## What Changed

- `hosted-guard.ts`: refuses to start unless `--hosted-base-url` is a
bare `https://` origin and `--hosted-allow` names the same host. Only
one stack under `.staging.paperclip.app` or `.test.paperclip.app` is
accepted. The shared staging instance, production, the fleet admin and
Cloud hosts are always refused. Every request is checked against that
origin before it is sent.
- `hosted-api.ts`: a bearer-token API client (token from
`PAPERCLIP_HOSTED_TOKEN` or the CLI store, never printed) that logs
every request for continuity, retries only reads, and a `/api/health`
probe.
- `hosted-release.ts`: pure release detection over health samples. It
reports each release's overlap window (two process identities
answering), failed and slow requests, time to the first answer from the
new build, its shape (`overlap`, `stop_then_start`, `clean_switch`) and
probe continuity (requests, failures, longest gap).
- `hosted-readers.ts`, `hosted-grade.ts`: API-backed readers for runs,
tasks, task views, pending questions and approvals, failure chains, next
events and duplicated comments and subtasks. The grade calls
`gradeReleaseSwapTask`, `summarizeGrades`, `swapMetrics` and
`releaseSwapMarkdown` from `release-swap-grade.ts` (#21126), clicks
Retry where the board offers it, and labels each failure. Fields the API
cannot supply (queued wakes, the owning server process, runs lost while
the old server lived, double executors, late-start and preparation
probes) are listed as unavailable.
- `hosted-setup.ts`: works in the signed-in user's company (Cloud
refuses company creation). Creates one labelled project, one labelled
agent per adapter and one labelled secret per provider key through the
normal APIs. Cleanup pauses and removes the agents, cancels their live
runs, hides their tasks, deletes the secrets and archives the project.
It touches only names that carry the run label.
- `hosted-workload.ts`: drives the staging-mix tasks from
`staging-mix.ts` (#21125) at `--hosted-concurrency` (default 5), posts
follow-ups during runs, and clicks an offered Retry once per failed run.
- `hosted.ts`, `hosted-config.ts`, `hosted-main.ts`: the entry point,
options, `--hosted-observe-only` (read-only release watch),
`--hosted-cleanup-only`, and the run. `main.ts` and `supervisor.ts` are
unchanged.
- Hands-off grading (`hosted-handsoff.ts`, `hosted-workload.ts`): no
Retry click during the wave or the settle period. Each task records when
a Retry banner first showed and whether it cleared by itself. After the
settle period the harness clicks Retry once on tasks that are not done
and records whether it worked. `SUMMARY.md` starts with `<pass> /
<started> tasks completed hands-off`, `Retry banner shown: <n>` and
`Retry worked / clicked: <w> / <c>`. `--hosted-retry-during-wave` keeps
the old behavior.
- Assigned-work invariant: the grade reads `GET
/api/companies/:id/assigned-work-invariant` once (from #21139) and
reports the company's violation counts and the sampled violations and
stalled slots of this run. A 404 on an older build is reported as not
available.
- `README.md`: a Hosted mode section with commands, the guard, sign-in,
setup, detection, grade, unavailable fields and cleanup.

## Verification

- `node server/node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/hosted-*.test.ts
tests/restart-chaos/release-swap-grade.test.ts`: 52 tests pass (38
hosted tests). They cover the host guard (allowed host, mismatched allow
flag, the shared staging instance, production, fleet admin, Cloud and
look-alike hosts, a userinfo trick in a route), token lookup, release
detection (overlap, stop-then-start, a restart on the same commit,
unsettled release, continuity), the API readers, the end-to-end API
grade with a Retry click and labels, the workload, and label-only
cleanup.
- `node server/node_modules/typescript/bin/tsc -p
tests/restart-chaos/tsconfig.json`: clean.
- Run without a token: the harness stops before any request and names
the missing sign-in. Run against the shared staging host: refused by the
guard.
- `pnpm verify` asks for broader CI coverage (exit 2, README is a global
CI input); required CI supplies it.
- Known failures on master, outside this diff:
`server/src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves
new parts in flight once when a parallel burst of parts is throttled",
expected 3 to be 4) also fails on master and is being fixed separately.
`packages/paperclip-runner`
`src/drivers/acpx/native-distribution-integrity.test.ts` ("expires a
real ACPX permission on native child death without replaying its
mutation") is in the Runner package; this PR changes only
`tests/restart-chaos/`.
- Live smoke on one hosted staging instance (2 tasks): setup, workload,
grade and cleanup ran end to end. It showed two harness defects, now
fixed: an activity window that was too short for 90 s tool rounds, and
agent removal that needed termination first.
- Not run here: a hosted run across real releases (none reached the
instance during this work).

## Risks

- Low risk to the product: harness-only, no product code changes.
- The run writes to one hosted instance. The guard, the per-request
origin check and the label filter limit it to that instance and to
objects it created.
- The grade over the API cannot see queued wakes. A task whose only
owner is a queued wake grades as having no owner. The report says so.
- #21125 and #21126 are on master; this branch merges master and adds
only hosted files and the README section.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Default context window; no
extended thinking mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:19:20 -06:00
..