mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) proves that agent
tasks survive control-plane restarts
> - Staging takes a release about every 40 minutes as an overlap swap: a
new server starts beside the old one on the same database, both run for
8–9 minutes, then the old one gets SIGTERM
> - On 2026-11-12 staging failed or interrupted 577 of 1,000 runs across
these swaps, but the harness restarts one server at a time, grades each
task once and never repeats the swap
> - This pull request adds the release swap cell RS1: repeated overlap
swaps on one database, with the outgoing server starting new runs during
each overlap, and one end-of-wave grade that reads each task the way the
board shows it
> - The benefit is one number per wave, tasks completed without human
action over tasks started, that measures what staging users see
## Linked Issues or Issue Description
**Area**
Restart chaos harness (`tests/restart-chaos/`), test-only.
**Problem**
The harness did not reproduce the staging release shape. It had no
repeated swaps, no run started by the outgoing server during the
overlap, and no grade of every task at the end of a multi-swap wave.
**Proposed solution**
Add the `--release-swap` cell (RS1). It schedules K15 rolling redeploys
on a fixed clock, drives probe tasks through the outgoing server,
records per-swap evidence, and grades every task once after a settle
period.
**Why it matters**
The release cadence will not slow down. The harness must catch the
failures that the swap causes before staging does.
## What Changed
- `release-swap-schedule.ts`: the `--release-swap` options and the
`--release-swap-short` preset (3 swaps, 12 min apart, 510 s overlap)
(count 6, spacing 40 min, overlap 510 s, settle 15 min), their expansion
into a K15 overlap storm (adds `--k15-fresh-home --ephemeral-server` on
Daytona), the swap plan and the `ReleaseSwapRun` clock (swap k at k ×
spacing, never while a swap is unsettled; tasks launch until the last
swap settled).
- `release-swap-driver.ts`: the per-swap driver. It creates probe tasks
through the outgoing server's own port when the incoming server spawns,
when it enters startup recovery, and so that a run starts about 20 s
before the outgoing server's SIGTERM. It samples runs and their owning
controller PID every 2 s and records which server started each run. It
has a per-swap plug-in list; a plug-in can choose the checkout the
incoming server runs (for the later mixed-version variant).
- `release-swap-grade.ts`: the pure grade. One verdict per task (`pass`,
`needed_retry` with whether the click worked, `open_no_owner`,
`blocked_non_person`, `waiting_on_person`, `still_working`), the
per-swap numbers (failed runs, runs lost while the old server was alive,
time from SIGTERM to each surviving run's next event), duplicate runs
and duplicated side effects, the one-number summary, and one failure
label per failing task (`nothing pending`, `blocked for a non-person
reason`, `needed Retry` with the click result).
- `release-swap-finish.ts`: the end-of-wave collection. It waits for the
settle period, reads every task with a run from the API and the wave
database, clicks Retry where the board offers it (board UI when served,
else the same API call as the observer), waits again, and writes
`release-swap.json` and `release-swap-summary.md`.
- Small hooks in shared files: `supervisor.ts` calls `swapHook` at five
phases of the rolling redeploy and lets the hook choose the next server
root (the file stays at 1,000 lines); `config.ts`, `scheduler.ts`,
`types.ts` and `main.ts` wire the option, the plan, the clock, launches
during the overlap and the final grade.
- README section for the cell, its options and its grading.
- Unit tests for the scheduler, the option parsing, the grade
classifier, the per-swap numbers and the overlap evidence.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`
(58 files, 704 tests pass, after merging master).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Local smokes through the wave queue (`--env local --adapters opencode
--workloads A1,T1 --release-swap --release-swap-count 2
--release-swap-spacing-minutes 5 --release-swap-overlap-seconds 90
--release-swap-settle-minutes 2 --concurrency 2 --tasks 40`):
- `w-20261112T151735-s7101` (first head): 10 / 16 tasks completed
without human action; 26 min wall. It showed that native runs had no
owner in the sampler (fixed here).
- `w-20261112T154840-s7102` (owner fix): 8 / 13 tasks completed without
human action; 24 min wall. The late probe's run was started by the
outgoing server 20.0 s before its SIGTERM.
- Both smokes caught the same product failure: runs the outgoing server
hands off at shutdown (`restartSuspendedRunIds`) are classified
`awaitingEvidence` by the incoming server's startup recovery, even while
the outgoing server is still alive, and are never settled (`I2_settled`
reconciliation violations). They stay `running` with no process, and the
runs queued behind them are never dispatched. This PR does not fix it
(harness only).
- `w-20261112T163456-s7103` (probe agents, drain): 28 / 29 completed;
the late probe started on the outgoing server 20.0 s and 20.2 s before
SIGTERM in both swaps.
- `w-20261112T171115-s7104`, with 40 preparation probes per swap
overloaded the probe agent (late probe dispatched by the new server 30 s
after SIGTERM); the cap is now 6 per swap.
- CI notes on earlier heads: `ci / e2e shard (1/8)` failed once in
`task-loading-frame` (UI, outside this diff; re-run). `ci / General
tests (server (11/12))` failed in
`src/__tests__/storage-s3-multipart-throttle.test.ts` ("halves new
parts…", expected 3 to be 4), outside this diff; it is a known master
failure being fixed separately and is not touched here.
- Not run: the Daytona smoke. No published runner image matches this
checkout's content ID; an image build for master is queued.
## Risks
- Low risk: test-only harness code. No product code changes.
- Waves without `--release-swap` keep their plans and behavior. The only
shared-path change is the K15 overlap redeploy, which now calls an
optional hook that is unset in other waves.
- The probe tasks and the sampling add a few API calls and one database
query every 2 s during each swap.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Extended reasoning on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>