Files
PaperClipAI/tests
DottaandClaude Opus 5.5 2f59b9c67c test(chaos): fire a planned restart while the server is mid S3 transfer (#20451)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness restarts the control plane while agent
runs are in flight, and `--storage s3` runs the server against a
loopback MinIO behind a fault proxy
> - Planned restarts fire by time or by run events. A time-based restart
seldom lands while the server is moving bytes to or from the store
> - The riskiest storage moments are a workspace snapshot upload, a
snapshot restore, and a run-log journal write. A restart in the middle
of one must not lose or corrupt the workspace or the log
> - This pull request adds a storage-triggered restart: the planned
fault fires the moment the proxy sees a matching S3 request, and the
proxy holds that request so the restart interrupts it
> - The benefit is that restart-during-transfer becomes a repeatable
wave with receipts that name the interrupted request

## Linked Issues or Issue Description

**Subsystem affected**
Restart-chaos test harness (`tests/restart-chaos`), cloud storage mode.

**Problem or motivation**
A K1/K2/K4 restart fires on a timer or on a run trigger. With `--storage
s3`, the timer almost never lands during a snapshot upload, a snapshot
download, or a run-log write. So the harness does not prove that a
restart in the middle of a transfer is safe.

**Proposed solution**
Add `--storage-trigger <target>`. At its planned time, each task's fault
is armed. The next server request that matches the target fires it. The
proxy holds the request open (up to 60 s) while the harness snapshots
the tasks and restarts the server.

**Alternatives considered**
A new `--triggers storage` id. It needs edits to the observer (near its
1,000-line limit), the trigger registry and the delay table. The storage
trigger is a wave-level mode in a new module instead.

## What Changed

- New `tests/restart-chaos/storage-trigger.ts`: targets
`snapshot-upload`, `snapshot-part` (multipart `UploadPart`),
`snapshot-download` and `run-log`; the `StorageTrigger` that arms
faults, fires on a match, holds the request and records receipts.
- `storage-proxy.ts`: an `observe(listener)` hook. The listener sees
every request and can hold it (the existing latency path).
- `main.ts`, `config.ts`, `types.ts`: registry lines only.
`--storage-trigger` (needs `--storage s3` and `--triggers time`); a due
fault is armed instead of fired; the wave writes `storage-trigger.json`.
- `storage-trigger.json` records the matching request (method, path with
upload ids hidden, declared bytes, outcome `aborted` or `completed`,
status), every other request in flight, and the fault id. A fault that
never matched is recorded with `missedReason`.
- README: "Restart during a transfer".

## Verification

- `PAPERCLIP_IN_WORKTREE=false npx vitest run --config
tests/restart-chaos/vitest.config.ts` (25 files, 304 tests pass).
- Red/green: `storage-trigger.test.ts` against the master
`storage-proxy.ts` fails 3 of 6 tests (no hook, so nothing is held or
fired); it passes with this change.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Live waves (local run-log, Daytona snapshot upload and download) are
recorded in a PR comment as they finish.

## Risks

- Low risk. Test harness only. Without `--storage-trigger`, the proxy
has no listener and behaves as before.
- A replay re-arms its faults at their observed offsets. It fires on the
next matching request, not on the exact same request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 02:40:54 -06:00
..