Files
PaperClipAI/tests
DottaandClaude Opus 5.5 95cb3c94c4 test(restart-chaos): cap store bandwidth during ST2 slow-store windows (#20452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness tests how runs survive restarts and cloud
storage faults
> - A Paperclip-sized remote workspace (R1 on Daytona with `--storage
s3`) saves about 730 MiB per snapshot as a multipart upload and restores
it as one large download
> - The ST2 "slow store" fault only adds latency per request. On a
healthy loopback store the whole multipart upload takes about 2 s, so a
latency fault adds a fixed delay per part and never makes a large
transfer slow
> - A real degraded store is slow per byte too. We need that to test
large saves and restores that take minutes
> - This pull request gives each ST2 window a seeded bandwidth cap that
all requests in the window share
> - The benefit is that ST2 waves now slow large snapshot uploads and
downloads by minutes, and the harness can find save and restore timeouts
on big workspaces

## Linked Issues or Issue Description

Refs #20299, #20313 (storage outage handling on small workspaces),
#20442 (saved workspace Files over 512 MiB).

**What existing behavior does this improve?**
The ST2 slow-store fault in the restart-chaos storage proxy.

**Subsystem affected**
Test harness (`tests/restart-chaos`).

**Current behavior**
ST2 adds 1–10 s of latency before each request. The bytes then flow at
loopback speed. A 730 MiB snapshot upload or download is still fast.

**Proposed behavior**
ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction
is shared by every request in the window, so parallel multipart parts
share the cap.

**Reason and benefit**
Large saves and restores now take minutes under ST2. This shows the save
and restore timing, the storage request counts, and any timeout on a
slow store.

## What Changed

- New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate
pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2
bandwidth draw (2–16 MiB/s).
- `storage-proxy.ts`: the `latency` fault takes an optional
`bytesPerSecond`. The proxy paces the request body and the response body
through one rate per direction for each rule.
- `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts
keep the rule, so a replay uses the same cap.
- README: the ST2 row describes the cap.

## Verification

- `PAPERCLIP_IN_WORKTREE=false npx vitest run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/storage-bandwidth.test.ts
tests/restart-chaos/storage-proxy.test.ts
tests/restart-chaos/storage-chaos.test.ts`: 14 pass.
- Red before: with the proxy and ST2 changes reverted, the proxy cap
test and the ST2 draw test fail. Green after.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass.

## Risks

- Low risk. Test harness only. ST2 now draws one more seeded number, so
a seed gives different later windows than before. Replays read the saved
receipts and are not affected.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:03:56 -06:00
..