mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 10:21:41 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests how runs survive restarts and cloud storage faults > - A Paperclip-sized remote workspace (R1 on Daytona with `--storage s3`) saves about 730 MiB per snapshot as a multipart upload and restores it as one large download > - The ST2 "slow store" fault only adds latency per request. On a healthy loopback store the whole multipart upload takes about 2 s, so a latency fault adds a fixed delay per part and never makes a large transfer slow > - A real degraded store is slow per byte too. We need that to test large saves and restores that take minutes > - This pull request gives each ST2 window a seeded bandwidth cap that all requests in the window share > - The benefit is that ST2 waves now slow large snapshot uploads and downloads by minutes, and the harness can find save and restore timeouts on big workspaces ## Linked Issues or Issue Description Refs #20299, #20313 (storage outage handling on small workspaces), #20442 (saved workspace Files over 512 MiB). **What existing behavior does this improve?** The ST2 slow-store fault in the restart-chaos storage proxy. **Subsystem affected** Test harness (`tests/restart-chaos`). **Current behavior** ST2 adds 1–10 s of latency before each request. The bytes then flow at loopback speed. A 730 MiB snapshot upload or download is still fast. **Proposed behavior** ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction is shared by every request in the window, so parallel multipart parts share the cap. **Reason and benefit** Large saves and restores now take minutes under ST2. This shows the save and restore timing, the storage request counts, and any timeout on a slow store. ## What Changed - New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2 bandwidth draw (2–16 MiB/s). - `storage-proxy.ts`: the `latency` fault takes an optional `bytesPerSecond`. The proxy paces the request body and the response body through one rate per direction for each rule. - `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts keep the rule, so a replay uses the same cap. - README: the ST2 row describes the cap. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/storage-bandwidth.test.ts tests/restart-chaos/storage-proxy.test.ts tests/restart-chaos/storage-chaos.test.ts`: 14 pass. - Red before: with the proxy and ST2 changes reverted, the proxy cap test and the ST2 draw test fail. Green after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. ## Risks - Low risk. Test harness only. ST2 now draws one more seeded number, so a seed gives different later windows than before. Replays read the saved receipts and are not affected. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>