mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
DottaandPaperclip
2a6aae3963
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server saves portable workspace archives before it moves file authority to object storage. > - Each capture checks and hashes every eligible file before and after archive creation. > - Those stable reads run in series and leave file I/O idle on workspaces with many files. > - This pull request overlaps at most four stable reads across the whole directory walk. > - It preserves entry order, source checks, Git checks, and archive bytes. > - The benefit is shorter archive capture with bounded extra I/O concurrency. ## Linked Issues or Issue Description **What existing behavior does this improve?** Portable remote workspace archive capture in the server. Related: #19396 preserves saved-upload recovery. #17132 applies a similar technique to the separate Runner file-diff snapshot path; this PR changes the server archive owner only. **Current behavior** The archive walker waits for each file's stable read and hash before visiting the next file. A local packed-Git fixture with 6,000 files and 52.4 MB of contents took a median 4.640 seconds in the final two serial samples. **Proposed behavior** Keep up to four stable reads active across all nested directories. Reserve each entry's position before reading it. Await all reads before returning success, failure, or cancellation. The final two treatment samples took a median 3.439 seconds, a 25.9% reduction in capture time. **Reason and benefit** Captures run twice during a fresh workspace migration. Bounded I/O overlap reduces time spent in this owner without changing the archive contract. This measurement does not establish an end-to-end migration speedup. **Breaking changes** None intended. The file format, manifest order, source validation, Git validation, API, and schema stay the same. ## What Changed - Share one four-read limit across the recursive walk. Preserve deterministic entry positions when reads finish out of order. - Drain active reads on every exit before callers can remove scratch files or return control to a source writer. - Check cancellation before creating a file read stream. - Exercise real file handles across nested directories, out-of-order completion, source replacement, cancellation, and serialized archive capture. ## Verification - Focused archive, archive-environment, and snapshot-storage suites: 113 tests passed. The first sandbox run could not bind local test sockets; the same suites passed with local listener access. - Full workspace typecheck passed before rebasing. The rebase changed neither touched file. - Full local `pnpm test:run` stopped at an unchanged chat integration test: `resolves ambiguous provider effects explicitly and never lets terminal failures starve reconciliation`, with `socket hang up`. The same case passed once in isolation on the rebased head. Its original cause remains unknown; this is not a full-suite pass. - Full `pnpm build` passed on the rebased head. Current-head CI is pending. - Benchmark: actual owning sources, Node 26.4 on macOS arm64, one importer, the same packed-Git fixture. All 12 trial archives had identical manifest digest, SHA-256, and 72,908,826 bytes. Final paired medians were 4.640 seconds serial and 3.439 seconds with four reads. Total process plus Git CPU increased about 12%. - An initial pair was slower with concurrency (23.352 seconds versus 11.228 seconds); the cause remains unknown. Later paired reductions ranged from 24% to 37%. The benchmark host was shared. Linux Node 24 and eight-importer throughput are not yet qualified. - No user commands, configuration, or document contracts changed. Code comments explain ordering and cleanup. ## Risks - Four simultaneous reads can increase I/O and CPU contention when several captures run together. The bound is per capture. Hosted throughput needs measurement after the ordinary release. - A shared-host local benchmark has limited predictive value. This PR claims a measured capture improvement on that fixture only. - Rollback is a code revert. No migration or stored-data rewrite is required. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated review. The runtime did not expose an exact deployment variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>