mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
DottaandPaperclip
6a932bb979
## Thinking Path > - Paperclip manages AI agents and preserves their work when execution stops. > - Remote workspace recovery must save the retained files before another executor can start. > - Daytona directory export first writes a complete tar on the source sandbox. > - When that disk is full, each preservation retry fails while writing the same temporary archive. > - This change streams an archive after a storage-capacity failure, without another full copy on the sandbox. > - Existing source ownership, archive validation, and task admission gates remain in force. ## Linked Issues or Issue Description Related: #14402 established safe workspace export and retained-source recovery. This PR handles the remaining temporary-archive storage requirement. **What happened?** A sandbox ran out of disk space. The provider could no longer save its state, and workspace preservation then failed with tar `No space left on device` or `Wrote only ... bytes`. Repeating the export could not resolve that failure because it required free space on the same disk. **Expected behavior** Export retained files without needing a second full copy on the source disk. Reject incomplete transfers and unsafe archives. Preserve the source and newer host files if export fails. **Steps to reproduce** Run the capacity regression with real tar and a file-size limit. The original export fails during archive creation. The fixed export transfers the same bytes through a real PTY. The credential-gated live test uses a real Daytona sandbox and `/dev/full` as the archive sink to reproduce ENOSPC without filling a provider disk. **Paperclip version or commit** Reproduced on `27be859044`; fix rebased onto `6d59f5ef7e`. **Deployment mode** Source checkout on macOS arm64; live provider verification on a disposable Ubuntu 24.04 Daytona sandbox with SDK 0.203.0. ## What Changed - Fall back only after archive storage-capacity errors. Keep the ordinary bulk download path. - Transfer 40 KiB archive chunks over a raw PTY. Wait for each host write acknowledgement and a final receipt acknowledgement. Send long scripts only after raw-mode readiness. - Require successful tar exit and complete framing before the existing host archive validator can import files. Keep the bounded safe-link retry and exclude rules. - Bound total archive bytes by the provider-reported disk allocation, preserve a 1 GiB host free-space reserve, and bound terminal cleanup to five seconds while closing the host file independently. - Add real-tar/PTY regressions, stream failure tests, a credential-gated provider test, and recovery documentation. ## Verification - Red before production changes: the capacity regression failed on the real tar write with exit 153. Green afterward: unchanged bytes, file mode, confined links, and source files survive. A tar producer that fails after sending bytes cannot overwrite newer host files. - 43 focused tests pass. Missing receipts, nonzero exits, transport loss, invalid base64, oversized frames, stalled transfers, and late connection cleanup fail closed. - Live Daytona test passes: stop/start, actual tar ENOSPC, matching file hashes and modes, unsafe-link filtering, intact source, then another stop/start and export with a later saved file. The disposable sandbox was deleted. Initial default-sandbox creation hit account quota; the successful test used a 1 GiB image. Large output bursts exposed a real socket closure, corrected by bounded chunks and verified in the successful run. - Repository `pnpm -r typecheck` and `pnpm build` pass. `node scripts/check-maps.mjs` passes. Standalone plugin typecheck passes. - Standalone plugin build and final typecheck pass. The final broad plugin run passed 482 tests, skipped seven credential-gated cases, and hit one existing 30-second archive-listing timeout. Earlier runs hit both large-listing cases. Both timeouts reproduce with unchanged `file-sync.ts` from master `6d59f5ef7e`, with 22 other archive-listing cases passing on both base and fixed source. Parallel, serial, and GNU-tar local runs retain those timeouts. The serial local `pnpm test:run` was stopped after 36 minutes once the complete current-head CI test suite passed; it is not counted as a full local pass. GitHub Actions on production-code head `9ffbd9e56e283a01c22f005c022b8f6838cf0143` passed typecheck, all general and serialized test shards, Runner verification, build, E2E, and canary dry run. - Review fixes have passed the live Daytona test again. Sparse-file budget enforcement, rejection before host writes, low host reserve, and stalled provider cleanup are covered by the 43 focused tests. Greptile is 5/5 and security scanning passes on `9ffbd9e56e283a01c22f005c022b8f6838cf0143`; both review threads are resolved. - Test portability: `d2280108f5` verifies GNU tool versions and Python before running the real-PTY regression. All 43 focused tests pass with prerequisites installed; restricting PATH to the macOS system tools cleanly skips the three integration cases. Standalone plugin typecheck passes. Current-head CI passes typecheck, build, all test shards, Runner verification, E2E, and canary dry run: https://github.com/paperclipai/paperclip/actions/runs/40910736373. Greptile and Paperclip Review both assess the final head 5/5, security checks pass, and no review threads remain unresolved. - Paperclip Review assessed the final head 5/5 with no actionable defects after the test-portability fix. Remaining accepted tradeoffs are recorded under Risks. - This is live provider-boundary acceptance. It does not prove a full task Retry journey, repair lost provider session state, or recover the original deployed task. ## Risks - Streaming requires Bash and GNU tar, dd, and base64. Missing tools or transfer failure reject export and retain the recovery obligation. - Acknowledged terminal transfer is slower than bulk download, especially across high-latency links. Very large exports can still reach the configured deadline. The ordinary bulk path is unchanged. - The fallback checks host free space before every 40 KiB write; this deliberately favors a strict reserve check over reducing local syscalls. It carries archive-download error context but does not yet emit a dedicated transfer span. Provider cleanup is bounded; a kernel-stalled local filesystem operation can still delay completion and file close. - This change does not increase sandbox capacity or reconstruct provider state that failed to persist. The plugin must reach the target installation before recovery can use it. No merge, deployment, or original-task replay is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell execution, and automated testing. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>