From 95d476c57b295098dc49445461d98c97f28e39c6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 18:21:08 -0600 Subject: [PATCH] docs(plans): record the measured seed experiment for per-turn uploads (#20139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path - #20100 proposed sending only the changed files on per-turn follow-ups, with the repository seed held inside Daytona in a snapshot or a volume. It asked for one measured experiment before building. - Snapshot and volume operations were blocked by our Daytona key's permissions. A restricted key with those permissions now exists, so the experiment has run. - This PR records the numbers and the decision in the design doc. ## Linked Issues or Issue Description Refs #20100 **What happened?** The #20100 design's seed-location choice was unmeasured: snapshot build time and volume read speed were both open questions. **Steps to reproduce** 1. Run the SDK timing of direct upload, volume extract, and snapshot build and create on the paperclip seed (58.7 MB gzip) in Daytona `us`, 2 vCPU. **Expected behavior** The design doc states the measured winner, the per-turn saving, the one-time cost, and the work needed to build it. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`: - The status now reads: measured; snapshot chosen; not built. - New "Experiment results" table: - direct upload: 6.4 s per turn; - volume: 6.8 s per turn, which rules it out; - snapshot: about 2–3.5 s per turn, after a 177 s build per seed generation. - Findings: fork is unsupported, compression is exhausted, and the snapshot builder runs as `daytona` and does not extract archives. - New "Work to build the snapshot seed" list: lifecycle, sandbox creation, verification and fallback, expiry and cleanup, and rebuild policy. ## Verification Docs only. Every Daytona resource the experiment created was deleted, and an account-wide listing confirmed none remained. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...8-incremental-per-turn-workspace-upload.md | 36 ++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md index c1e3a5a485c..7bc6e373c08 100644 --- a/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md +++ b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md @@ -1,6 +1,6 @@ # Per-turn follow-ups without re-uploading the repository -Status: proposed design. The next step is one bounded experiment; no code yet. +Status: measured. Snapshot seed chosen; not built. It is a scoped multi-day feature awaiting a decision (see "Experiment results"). ## Problem @@ -98,6 +98,40 @@ Choose the location whose first-token time is lowest, then build it red/green with a before-and-after measurement on the clone. That needs one Phase 3 harness slot. +## Experiment results (2026-11-08) + +Setup: +- Daytona `us`, 2 vCPU sandboxes, runner image `paperclip-daytona-runner@sha256:4b13c70a…`. +- The seed is a git archive of the paperclip clone: 58.7 MB gzip, 14,766 files. +- Timed with the Daytona SDK from the controller host, using a restricted key with snapshot and volume permissions. +- Every resource was deleted, and an account-wide listing confirmed none remained. + +| Path | Per turn | One-time cost per seed generation | +|---|---|---| +| Direct upload (today) | create 0.97 s + upload 3.97 s + extract 1.43 s = **6.4 s** | none | +| Volume | create 0.86 s + extract from the mounted volume 5.95 s = **6.8 s** | create 6.8 s + writer sandbox 6.2 s + upload 5.3 s, about 18 s | +| Snapshot | create from snapshot 1.83 s; the seed is already in the image (copy 0.26 s; extracting it adds about 1.4 s unless it is extracted at build time) = **about 2–3.5 s** | **build 177 s** | + +Findings: +- **The volume is ruled out.** Reading the archive from a mounted volume is slower than uploading it from the controller. +- **The snapshot wins.** It saves about 3–4 s per remote turn against this run's direct upload. It saves more when the controller link is slow: production follow-up transfers measured 6–16 s with zstd. That is below the 12–13 s estimated in "Goal", because direct upload was fast in this run. +- **Fork is unsupported** for these sandboxes (measured earlier the same day). +- **Transfer compression is exhausted.** The runner image already ships `zstd`, so production transfers are already compressed. #20104 adds a gzip fallback for images without zstd. +- **Snapshot builder facts:** + - It runs as uid 1001 (`daytona`). + - `Image.addLocalFile` copies an archive without extracting it. + - Build steps must write somewhere that user owns. `/opt` is not writable by it, and it cannot delete a root-owned file in `/tmp`. + +## Work to build the snapshot seed + +This is a multi-day feature, not a single PR: + +1. **Seed snapshot lifecycle.** Build a snapshot in the background after a full upload, keyed by repository, base commit, seed digest and the resolved runner image digest. A build takes about 177 s. Never block a turn on it. +2. **Sandbox creation.** The Daytona plugin creates per-turn sandboxes from the seed snapshot, which is layered on the environment's configured runner image. A runner image update changes the key, so it needs a new build; until that build finishes, turns use the full upload on the new image. A sandbox never starts from a snapshot of an older image. +3. **Verification and fallback.** Check the manifest digest before using the seed. On any mismatch or missing snapshot, do today's full upload. Keep the workspace sentinel and `baselineSha256` checks as the final gate. +4. **Expiry and cleanup.** Expire seeds by age, delete them when their environment is deleted, and handle provider snapshot quotas. +5. **Rebuild policy.** A base-commit move needs a new 177 s build; until then the overlay grows. Rebuild when the overlay passes a threshold. + ## Out of scope - Runner and provider session startup (`runner.session.startup`, about 9 s).