From 2a16be36abaf2b67ccad528545aa11a52f91bf47 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 14:21:24 -0600 Subject: [PATCH] docs(plans): per-turn follow-ups without re-uploading the repository (#20100) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path - Per-turn remote sandboxes stay the default while warm reuse (#20048) is on hold. - On the paperclip clone, a per-turn follow-up spends about 13 s in `workspace_sync.transfer`, though its changes pack in 0.5 s: every fresh sandbox receives the whole 70 MB (gzip) seed again, at about 5 MB/s from the controller. - The coordinator asked for a design that uploads only changed files into a fresh sandbox, with a full-upload fallback, measured against the spans. It also asked to build it only if the implementation is bounded and safe. ## Linked Issues or Issue Description Refs #20048 **What happened?** Per-turn follow-ups on large repositories wait about 13 s re-uploading the full workspace seed even when almost nothing changed (pack 0.54 s, transfer 13.3 s, time to first token about 31 s). **Steps to reproduce** 1. Run `tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo ` with per-turn sandboxes. 2. Compare `workspace_sync.pack` and `workspace_sync.transfer` on the follow-up turn. **Expected behavior** A follow-up uploads only its overlay into a sandbox that already holds the seed, falling back to the full upload when unsure. ## What Changed - `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`. It contains: - the measured span table and the size analysis (220 MB archive, 70 MB gzip, about 5 MB/s); - the expected saving: about 12–13 s per follow-up, time to first token about 31 s → about 18 s; - a design keyed by seed generation with a content-hashed manifest, using a Daytona snapshot (preferred) or a Daytona volume to hold the seed; - the overlay-only turn flow, with a verified fallback; - one bounded experiment to choose the seed location before building. - Not built yet: both seed locations need provider capabilities (snapshot build time, volume read speed) that must be measured first, so the implementation isn't bounded until the experiment picks one. ## Verification Docs only. ## Risks None at runtime. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- ...8-incremental-per-turn-workspace-upload.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md diff --git a/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md new file mode 100644 index 00000000000..c1e3a5a485c --- /dev/null +++ b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md @@ -0,0 +1,105 @@ +# Per-turn follow-ups without re-uploading the repository + +Status: proposed design. The next step is one bounded experiment; no code yet. + +## Problem + +Per-turn remote sandboxes are the default (warm reuse, #20048, is on hold). +Each follow-up turn gets a fresh Daytona sandbox, and the controller uploads the +workspace into it again. + +Measured on the paperclip repository clone (Daytona, Codex, per-turn), from the +run log's `workspace_sync.*` spans (baseline `ws-daytona-phases2`): + +| Span | Cold turn | Follow-up turn | +|---|---|---| +| `workspace_sync.pack` (changed files) | 4.1 s | **0.54 s** | +| `workspace_sync.transfer` | 13.6 s | **13.3 s** | +| `workspace_sync.stage_workspace` | 17.8 s | 13.8 s | +| `runner.session.startup` | 5.5 s | 9.1 s | +| time to first token | 33.0 s | 31.2 s | + +The follow-up's changes pack in half a second, but the transfer barely shrinks. +Each fresh sandbox receives the whole seed: the repository archive is 220 MB +(70 MB with gzip, 14,769 files), and 70 MB in about 13 s is about 5 MB/s from +the controller host to Daytona. Earlier per-turn runs measured 13–22 s per turn +for this transfer. + +The host side already avoids rebuilding the seed: the workspace seed cache +(`workspace.seed_lookup`, `seed_replay`, overlay archive, deleted-file +manifest) packs only the overlay against a cached seed generation. The cost is +re-sending that seed over the controller's link on every turn. + +## Goal + +A follow-up turn on a fresh per-turn sandbox sends only the overlay (the +changed files and a deleted-file manifest against a content-hashed seed +manifest), and gets the seed from somewhere already inside the provider's +network. When anything is uncertain, it falls back to today's full upload. + +Expected saving: about 12–13 s of the 13.3 s transfer per follow-up turn +(overlay upload under 1 s for typical turns), taking per-turn follow-up time to +first token from about 31 s to about 18 s on this repository. The remaining +large item is `runner.session.startup` (about 9 s: runner boot plus provider +session resume), which this design does not change. + +## Design + +**Seed identity.** The workspace seed cache already names a seed generation. +Key the provider-side seed by `(repository, base commit, seed generation +digest)`, with a content-hashed manifest (path → sha256, mode, size) stored next +to the seed. + +**Where the seed lives, one of:** + +1. **A Daytona snapshot per seed (preferred if it is fast to create).** The + Daytona plugin already creates sandboxes from a named snapshot + (`config.snapshot`). Build a snapshot whose image layers the seed over the + runner image, once per seed key, and create follow-up sandboxes from it. + - **Open question:** time to build a snapshot from a 70 MB context, and + whether snapshot creation can run in the background after the first turn. +2. **A Daytona volume per seed.** Upload the seed archive to a volume once, then + mount it in each per-turn sandbox and extract inside the sandbox. + - **Open question:** volume read and extract time for 220 MB, which must stay + well under 13 s to be worth it. + +**Turn flow.** +1. Look up the seed key on the provider (snapshot or volume present and + verified against its manifest digest). +2. If present: create the sandbox from the snapshot (or mount the volume and + extract), then upload only the overlay through the existing `syncIn` path + (files plus the deleted-file manifest), and verify the result with the + existing workspace sentinel and `baselineSha256`. +3. If absent, unverifiable, or the overlay exceeds a threshold (for example + 20% of seed bytes, because the base moved far): do today's full upload, and + build or refresh the provider seed in the background for the next turn. + +**Safety.** +- The seed is read-only, keyed by content. A mismatch in the manifest digest + falls back to the full upload, never to a partial workspace. +- The existing post-upload verification (sentinel, baseline hash) stays the + final check before `turn.start`. +- Provider seeds are cleaned up with the environment and expire by age. +- Credentials never enter the seed: it is built from the same filtered + workspace as today's upload. + +## Next step: one bounded experiment + +Before building anything, measure both seed locations with the existing harness +(`tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo `): + +1. Time to create a Daytona snapshot from the paperclip seed, and time to create + a sandbox from it. +2. Time to mount a volume holding the seed archive and extract it inside a + sandbox. +3. Overlay upload time for a typical follow-up (pack output 0.5 s). + +Choose the location whose first-token time is lowest, then build it red/green +with a before-and-after measurement on the clone. That needs one Phase 3 harness +slot. + +## Out of scope + +- Runner and provider session startup (`runner.session.startup`, about 9 s). + Keeping the runner alive is the warm path, which stays on hold. +- Local environments; they have no upload.