diff --git a/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md new file mode 100644 index 00000000000..c1e3a5a485c --- /dev/null +++ b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md @@ -0,0 +1,105 @@ +# Per-turn follow-ups without re-uploading the repository + +Status: proposed design. The next step is one bounded experiment; no code yet. + +## Problem + +Per-turn remote sandboxes are the default (warm reuse, #20048, is on hold). +Each follow-up turn gets a fresh Daytona sandbox, and the controller uploads the +workspace into it again. + +Measured on the paperclip repository clone (Daytona, Codex, per-turn), from the +run log's `workspace_sync.*` spans (baseline `ws-daytona-phases2`): + +| Span | Cold turn | Follow-up turn | +|---|---|---| +| `workspace_sync.pack` (changed files) | 4.1 s | **0.54 s** | +| `workspace_sync.transfer` | 13.6 s | **13.3 s** | +| `workspace_sync.stage_workspace` | 17.8 s | 13.8 s | +| `runner.session.startup` | 5.5 s | 9.1 s | +| time to first token | 33.0 s | 31.2 s | + +The follow-up's changes pack in half a second, but the transfer barely shrinks. +Each fresh sandbox receives the whole seed: the repository archive is 220 MB +(70 MB with gzip, 14,769 files), and 70 MB in about 13 s is about 5 MB/s from +the controller host to Daytona. Earlier per-turn runs measured 13–22 s per turn +for this transfer. + +The host side already avoids rebuilding the seed: the workspace seed cache +(`workspace.seed_lookup`, `seed_replay`, overlay archive, deleted-file +manifest) packs only the overlay against a cached seed generation. The cost is +re-sending that seed over the controller's link on every turn. + +## Goal + +A follow-up turn on a fresh per-turn sandbox sends only the overlay (the +changed files and a deleted-file manifest against a content-hashed seed +manifest), and gets the seed from somewhere already inside the provider's +network. When anything is uncertain, it falls back to today's full upload. + +Expected saving: about 12–13 s of the 13.3 s transfer per follow-up turn +(overlay upload under 1 s for typical turns), taking per-turn follow-up time to +first token from about 31 s to about 18 s on this repository. The remaining +large item is `runner.session.startup` (about 9 s: runner boot plus provider +session resume), which this design does not change. + +## Design + +**Seed identity.** The workspace seed cache already names a seed generation. +Key the provider-side seed by `(repository, base commit, seed generation +digest)`, with a content-hashed manifest (path → sha256, mode, size) stored next +to the seed. + +**Where the seed lives, one of:** + +1. **A Daytona snapshot per seed (preferred if it is fast to create).** The + Daytona plugin already creates sandboxes from a named snapshot + (`config.snapshot`). Build a snapshot whose image layers the seed over the + runner image, once per seed key, and create follow-up sandboxes from it. + - **Open question:** time to build a snapshot from a 70 MB context, and + whether snapshot creation can run in the background after the first turn. +2. **A Daytona volume per seed.** Upload the seed archive to a volume once, then + mount it in each per-turn sandbox and extract inside the sandbox. + - **Open question:** volume read and extract time for 220 MB, which must stay + well under 13 s to be worth it. + +**Turn flow.** +1. Look up the seed key on the provider (snapshot or volume present and + verified against its manifest digest). +2. If present: create the sandbox from the snapshot (or mount the volume and + extract), then upload only the overlay through the existing `syncIn` path + (files plus the deleted-file manifest), and verify the result with the + existing workspace sentinel and `baselineSha256`. +3. If absent, unverifiable, or the overlay exceeds a threshold (for example + 20% of seed bytes, because the base moved far): do today's full upload, and + build or refresh the provider seed in the background for the next turn. + +**Safety.** +- The seed is read-only, keyed by content. A mismatch in the manifest digest + falls back to the full upload, never to a partial workspace. +- The existing post-upload verification (sentinel, baseline hash) stays the + final check before `turn.start`. +- Provider seeds are cleaned up with the environment and expire by age. +- Credentials never enter the seed: it is built from the same filtered + workspace as today's upload. + +## Next step: one bounded experiment + +Before building anything, measure both seed locations with the existing harness +(`tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo `): + +1. Time to create a Daytona snapshot from the paperclip seed, and time to create + a sandbox from it. +2. Time to mount a volume holding the seed archive and extract it inside a + sandbox. +3. Overlay upload time for a typical follow-up (pack output 0.5 s). + +Choose the location whose first-token time is lowest, then build it red/green +with a before-and-after measurement on the clone. That needs one Phase 3 harness +slot. + +## Out of scope + +- Runner and provider session startup (`runner.session.startup`, about 9 s). + Keeping the runner alive is the warm path, which stays on hold. +- Local environments; they have no upload.