diff --git a/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md index c1e3a5a485c..7bc6e373c08 100644 --- a/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md +++ b/doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md @@ -1,6 +1,6 @@ # Per-turn follow-ups without re-uploading the repository -Status: proposed design. The next step is one bounded experiment; no code yet. +Status: measured. Snapshot seed chosen; not built. It is a scoped multi-day feature awaiting a decision (see "Experiment results"). ## Problem @@ -98,6 +98,40 @@ Choose the location whose first-token time is lowest, then build it red/green with a before-and-after measurement on the clone. That needs one Phase 3 harness slot. +## Experiment results (2026-11-08) + +Setup: +- Daytona `us`, 2 vCPU sandboxes, runner image `paperclip-daytona-runner@sha256:4b13c70a…`. +- The seed is a git archive of the paperclip clone: 58.7 MB gzip, 14,766 files. +- Timed with the Daytona SDK from the controller host, using a restricted key with snapshot and volume permissions. +- Every resource was deleted, and an account-wide listing confirmed none remained. + +| Path | Per turn | One-time cost per seed generation | +|---|---|---| +| Direct upload (today) | create 0.97 s + upload 3.97 s + extract 1.43 s = **6.4 s** | none | +| Volume | create 0.86 s + extract from the mounted volume 5.95 s = **6.8 s** | create 6.8 s + writer sandbox 6.2 s + upload 5.3 s, about 18 s | +| Snapshot | create from snapshot 1.83 s; the seed is already in the image (copy 0.26 s; extracting it adds about 1.4 s unless it is extracted at build time) = **about 2–3.5 s** | **build 177 s** | + +Findings: +- **The volume is ruled out.** Reading the archive from a mounted volume is slower than uploading it from the controller. +- **The snapshot wins.** It saves about 3–4 s per remote turn against this run's direct upload. It saves more when the controller link is slow: production follow-up transfers measured 6–16 s with zstd. That is below the 12–13 s estimated in "Goal", because direct upload was fast in this run. +- **Fork is unsupported** for these sandboxes (measured earlier the same day). +- **Transfer compression is exhausted.** The runner image already ships `zstd`, so production transfers are already compressed. #20104 adds a gzip fallback for images without zstd. +- **Snapshot builder facts:** + - It runs as uid 1001 (`daytona`). + - `Image.addLocalFile` copies an archive without extracting it. + - Build steps must write somewhere that user owns. `/opt` is not writable by it, and it cannot delete a root-owned file in `/tmp`. + +## Work to build the snapshot seed + +This is a multi-day feature, not a single PR: + +1. **Seed snapshot lifecycle.** Build a snapshot in the background after a full upload, keyed by repository, base commit, seed digest and the resolved runner image digest. A build takes about 177 s. Never block a turn on it. +2. **Sandbox creation.** The Daytona plugin creates per-turn sandboxes from the seed snapshot, which is layered on the environment's configured runner image. A runner image update changes the key, so it needs a new build; until that build finishes, turns use the full upload on the new image. A sandbox never starts from a snapshot of an older image. +3. **Verification and fallback.** Check the manifest digest before using the seed. On any mismatch or missing snapshot, do today's full upload. Keep the workspace sentinel and `baselineSha256` checks as the final gate. +4. **Expiry and cleanup.** Expire seeds by age, delete them when their environment is deleted, and handle provider snapshot quotas. +5. **Rebuild policy.** A base-commit move needs a new 177 s build; until then the overlay grows. Rebuild when the overlay passes a threshold. + ## Out of scope - Runner and provider session startup (`runner.session.startup`, about 9 s).