docs(plans): per-turn follow-ups without re-uploading the repository (#20100)

## Thinking Path

- Per-turn remote sandboxes stay the default while warm reuse (#20048)
is on hold.
- On the paperclip clone, a per-turn follow-up spends about 13 s in
`workspace_sync.transfer`, though its changes pack in 0.5 s: every fresh
sandbox receives the whole 70 MB (gzip) seed again, at about 5 MB/s from
the controller.
- The coordinator asked for a design that uploads only changed files
into a fresh sandbox, with a full-upload fallback, measured against the
spans. It also asked to build it only if the implementation is bounded
and safe.

## Linked Issues or Issue Description

Refs #20048

**What happened?**
Per-turn follow-ups on large repositories wait about 13 s re-uploading
the full workspace seed even when almost nothing changed (pack 0.54 s,
transfer 13.3 s, time to first token about 31 s).

**Steps to reproduce**
1. Run `tests/restart-chaos/ttft-baseline.ts --env daytona
--workspace-repo <paperclip clone>` with per-turn sandboxes.
2. Compare `workspace_sync.pack` and `workspace_sync.transfer` on the
follow-up turn.

**Expected behavior**
A follow-up uploads only its overlay into a sandbox that already holds
the seed, falling back to the full upload when unsure.

## What Changed

- `doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md`. It
contains:
- the measured span table and the size analysis (220 MB archive, 70 MB
gzip, about 5 MB/s);
- the expected saving: about 12–13 s per follow-up, time to first token
about 31 s → about 18 s;
- a design keyed by seed generation with a content-hashed manifest,
using a Daytona snapshot (preferred) or a Daytona volume to hold the
seed;
  - the overlay-only turn flow, with a verified fallback;
  - one bounded experiment to choose the seed location before building.
- Not built yet: both seed locations need provider capabilities
(snapshot build time, volume read speed) that must be measured first, so
the implementation isn't bounded until the experiment picks one.

## Verification

Docs only.

## Risks

None at runtime.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/plans/2026-11-08-incremental-per-turn-workspace-upload.md

@@ -0,0 +1,105 @@
+# Per-turn follow-ups without re-uploading the repository
+
+Status: proposed design. The next step is one bounded experiment; no code yet.
+
+## Problem
+
+Per-turn remote sandboxes are the default (warm reuse, #20048, is on hold).
+Each follow-up turn gets a fresh Daytona sandbox, and the controller uploads the
+workspace into it again.
+
+Measured on the paperclip repository clone (Daytona, Codex, per-turn), from the
+run log's `workspace_sync.*` spans (baseline `ws-daytona-phases2`):
+
+| Span | Cold turn | Follow-up turn |
+|---|---|---|
+| `workspace_sync.pack` (changed files) | 4.1 s | **0.54 s** |
+| `workspace_sync.transfer` | 13.6 s | **13.3 s** |
+| `workspace_sync.stage_workspace` | 17.8 s | 13.8 s |
+| `runner.session.startup` | 5.5 s | 9.1 s |
+| time to first token | 33.0 s | 31.2 s |
+
+The follow-up's changes pack in half a second, but the transfer barely shrinks.
+Each fresh sandbox receives the whole seed: the repository archive is 220 MB
+(70 MB with gzip, 14,769 files), and 70 MB in about 13 s is about 5 MB/s from
+the controller host to Daytona. Earlier per-turn runs measured 13–22 s per turn
+for this transfer.
+
+The host side already avoids rebuilding the seed: the workspace seed cache
+(`workspace.seed_lookup`, `seed_replay`, overlay archive, deleted-file
+manifest) packs only the overlay against a cached seed generation. The cost is
+re-sending that seed over the controller's link on every turn.
+
+## Goal
+
+A follow-up turn on a fresh per-turn sandbox sends only the overlay (the
+changed files and a deleted-file manifest against a content-hashed seed
+manifest), and gets the seed from somewhere already inside the provider's
+network. When anything is uncertain, it falls back to today's full upload.
+
+Expected saving: about 12–13 s of the 13.3 s transfer per follow-up turn
+(overlay upload under 1 s for typical turns), taking per-turn follow-up time to
+first token from about 31 s to about 18 s on this repository. The remaining
+large item is `runner.session.startup` (about 9 s: runner boot plus provider
+session resume), which this design does not change.
+
+## Design
+
+**Seed identity.** The workspace seed cache already names a seed generation.
+Key the provider-side seed by `(repository, base commit, seed generation
+digest)`, with a content-hashed manifest (path → sha256, mode, size) stored next
+to the seed.
+
+**Where the seed lives, one of:**
+
+1. **A Daytona snapshot per seed (preferred if it is fast to create).** The
+   Daytona plugin already creates sandboxes from a named snapshot
+   (`config.snapshot`). Build a snapshot whose image layers the seed over the
+   runner image, once per seed key, and create follow-up sandboxes from it.
+   - **Open question:** time to build a snapshot from a 70 MB context, and
+     whether snapshot creation can run in the background after the first turn.
+2. **A Daytona volume per seed.** Upload the seed archive to a volume once, then
+   mount it in each per-turn sandbox and extract inside the sandbox.
+   - **Open question:** volume read and extract time for 220 MB, which must stay
+     well under 13 s to be worth it.
+
+**Turn flow.**
+1. Look up the seed key on the provider (snapshot or volume present and
+   verified against its manifest digest).
+2. If present: create the sandbox from the snapshot (or mount the volume and
+   extract), then upload only the overlay through the existing `syncIn` path
+   (files plus the deleted-file manifest), and verify the result with the
+   existing workspace sentinel and `baselineSha256`.
+3. If absent, unverifiable, or the overlay exceeds a threshold (for example
+   20% of seed bytes, because the base moved far): do today's full upload, and
+   build or refresh the provider seed in the background for the next turn.
+
+**Safety.**
+- The seed is read-only, keyed by content. A mismatch in the manifest digest
+  falls back to the full upload, never to a partial workspace.
+- The existing post-upload verification (sentinel, baseline hash) stays the
+  final check before `turn.start`.
+- Provider seeds are cleaned up with the environment and expire by age.
+- Credentials never enter the seed: it is built from the same filtered
+  workspace as today's upload.
+
+## Next step: one bounded experiment
+
+Before building anything, measure both seed locations with the existing harness
+(`tests/restart-chaos/ttft-baseline.ts --env daytona --workspace-repo <paperclip clone>`):
+
+1. Time to create a Daytona snapshot from the paperclip seed, and time to create
+   a sandbox from it.
+2. Time to mount a volume holding the seed archive and extract it inside a
+   sandbox.
+3. Overlay upload time for a typical follow-up (pack output 0.5 s).
+
+Choose the location whose first-token time is lowest, then build it red/green
+with a before-and-after measurement on the clone. That needs one Phase 3 harness
+slot.
+
+## Out of scope
+
+- Runner and provider session startup (`runner.session.startup`, about 9 s).
+  Keeping the runner alive is the warm path, which stays on hold.
+- Local environments; they have no upload.