## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.
**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.
**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.
**Paperclip version or commit**
`master` at 1b7fe33432.
Refs #20291 (the quiesced case: a save with no active invocation) and
#20233 (the same ownership pattern for local workspace commands).
## What Changed
- `saveRemoteWorkspace` records the controller process stamp of each
invocation in `saveWork.owners` when it registers the invocation. It
removes that entry when the invocation settles. A module-level set marks
invocations that are still running in this process
(`isRemoteWorkspaceSaveInvocationLive`).
- The ownership registry owns a `remote_snapshot_save` with active
invocations by that stamp. A save whose active invocations have no
stamps was written before this change and is owned by its run, as with
finalize attempts. A mix of stamps, or more than one process, is
unprovable.
- `settleAbandonedRemoteSnapshotSave` settles a save with active
invocations once its owner is proven gone:
- Process-owned: the proven stamp must still match every active
invocation, and none may be live in this process.
- Legacy run-owned: the run must be terminal.
- Either way it clears the active invocations, records them as
`abandonedInvocationIds`, and marks the save `failed` with `lastFailure:
remote_workspace_save_abandoned`. The #20291 re-arm of blocked fallbacks
still applies.
- The controller instance lease counts running saves stamped with this
boot as held work, so a long save keeps its owner provably alive.
- The ownership inventory row for this kind describes the stamp.
## Verification
- Red before the fix, green after: `settles a remote snapshot save whose
process died mid-save, and keeps live, foreign or unprovable ones`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
rows. With the fix it settles exactly two:
- a save stamped by a dead process (proof `pid_dead`);
- the wave's shape: an unstamped active invocation on a `process_lost`
run.
It keeps five kinds of save running: a save stamped by this live
process, a stamp from another PID space, a legacy save on a live run, a
save with mixed stamps, and (in the older #20291 test) a stamped active
invocation.
- `stamps each save invocation with the process that runs it and clears
the stamp when it settles` (`remote-workspace-save-concurrency.test.ts`)
pauses a real save mid-flight. It checks the owner stamp and the
in-process live mark, then checks that both are cleared after the save
finishes.
- The reconciler, save-concurrency, remote workspace lifecycle, snapshot
retention, retained-source fallback, preparation gates, and controller
instance lease suites pass (153 tests, 1 environment-gated file
skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Not yet verified: the full Daytona wave replay that shows the queued
retry starts after the save settles. I asked the chaos harness owner to
replay it on this build.
## Risks
- A legacy save (written before this change) with an active invocation
is settled when its run is terminal. The finalize-attempt kind uses the
same rule. During a rolling deploy, an old-release process could still
be running such a save for a terminal run. If that save later succeeds,
its publication still marks the operation `succeeded`.
- If the retry in the wave is also held by the dead writer's dirty
workspace authority, settling the save may not be enough to start it.
The replay will show whether a second change is needed.
- No schema or API change. `saveWork.owners` is new optional metadata.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
