mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
DottaandClaude Opus 5.5
ffd9591ab4
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run writes its task files in a remote sandbox. The task
workspace records that run as its writer until the files are saved.
> - When a user stops or reassigns a run, the cancelled-workspace
preservation worker records the unsaved files as file recovery work.
> - A warm run (`keep_running`) that is not successful releases its
reusable sandbox through stop-and-retain. That release can finish before
the worker sees a terminal coordinator.
> - The worker accepted only an active lease or a released ephemeral
lease. It ignored the released reusable lease, so it never recorded the
file recovery.
> - The workspace kept the ended writer's hold. Each successor run
waited, then failed with "The workspace still has an unfinished writer
or save operation" after the one-hour cutoff.
> - This pull request accepts a released reusable lease that has a
confirmed stop-and-retain receipt as a preservation source.
> - The benefit is that the stopped run's files become recovery work,
and the task is not stranded behind a hold that nothing settles.
## Linked Issues or Issue Description
Refs #18820 (preserves files when Stop finds no live controller). Refs
#17677 (schedules cancelled workspace preservation). Refs #19285 (parks
a workspace wait behind an ended writer).
**What happened?**
A board user reassigned a task during a warm native run. Stop cancelled
the run with `issue_reassigned`, and the Stop acknowledgement was
recorded. The executor settled the coordinator to `terminal_failure`
with no owner. About eight seconds later, terminal release stopped and
retained the reusable Daytona sandbox. The lease became `released` with
`leasePolicy: reuse_by_environment`, a `stopped` termination receipt and
a stop-and-retain receipt. The task workspace stayed `dirty` with
`writerRunId` set to the cancelled run. No file recovery was recorded.
The new assignee's runs waited as `workspace_busy` for one hour. Then
each run failed `setup_failed`, and the task became blocked.
The heartbeat run row showed `nativePhase: "observed"`. That value is
stale: the checkpoint persist wrote it during teardown. The coordinator
itself was `terminal_failure`: the run's execution projection was
`recovery_needed` with no matching recovery action.
**Expected behavior**
The preservation worker records the stopped run's prepared files as file
recovery work. The lease keeps its release status and receipts. No
second stop and no new provider turn start.
**Steps to reproduce**
1. Cancel a native run with an acknowledged run-only Stop. Its remote
workspace reference is `prepared`, and the coordinator is
`terminal_failure` with no owner.
2. Release its reusable lease through stop-and-retain (`released`,
`reuse_by_environment`, confirmed stop receipt).
3. Run the cancelled-workspace preservation worker or the startup scan.
4. On master, the run has no `workspaceRestoreRecovery`, and no recovery
action exists.
## What Changed
-
`server/src/services/native-runtime/native-failed-workspace-recovery.ts`:
add one source check, `cancelledWorkspaceSource`, for cancelled
preservation. The worker check and the startup scan both use it. It
keeps the active and ephemeral cases. It adds a released reusable lease
only when the stop-and-retain receipt is confirmed
(`hasConfirmedSandboxStopAndRetain`) and `preparedRemoteWorkspaceSource`
proves the exact remote-workspace binding. This is the same proof the
restore recorder uses. Native-sync runs have no such proof and stay
ineligible. An ineligible source causes no run write or notification.
- The restore recorder (`preserveLegacyWorkspaceRestoreSources`) already
accepts this source. It checks the confirmed receipt and the exact
prepared workspace binding, and it keeps the lease status, policy and
receipts. This pull request does not change it.
- `native-cancelled-workspace-preservation.test.ts`: add the incident
shape as a regression for the worker and the startup scan, and add
safety cases.
- `doc/execution-semantics.md`: document the stopped reusable source and
the writer hold until recovery.
## Verification
From `server`:
```sh
npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts
```
- Red before the fix: the two new regression cases failed (`expected {
nativeCancellation: … } to match object { …, workspaceRestoreRecovery: …
}`). 30 other cases passed.
- Green after the fix: 36 passed.
- Review follow-up: the cases `invalid-receipt`, `expired-lease` and
`native-sync` failed on the previous head 1d461b4605. The scan or the
worker rewrote the run row. These cases pass now. Each case runs the
worker and two scans, then asserts that the run row and the lease row
are unchanged.
- Recovery-time fencing: after preservation,
`inspectWorkspaceRecoverySource` rejects with `workspace_writer_active`
while a task run is running. It rejects with
`allocation_ownership_ambiguous` when a later lease shares the
allocation.
- The regression checks: recovery recorded with the lease ID, run still
`cancelled` with `issue_reassigned`, lease row unchanged (status, policy
and receipts), one recovery action, recovery source inspection finds one
source, the worker disarms, and a repeated pass records nothing again
and creates no run or wake.
- Safety cases, all with no recovery record and unchanged run and lease
rows: a live coordinator owner, a `pending` (not acknowledged) Stop, a
lease without a stop receipt, an invalid receipt, an expired lease, a
native-sync run, and a `running` run.
- Affected suites were run locally: `native-failed-workspace-recovery`,
`native-workspace-reconstruction`, `workspace-recovery`,
`remote-workspaces.integration`, `native-environment-settlement-work`,
`server-startup-feedback-export`, `heartbeat/run-control`,
`heartbeat-remote-workspace-busy`. On a heavily loaded machine, some
cases timed out at 30 s. A rerun of those two files passed 135/135
tests; one file still reported a 30 s hook timeout. CI runs these
suites.
- `pnpm --filter @paperclipai/server typecheck` passes.
- Not done: a live staging run. The fix is not deployed.
## Risks
- Low. The new source applies only to a cancelled native run that has an
acknowledged run-only Stop, a terminal coordinator with no owner and no
accepted result, a prepared workspace reference, and a confirmed
stop-and-retain receipt for the same run, lease and allocation.
- Preservation records recovery work only. It does not stop the sandbox
again, change the lease policy, or grant execute rights. Recovery later
checks live owners and retired predecessors before it uses the sandbox.
- The writer hold on the workspace does not change. Successor runs still
wait until recovery saves the files, so unsaved files are not
overwritten. The one-hour cutoff stays.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only browser reads of
staging diagnostics).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>