Files
PaperClipAI/server
DottaandClaude Opus 5.5 780f378f60 fix(recovery): settle a local workspace command whose server process died mid-command (#20233)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run prepares its execution workspace with local commands
(`workspace_provision`, `worktree_prepare`, teardown, cleanup). The
server records each command as a `workspace_operations` row.
> - A restart can stop the server while such a command runs. The row
then stays `running` forever, because no process or recovery path owns
it.
> - The restart-chaos harness found this. A K2 drain interrupted a run,
then the server started `bash ./scripts/provision-worktree.sh` for the
run and exited before it finished. The retry succeeded, but the old row
stayed `running`.
> - The ownership reconciler already settles finalize attempts whose
process died. Local command operations had no owner stamp and no
ownership kind.
> - This pull request stamps each local command operation with the
process that runs it and adds a `workspace_command_operation` ownership
kind that fails an abandoned one.
> - The benefit is that a restart never leaves a stuck workspace
operation behind.

## Linked Issues or Issue Description

**What happened?**
In restart-chaos wave seed 9807 (local, R1, faults K1/K2), tasks t03 and
t04 graded R6 with `running_workspace_operations`. A K2 drain
interrupted the run with `server_shutdown_interrupted`. A
`workspace_provision` operation started 0.3 s later in the same process
and never finished. The retry run succeeded. The orphan operation stayed
`running` forever.

**Expected behavior**
When the process that ran a local workspace command is gone, recovery
closes the operation. It does not change the run or its outcome.

**Steps to reproduce**
1. Start a run that provisions a worktree with a provision command.
2. Drain or restart the server while the command runs.
3. After the restart, see that the `workspace_operations` row for the
command stays `running`.

**Paperclip version or commit**
master at d2fe85e57c

## What Changed

- `workspace-operations.ts`: a local command phase (`worktree_prepare`,
`workspace_config_freshness`, `workspace_provision`, `workspace_seed`,
`workspace_runtime_provision`, `workspace_repair`, `workspace_teardown`,
`worktree_cleanup`) without a runtime-control stamp now records
`commandOwner` (the controller process stamp). This process tracks the
operation as live while it runs. Finalize attempts and command
operations share one live-id set.
- `workspace-finalize-attempt-settlement.ts`: the finalize settle is now
a factory. A second instance settles abandoned command operations with
`abandonedCommand` provenance and an activity entry. Both instances skip
a row with a runtime-control stamp.
- `ownership-registry.ts`: new owner-gone-attestable kind
`workspace_command_operation`. The owner is the `commandOwner` process.
A row without a stamp is owned by its run.
- `controller-instance-lease.ts`: a running command stamped with this
boot holds the controller instance lease, like a finalize attempt.

## Verification

- Red before the fix: the two new tests and the kind-list test failed.
Green after it.
- `server/src/__tests__/ownership-reconciliation-report.test.ts` (29
pass): a dead-process `workspace_provision` operation and an unstamped
one with a terminal run are failed. A live, foreign-PID-space,
active-run, unowned or runtime-control operation stays `running`. The
recorder stamps `commandOwner` on a running provision command.
- Related suites pass: `workspace-runtime`,
`workspace-operations-reconciliation`,
`workspace-runtime-control-recovery`, `execution-preparation-phases`,
`controller-instance-lease`, `heartbeat-workspace-finalize-branch`,
`execution-workspace-runtime-control-conflict` (265 tests).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passes.

## Risks

- Low. The settle fails only a row whose stamped process is proven gone,
or an unstamped row whose run is terminal. It compares the owner stamp
in the same transaction, and it never touches an operation that this
process still runs.
- A failed command row is now visible as `failed` with an interruption
message instead of `running`. Run outcomes do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 04:33:20 -06:00
..