Files
PaperClipAI/doc
DottaandClaude Opus 5.5 2ce5f03cd8 fix(workspaces): name remote restore failures and leave a board recovery action (#19207)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A remote task restores its saved workspace into a sandbox before the
agent starts.
> - When that restore fails, the run records only
`remote_workspace_operation_failed: operation=restore exit=1`.
> - The run then fails as `setup_failed`. No retry starts, and a
`blocked` task gets no recovery action.
> - An operator cannot tell which step failed, and nobody owns the next
step.
> - This pull request makes the sandbox scripts name their failed step
in a closed grammar, and it gives a restore failure a bounded retry or a
named board recovery action.
> - The benefit is a visible cause and a clear owner. Saved work stays
protected.

## Linked Issues or Issue Description

Refs #19152 and #19187. Those changes name copy-back (save) failures.
Refs #18852 and #19195. Those changes added the bounded bootstrap retry
that this change reuses.

**What happened?**

A board reply woke an agent on a `blocked` parent task. The run failed
during setup 13 seconds later. The only run event was
`remote_workspace_operation_failed: operation=restore exit=1`. The run
had no workspace operation row. The task stayed `blocked`, and its
recovery actions showed `active: null`. The same opaque family also
appeared for workspace preparation (`exit=1` only).

**Expected behavior**

The run names which restore step failed, without command output or
paths. A failure before provider work uses a bounded retry when a retry
can help. Otherwise, and when the retry budget is spent, the board gets
a named recovery action. The saved version is never changed or
overwritten.

**Steps to reproduce**

1. Save a remote workspace, then start a new run on a different
allocation.
2. Put a folder at the published workspace path in the sandbox before
restore.
3. Observe that the base code records only `operation=restore exit=1`,
then `setup_failed`, with no recovery action for a `blocked` task.

**Paperclip version or commit**

Base: `87dfba6818`.

**Deployment mode**

Legacy runtime with sandbox (Daytona) remote workspace storage.

## What Changed

- The prepare, repository, and restore sandbox scripts track their
current stage. On failure they print `remote_workspace_<op>_failed:
stage=<stage>` plus closed fields: `reason=` (the script's own
snake_case refusal), `archive=` (the archive helper's fixed reason),
`code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git
output), and the existing `exit`/`signal`/`program` fields.
- The server accepts that text only when it matches a strict grammar for
the same operation. Other output is dropped, as before.
- Provider refusals during the restore upload and restore command are
recorded by closed labels, like copy-back.
- A failed saved-workspace restore raises `RemoteWorkspaceRestoreError`
with the server step (`snapshot_lookup`, `snapshot_download`,
`sandbox_restore`). It is retryable only when the owner re-reads the
same writer, epoch, and saved version with no unsaved work, and the
sandbox did not name a refusal reason.
- Setup failure handling classifies a pre-dispatch legacy restore
failure as `remote_workspace_restore_failed`. A retryable failure uses
the existing bounded bootstrap retry. A non-retryable or exhausted
failure records a board-owned recovery action with the redacted cause,
when stranded-work recovery does not scan the task status (for example
`blocked`).
- Stranded-work recovery gets a named notice for this code. Generic
continuation recovery does not grant it a second budget, and legacy
reconciliation does not invent provider actions for it.
- `doc/execution-semantics.md` documents the contract.

## Verification

- Red: the new transport tests failed on the base code with
`remote_workspace_operation_failed: operation=restore exit=1` (and the
same for prepare and repository). The new completion tests failed
because the run became `setup_failed` with no retry and no recovery
action.
- Green: `npx vitest run src/services/remote-workspace-transport.test.ts
src/services/heartbeat/run-completion.test.ts
src/services/remote-workspaces.integration.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery
src/services/remote-workspace-save-concurrency.test.ts
src/services/native-runtime/native-workspace-finalizer-failure.test.ts
src/__tests__/native-workspace-finalization-work.test.ts` passes (563
tests).
- The transport tests use real sandbox commands. They prove an occupied
folder reports `stage=inspect_root reason=occupied_root` and keeps its
files. They prove a wrong manifest reports `stage=extract
archive=manifest_mismatch`. They prove a missing branch reports
`stage=checkout_branch`, and a changed origin reports
`stage=verify_origin`. Forged output with a path is dropped.
- The real-PostgreSQL cold-restore test proves an occupied root gives a
non-retryable `sandbox_restore` failure and keeps the unknown files.
- The completion tests cover retried, exhausted, refused, post-dispatch,
and `in_progress` cases.
- `tsc --noEmit` for the server reports no errors in the changed files.

## Risks

- A retry keeps the same restore path. If the cause is deterministic but
unnamed, the run uses its two retries before the board action. Named
refusals skip the retries.
- The board action is recorded only for statuses that stranded-work
recovery does not scan, and only when no other action is active. This
prevents a second recovery identity for one failure.
- Script stderr grammar is new. Old sandboxes run the script that the
server sends, so no runner update is necessary.
- No migration, deployment, or credential change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with
tool use, code execution, and browser tools for read-only staging
evidence. Context window and reasoning mode are host-managed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:40:01 -06:00
..
…
…
…
…
…