0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-09 16:02:36 +01:00

Files

T

DottaandClaude Opus 5.5 f09b4704b6 fix(recovery): keep the agent's work when Retry continues after exhausted or blocked recovery (#19666 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task recovery restarts agent work after a control-plane restart or a
failed run
> - When automatic recovery runs out or is blocked, the user clicks
Retry
> - Retry then started a fresh isolated attempt cloned from committed
HEAD, so the agent lost its uncommitted files, its task branch and the
worktree Git link
> - A restart must never lose the agent's work
> - This pull request makes Retry keep the task's own workspace when
that is safe, and tells the agent where the old files are when a fresh
attempt is still required
> - The benefit is that a Retry continues the real work instead of
starting over

## Linked Issues or Issue Description

**What happened?**
After `native_restart_recovery_blocked` or an exhausted recovery
generation, a Retry ran in
`isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That
directory was a detached clone of the base ref. It did not contain the
uncommitted edits from the issue's git worktree. It was not on the task
branch. The agent was not told where the old files were. A restart-chaos
wave graded this as an R2 failure (`pre_fault_files_lost`). A second
defect: when a restart hit the bootstrap of a fresh attempt, preparing
the same run again failed with
`isolated_task_directory_binding_mismatch`, so the Retry could not
finish.

**Expected behavior**
Retry continues in the task's own workspace with its edits and `.git`
intact when no other writer can hold it. When a fresh attempt is really
needed, the old directory stays as it was and the agent gets its path
and its changed files. Preparing the same attempt again after a restart
is idempotent.

**Paperclip version or commit**
`master` at `e482f0f632`, restart-chaos waves on 2026-11-06.

**Steps to reproduce**
1. Start a native task in a git worktree. Let the agent write an
uncommitted file.
2. Kill the control plane so recovery ends in
`native_restart_recovery_blocked` or an exhausted generation.
3. Click Retry.
4. The successor runs in a fresh clone without the uncommitted file or
the task branch.

Root cause:
- `server/src/services/fresh-workspace-continuation.ts`
`canSelectFreshWorkspace` returns a fresh intent for every task Retry
that has a system recovery record.
- `server/src/services/explicit-native-continuation.ts` (fresh selection
in the source loop) then always persisted `fresh_isolated_workspace`.
- `server/src/services/fresh-workspace-seed.ts`
`materializeFreshWorkspaceSeed` clones the project source at the base
ref with `--no-local`, checks out a detached commit and removes
`origin`. Uncommitted work and the branch are not carried over.
- RC-35: `server/src/services/heartbeat/workspace-preparation.ts`
rejected any reusable existing workspace while `fresh` was true,
including the attempt directory that the same run had already created
and bound.

## What Changed

- New continuation mode `reuse_issue_workspace`. At Retry admission,
after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that
the task's bound workspace is the single local directory the failed run
used, owned by the task, not a shared workspace, not under remote
storage authority, with no other lease, failed cleanup or running
workspace operation, a settled recovery generation, no workspace-export
failure, and a readable Git state rooted at that directory. Then the
Retry continues there with a fresh session. Otherwise the existing fresh
attempt is kept.
- The reuse decision consumes the generation's one alternative attempt
(`freshAttemptRunId`), sets the Retry strategy to `continue`, and
records `issue.retry_workspace_reused`.
- Environment selection pins the reused workspace binding and fails with
`retry_workspace_binding_mismatch` if the binding changed before
dispatch.
- Fresh fallback: the run's continuation context gets `priorWorkspace`
(path, branch, bounded list of uncommitted files). Git status is read
with optional locks disabled. The prompt tells the agent to inspect it
and carry over the work. The old directory is not changed.
- RC-35: preparing the same fresh attempt again after a restart reuses
the attempt directory that this run already bound.
- Reuse checks run Git through the shared workspace scan scheduler with
short deadlines, because admission can hold the issue lock.
- `doc/task-recovery.md` and `doc/execution-semantics.md` describe the
new contract.

## Verification

- `server/src/services/retry-workspace-continuity.test.ts` (embedded
Postgres, real git worktree with uncommitted edits):
- exhausted-generation Retry and blocked-recovery Retry continue in the
same worktree, `.git` file and branch intact. Both failed before the fix
(readiness was `fresh_isolated_workspace`).
- Blocked recovery without a generation and duplicate holds for one run
also reuse the worktree. A running operation orphaned by a restart of
the stopped source does not block reuse.
- The fresh attempt is kept when the workspace is unsafe: live lease,
running workspace operation, corrupt `.git`, missing directory, remote
authority, shared workspace, running export generation. `priorWorkspace`
is omitted for the unsafe writer and shared cases.
- `workspace-preparation.test.ts`:
- A `reuse_issue_workspace` Retry prepares in the original worktree with
its uncommitted file and branch.
- RC-35 red/green: re-preparing the same fresh attempt reuses its bound
directory and keeps files written before the restart. Another run still
gets `isolated_task_directory_binding_mismatch`.
- The fresh fallback sets `priorWorkspace` with the old path and
`unpublished.txt`.
- `environment-selection.test.ts`: the new mode pins the binding and
rejects a changed binding or cwd.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt names
the retained directory and its files.
- Regression files run locally: `workspace-preparation`,
`environment-selection`, `task-retry`, `quarantined-task-retry`,
`explicit-native-continuation`, `workspace-recovery`.
- Restart-chaos (`tests/restart-chaos`, local, B1+B5, codex+claude,
K1/K4/K6, concurrency 2, rc06-masked):
- Final head: wave `w-20261107T014633-s7206` on `c6776283f3`: 8/8 PASS,
all 8 restart-crossing. 5 tasks needed a Retry (t03-t07: codex B1 K6,
codex B1 K1, codex B5 K6, codex B5 K1, claude B5 K6). All five finished
in the original git worktree with `.git` and every pre-fault file
present (`workspaceMovedTo` empty, `lostFiles` empty); no "Retry keeps a
fresh attempt" log line.
- Wave `w-20261107T004829-s7205` on `29373a6b6a`: 8/8 PASS, all 8
restart-crossing. 3 tasks needed a Retry (t02, t06, t07). All three
finished in the original git worktree with `.git` and every pre-fault
file present; no "Retry keeps a fresh attempt" log line. (`cc178492ef`
only makes the operation filter stricter for run-less jobs.)
- Wave `w-20261107T000412-s7204` on `d197082107`: 8/8 PASS, 7
restart-crossing. 3 tasks needed a Retry (t02, t04, t06). All three
finished in the original git worktree (`workspaceMovedTo` is empty), R2
passed with `.git` and every pre-fault file present, and no "Retry keeps
a fresh attempt" log line appeared.
- Wave `w-20261106T231416-s7203` on the first commit: 6/6 PASS, but its
t01 Retry still moved to `isolated-workspace-attempts`. That instance
was not kept, so the exact rejection was not recorded. The likely cause
is a `running` workspace operation that a killed server left on the
stopped source (the source restarted three times). The second commit
treats such restart orphans as not live and logs the rejection reason. A
`--replay` of that t01 on `d197082107` finished in the original worktree
with its `.git` file and no attempt directory.
- Review follow-ups in `d197082107`, `29373a6b6a`, `cc178492ef` and
`c6776283f3` (run-less workspace jobs still block reuse; red/green
covered by the `running_operation` case): holds are grouped by source
run; the readiness fallback (blocked recovery without a generation) also
reuses the task workspace; Git probes go through the shared scan
scheduler with 10-15 s deadlines; `priorWorkspace` is only set for a
task-owned directory with no other writer; the reuse rejection reason is
logged; `priorWorkspace` uses the same task-owned, settled,
single-writer predicate as reuse; Git probes share one budget per
workspace; the RC-35 check is a pure path check; renames are parsed from
either porcelain column.

## Risks

- A Retry now writes into the old directory instead of a new one. The
checks above prevent a second writer. A stale binding fails the run with
`retry_workspace_binding_mismatch`; it does not write to another
directory.
- If the cause of the failure was inside the workspace itself, reuse
could repeat it. Workspace-export failures, restore failures and
required workspace recovery keep the fresh attempt.
- A workspace operation of the stopped source that started before the
source settled and before the current server process no longer blocks
reuse (a restart orphan). Operations with no run, of another run, or
started in the current process still block it.
- `priorWorkspace` is an optional new field in the continuation
envelope. Older adapters ignore it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-06 20:27:22 -06:00

src

fix(recovery): keep the agent's work when Retry continues after exhausted or blocked recovery (#19666 )

2026-11-06 20:27:22 -06:00

CHANGELOG.md

fix: bound workspace Git scans (#11572 )

2026-08-17 22:11:30 -05:00

package.json

feat: configure company agent observability (#16874 )

2026-10-19 15:36:04 -05:00

README.md

fix(workspaces): size archive deadlines and preserve resume budgets (#19364 )

2026-11-05 16:09:32 -08:00

tsconfig.json

Fix root TypeScript solution config

2026-03-09 14:09:30 -05:00

README.md

@paperclipai/adapter-utils

Shared utilities for Paperclip adapters: process spawning, environment injection, sandbox/SSH transport, workspace sync, and the round-trip helpers that move code between the local execution-workspace cwd and wherever the agent actually runs.

For the adapter-author guide see docs/adapters/creating-an-adapter.md and the in-repo notes at packages/adapters/AUTHORING.md.

Sandbox bridge deadlines

Command-managed bridge control operations (queue reads, input delivery, setup, and cleanup) have a host-enforced deadline of at most 30 seconds per shell command. A shorter configured timeout still applies. These operations cannot inherit the agent run's hours-long lifetime or wait forever on a provider that ignores its timeout. Long-lived agent session commands keep their own limits.

A control timeout reports a transport failure. It does not prove that the remote command stopped and does not replay an uncertain write. The process session closes failed input delivery and uses its existing shutdown path; normal execution settlement must still verify termination.

Native archive transfer deadlines

Native syncIn and syncOut calls accept an optional minimumTimeoutMs budget. The server validates this budget and extends the worker RPC deadline. Providers can honor this field for file transfers without changing their configuration. Daytona uses the larger of this minimum and its configured timeout. Other providers keep their existing behavior. The minimum is limited to 30 minutes and does not change sandbox lifetimes, stored settings or the timeout for other commands.

The remote-workspace archive path uses the verified capture size for downloads and the local archive size for uploads. Large archives get one second per MiB plus 60 seconds, capped at 30 minutes. Small archives keep the existing provider timeout when that calculation is no greater than five minutes. Archive size, checksum, confinement and host space checks still apply.

An exact retained-sandbox resume keeps its recorded provider timeout and RPC overhead. It uses the normal one-minute fallback only when no timeout was recorded. A five-minute cold restore must not be cut off by that fallback. Resume still targets the recorded allocation and does not acquire a replacement.

No-remote-git contract

The local execution-workspace cwd is the only persistence boundary across runs. No adapter may depend on a git remote for cross-run state.

Adapters that run the agent on a different host should use the SSH round-trip helpers in src/ssh.ts:

prepareWorkspaceForSshExecution({ spec, localDir, remoteDir }) — bundles the local cwd (tracked files, dirty edits, untracked additions, and the git history needed to reconstruct it) to remoteDir before the run starts. Runs with no git remote configured.
restoreWorkspaceFromSshExecution({ spec, localDir, remoteDir, ... }) — syncs the remote cwd back into localDir after the run, including any new commits the agent created. Also runs with no git remote configured.

prepareRemoteManagedRuntime in src/remote-managed-runtime.ts wraps both calls for adapters that want a per-run remote workspace and an automatic restoreWorkspace() finally hook.

The invariant is pinned by the no-remote-git contract case in src/ssh-fixture.test.ts, which asserts that a remote-only commit propagates to the local worktree through the prepare → restore round-trip with no git remote configured at any point. Do not regress that test.