Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 57b787d3d2 fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run finishes but copying its files out of the sandbox fails,
Paperclip holds the task and tries to recover the saved files. When
recovery gives up, the task offers Retry.
> - Some saved files can never be recovered: the sandbox that held them
was destroyed, or Paperclip cannot prove which runs used a shared
sandbox. Each Retry ran the same check again, failed again, and parked
the Retry in a wait that never ends.
> - The task told the person to press Retry again or to wait for a
Paperclip update. That breaks the rule that recovery must work and that
a Retry that is shown must work.
> - This pull request makes Retry continue the task in a fresh session
and a new workspace from the work that is recorded, when the saved files
are provably unrecoverable or cannot be proven safe. The task says
clearly that the unsaved files were not recovered.
> - The benefit is that an offered Retry starts a run, and nobody waits
for a fix that another click cannot bring.

## Linked Issues or Issue Description

Refs #21315, #21295, #21267

**What happened?**

On hosted staging, two blocked tasks offered Retry (`canRetry: true`).
Each Retry was accepted, and within about 3 seconds it was parked in a
wait. No run started. Every later Retry did the same.

- Task A: the hold cause was
`native_workspace_sync_out_retry_exhausted`. Recovery ended with
`source_unavailable` (`use_alternative_strategy`). The sandbox that held
the files had a signed absence proof and a `destroyed` termination
receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the
agent's files failed. Retry to try again".
- Task B: the same hold cause. Recovery ended with
`allocation_ownership_ambiguous` (`service_required`), because the
sandbox was a reusable sandbox that other runs also used. Each Retry
opened a new recovery generation, ran the same check, and was parked
with `saved_work_recovery_refused`: "Retry runs the same check again and
succeeds only after Paperclip is updated."
- In both tasks, the agent's turn had finished and its result was
accepted. Its work was recorded outside the sandbox: a plan document and
screenshot attachments for task A, and a pushed branch with an open pull
request and an implementation document for task B.

Root cause: admission had a branch for an accepted result whose files
were not yet saved. When the recovery budget was spent, the branch
always returned a `saved_work_recovery_*` wait. It did not check whether
the files could ever be recovered, so the fresh-workspace path that
admission already has was never reached. The automatic retained-source
fallback did not help either, because it handles only the remote
workspace v1 reference, and these runs use the native workspace sync v3
reference. Also, Retry opened a new recovery generation for an ownership
refusal, which only repeated the same check.

**Expected behavior**

When the saved files are provably unrecoverable or cannot be proven
safe, Retry continues the task in a fresh session and a new workspace
from the recorded work. The old files are never applied and the source
sandbox is not changed. No provider turn is replayed. The task says that
the unsaved files could not be recovered. The Retry offer and admission
agree.

**Steps to reproduce**

1. Let a native run finish with an accepted result, and let its
workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`).
2. Let saved-work recovery end with `source_unavailable` (sandbox
destroyed) or with `allocation_ownership_ambiguous` (shared reusable
sandbox).
3. Press Retry.
4. Observe that the Retry receipt waits with
`saved_work_recovery_failed` or `saved_work_recovery_refused`, and no
run starts. Press Retry again and observe the same result.

**Paperclip version or commit**

Staging on `origin/master` after #21315 (`bbafa78fe2`).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (native runtime runs).

## What Changed

- `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure`
(`system-recovery-state.ts`) are the one rule for saved files that no
recovery can apply. The rule is true for a spent recovery whose failure
is a permanent source refusal: `source_unavailable`,
`allocation_ownership_ambiguous` or `historical_source_missing`. It is
false for `source_binding_invalid`, because a new recovery generation
can still verify that source. It is false for `archive_unsafe` and
`snapshot_verification_failed`, because a fresh attempt still imports
earlier saved snapshots and would fail before the agent starts. It is
also false after a fresh attempt is recorded.
- Admission (`explicit-native-continuation.ts`) no longer parks a Retry
of an accepted result in `saved_work_recovery_*` when every live
recovery of the run is unrecoverable. The Retry takes the existing
fresh-workspace path (`fresh_isolated_workspace`). That path keeps all
stop proofs, leaves the source sandbox and workspace untouched, and
starts a fresh session.
- When such a Retry starts, the task gets one system comment. The
comment says that Paperclip could not recover the unsaved files, why,
and that the task continues from its recorded work (comments, documents,
attachments and pushed commits). The settled hold records the same text.
- Retry (`task-retry.ts`) binds to an unrecoverable recovery generation.
It does not open a new generation that repeats the same check.
- The blocker text for unrecoverable files says that Retry continues the
task in a new workspace from its recorded work. It no longer says that
Retry succeeds only after an update.
- `canAdoptNativeSourceRetention` and the permanent source-refusal lists
moved into `system-recovery-state.ts`, so that admission, Retry and
messages share them. The old modules re-export them.

## Verification

- Red/green: `cd server && npx vitest run
src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on
`origin/master` sources (or on the earlier head for the snapshot case)
and pass with this change. They are built from the two staging shapes
(destroyed sandbox with `source_unavailable`, and a shared reusable
sandbox with `allocation_ownership_ambiguous`):
- offers a Retry that continues the task (blocker and task view agree,
and the text names the fresh workspace)
- starts a fresh run from the recorded work: the Retry receipt has no
wait and a run, the strategy is `fresh_isolated_workspace`, the hold is
settled with the fresh attempt, one system comment explains the lost
files with no raw IDs, the source lease is unchanged, and no new
recovery generation opens
- keeps a Retry out of a fresh attempt that would import a succeeded
snapshot whose archive is unsafe
- `workspace-recovery-warm-handoff.test.ts` › "wakes physical
preservation from HTTP Retry before passive history without replaying
the provider" failed on an earlier head of this PR, which also treated
`source_binding_invalid` as unrecoverable. It passes unchanged now.
- Updated tests that pinned the old repeat-the-check behavior:
`workspace-recovery-preparation-gates`, `task-retry`,
`system-recovery-state`, `workspace-recovery-retry-copyback`.
- Related suites pass: `system-recovery-state`,
`workspace-recovery-preparation-gates`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`task-retry`, `saved-work-retry-eligibility`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
`workspace-recovery-admission`, `task-interruption-query`,
`native-workspace-sync-retention`.
- `cd server && node scripts/typecheck-layers.mjs` passes (source and
test layers).

## Risks

- Medium-low. A Retry of an unrecoverable hold now starts a new agent
turn in a new workspace instead of waiting. The turn starts only through
the existing fresh-workspace admission, which keeps every stop and
cleanup proof.
- Unsaved files of the old run are not applied. That was already true;
the change is that the task continues and says so.
- `source_binding_invalid`, `snapshot_verification_failed` and
`archive_unsafe` keep today's path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:23:23 -06:00
7d1e9b3477 fix(recovery): retain exact warm native export sources (#17916)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task recovery must save unfinished workspace exports before
continuation.
> - A prepared export can still refer to a sandbox with a warm reuse
policy.
> - The strict recovery inspector rejects that policy and exhausts
source verification.
> - Ordinary reuse must not give the same unfinished allocation to
another run.
> - An existing user Retry can authorize bounded file maintenance after
exact ownership checks.
> - This change retains that exact source and preserves the original
failure record.
> - Recovery completes only when files, stop evidence and policy
settlement agree.

## Linked Issues or Issue Description

**What happened?**

A failed native run has a prepared `keep_running` workspace export. Its
exact sandbox lease still has `reuse_by_environment` policy. Native
export cleanup can preserve that policy while it stops the sandbox.
Generic workspace recovery then rejects the source with
`lease_policy_not_retained`. A saved user Retry remains blocked before
file export.

**Expected behavior**

Keep unfinished exports out of ordinary warm reuse. With current user
authority and an unchanged saved Retry, retain the exact source for one
bounded maintenance generation. Preserve files and return continuation
to normal admission. Keep all ambiguous sources held.

**Steps to reproduce**

1. Prepare a native workspace export with a warm sandbox and
`keep_running` disposition.
2. Save a normal board Retry for the failed source run.
3. Persist its first exhausted source-verification failure before file
export.
4. Run the normal maintenance worker. Before this change, source policy
rejection leaves the saved files unrestored.

Refs #17773 for source diagnostics. Related #17839 records a missing
preservation obligation during blocked restart. This change handles an
existing prepared warm export whose verification already failed.

## What Changed

- Exclude unfinished physical allocations from reuse and obsolete-lease
cleanup across every historical lease row. Reuse a clean candidate or
acquire a fresh sandbox through normal selection. Preserve exact
same-run reattachment, workspace-less execution and ordinary task
admission checks.
- Preserve the newer prepared remote-workspace writer proof for native
and legacy runs. Share its binding check between single-source recovery
and one batched allocation read.
- Keep persisted recovery-action IDs when constructing continuation
source checks. Synthetic sources cannot grant snapshot receipt
authority.
- Permit one audited maintenance adoption for the first exhausted
source-verification attempt. Require an unadmitted canonical saved board
Retry, current user authority, unchanged assignment and latest source
run.
- Keep all company, source, allocation, workspace writer, controller and
physical finalization ownership checks. The public read-only inspector
keeps its strict policy check.
- Pin the exact source for file maintenance. Preserve the previous
generation and reuse the existing bounded recovery worker. Create no
provider turn or new user request.
- Restore the warm policy in the same transaction as verified native
copyback completion. A failed write or changed settlement keeps a
recovery obligation.
- When local merge metadata is missing, settle the exact verified
snapshot while keeping its unfinished allocation retained. Deliver the
saved Retry through normal fresh isolation and attach the verified
files. Keep the accepted result as history.
- Require native copyback proof before reopening an accepted result's
original workspace finalizer.
- Validate snapshot receipts through one shared boundary for warm,
retained and ephemeral sources. Bind the receipt to the current
generation's operation IDs, company, issue, run, recovery action, lease
and provider allocation.
- Retire fixture input before removing recovery proof, and use the same
configured heartbeat for HTTP and worker dispatch. Keep the exact
catalog refusal and no-provider-work assertions.
- Document the recovery contract in technical documentation. Leave
capability maps unchanged for this bug fix.

## Verification

- Candidate-selection RED: all six clean-candidate and fresh-allocation
controls fail on unchanged `f54e7baf538` because a pending export aborts
the whole acquisition. GREEN: the focused suite passes **15/15 cases**
after candidate filtering. Both provider paths preserve the protected
lease and never resume or destroy its allocation. Obsolete fingerprints
cannot send that allocation to cleanup. Historical-row fencing, modern
native/legacy writers, exact same-run reuse, workspace-less execution
and the single-read allocation-history check remain covered.
- The existing reusable-runtime selection cases passed **40/40**.
Driver/capability contract suites passed **5/5**. Reusable service
controls passed **2/2**. After merging current master, the combined
lease/provider suites pass 61 cases and the recovery integration
selection passes 36 cases, including the upstream inherited-session and
resolved-interruption assertions. This gives 97 passing targeted cases
on the final source. A duplicate test-app declaration from the automatic
merge was removed while retaining the upstream assertions.
- The source-history repair has meaningful red/green evidence: three
valid retired-handoff cases could not complete under the old blanket
history-row rejection. Settlement now uses the same full source
inspector as adoption. It accepts exact retired handoffs and still
rejects a predecessor that becomes active after export. The 26 focused
history, rollback, snapshot and ambiguity cases passed before this last
merge.
- Server typecheck passes on the final merged source. Module boundaries
and `git diff --check` pass. Full local repository tests and a
successful current-head full build are not claimed. Current-head CI is
complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required
gates pass, with 54 successful checks, three expected skips, one neutral
Paperclip review check and a successful security status. [Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370).
Greptile reviewed that exact head at **5/5** with no actionable
findings. Its check completed successfully. [Ponytail also reviewed the
exact head at
5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408).
No review threads remain unresolved, and the branch is mergeable.
- Earlier local broad runs exposed a dropped recovery-action ID in the
merged source projection and deferred test input being revisited after
its records were removed. Both were repaired. Later local disk
exhaustion caused manifest and saved-file reserve failures; generated
caches were removed only from these checkouts. The affected saved-file
control passed unchanged after cleanup. The original logs are retained.
No disk reserve or timeout was lowered.
- On the prior head, the first [full CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530)
received simultaneous runner shutdowns. Four jobs were cancelled, and
two late jobs have lost-runner annotations with unavailable logs. The
Runner job had five earlier ACPX failures whose full assertion report
was lost on interruption. Their cause remains unproven. Identical Runner
source passed the [#19004
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725),
including all seven enabled tests in that file.
- That attempt's [browser
shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595)
returned to the exact saved task URL, but its conversation was still
loading at the unchanged heading deadline. The
[artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931)
is retained. All five Cloud-auth cases passed locally on the unchanged
head and in the [same-source CI
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966).
The delay remains unexplained. One rerun of failed/interrupted jobs was
submitted after the workflow completed. This new source revision
requires fresh CI.
- The earlier [original-head
workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175)
passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5.
These results do not establish readiness for the new head. The earlier
Superagent scan was neutral after a pinned-base checkout failure; its
external analysis remains unverified.
- These tests use real PostgreSQL and controlled provider boundaries.
Hosted rollout and original-task acceptance remain unverified.


## Risks

- Maintenance can resume and stop the exact sandbox for file export. It
cannot admit a provider turn, choose another allocation or bypass normal
continuation gates.
- A saved Retry alone is insufficient. A changed source, newer run,
active writer, competing lease, lost user authority or previous adoption
keeps the hold.
- Policy restoration can fail. Completion and restoration now commit
together, so normal recovery remains responsible for a retry.
- Snapshot preservation cannot repair missing local merge metadata. Its
old allocation remains explicitly retained. A snapshot receipt grants no
permission to reuse or destroy that allocation, or to replay the
accepted result.
- The allocation check uses one projected join per candidate over the
existing provider lease index. It adds no polling timer or recurring
scan. A fresh fallback can create another sandbox while the unfinished
source remains retained; it grants no authority to execute a provider
turn.
- This change does not establish the cause of a prior provider failure
or prove that any specific hosted sandbox still contains its original
files.

## Model Used

OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs.
Both used repository tools and PostgreSQL verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 14:47:42 -08:00