fix(recovery): wake retained workspace preservation after Retry (#17741)

## Thinking Path

> - Paperclip lets people manage AI agents and their saved work.
> - A stopped run can owe workspace preservation before saved input can
continue.
> - Retry saved a recovery request but did not wake the worker that owns
preservation.
> - Older recovery records could also take the bounded scan slots before
a due request.
> - A second Retry could change the revision while the physical owner
was still saving files.
> - This pull request wakes a separate worker after commit, selects due
work first, and keeps the active owner's revision.
> - Saved work can progress without replaying the stopped provider turn
or blocking other finalization.

## Linked Issues or Issue Description

**What happened?**

An operator approved a saved design and selected Retry after native
workspace finalization failed. The API accepted one canonical request.
The source remained scheduled at zero preservation attempts for more
than 30 minutes. No new provider turn started. A local regression also
found that repeated Retry during export invalidated the physical owner's
completion revision.

**Expected behavior**

Committed Retry must wake bounded source preservation. Due recovery must
progress ahead of passive cleanup obligations and unrelated history.
Duplicate requests must keep one physical attempt. Saved input must
still pass normal admission after verified recovery.

**Steps to reproduce**

1. Retain a terminal run with an exhausted workspace recovery and saved
authorized input.
2. Let the delivery coordinator become quiet, then request Retry.
3. Observe the recovery state and physical preservation receipt. Repeat
Retry while the export is held.

**Paperclip version or commit**

Observed on served commit `4b89eec644d6e1a99d58926e77ae58927c3fa87b`.
Local baseline: `a1f77821b9cd5c9c846fdf7a0fa5cc620ad2b6bb`.

Related work: Refs #17053, #17267, #17094. This change repairs explicit
Retry delivery. It is separate from the run instruction projection in
#17724 and historical unsafe-export discovery in #17687.

## What Changed

- Notify a dedicated workspace recovery duty from the transaction that
saves Retry, recovery admission, adoption, or recovery progress.
- Reconstruct work and saved retry deadlines at startup. Fill each free
export slot independently, track physical work for idle drain, and join
the dedicated worker's exports on shutdown.
- Drain one bounded pass through completed cleanup and saved-input
obligations. Restart that pass after committed changes without
repeatedly rechecking paused input.
- Share source eligibility between deadline selection and candidate
consumption. Keep resolved files-only obligations and ignore unrelated
resolved or cancelled history.
- Select due unfinished preservation before passive historical records.
Rotate past source locks held by other workers and retain the indexed
source lookup.
- Bind the canonical Retry intent without changing an active physical
attempt's generation or revision. Preserve that binding across later
progress writes.
- Add real producer, physical preservation, startup, deadline, idle, and
independent finalization regressions. Document the delivery contract.

## Verification

- Red controls: Retry without a notification stayed scheduled; duplicate
Retry invalidated the active export; whole-batch waiting stranded a
third Retry; a scan without cursor rotation starved a source behind two
actual advisory locks. Removing unfinished-first selection left HTTP
Retry scheduled behind five real recovered sources still owing cleanup.
The fixture explicitly verifies their eligibility with the worker's
actual query filter.
- Further red controls: the previous worker visits only two of three
eligible completed sources. Disabling the committed-change cursor reset
misses a new earlier source arriving while two callbacks are held. Both
pass with the final worker.
- Green: the public Retry HTTP route, registered production worker, real
file copyback, duplicate canonical receipts, and public recovery receipt
reload passed on an owned local fixture. Its agent is paused to keep
this files-only check from starting a provider turn.
- Full workspace recovery suite passed 56 tests, including the
foreign-lock regression.
- Execution-control lifecycle suite passed 24 tests. It covers idle SQL,
stored deadlines, cancelled and resolved history, independent
finalization, and shutdown. One, three, and seven paused completed
sources are each visited in a bounded pass, then stay deferred without a
successor or SQL during 90 seconds of fake-clock idle time. A committed
earlier source is evaluated when it arrives during two held callbacks.
- Both GitHub and Telegram `automatic_running` continuation regressions
passed with the new canonical intent binding.
- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- `node scripts/check-maps.mjs` and `node
scripts/check-module-boundaries.mjs` passed.
- Local `pnpm test:run` could not complete: embedded PostgreSQL
repeatedly failed to start. The owned run was stopped after this setup
failure repeated. Latest-head CI supplies broad coverage.
- Earlier setup failures included a localhost sandbox denial and loss of
the owned disposable Docker database. These were separate from the
product red controls above.
- Local database tests use an explicitly named disposable PostgreSQL
`_test` database. That opt-in database is reset before the suite.
Default embedded test behavior is unchanged.
- Exact-head CI is green at `ef7dcf1ba1060864c9d216c33b51f4d9f8f57fbd`;
Greptile and Ponytail both rate it 5/5, with no unresolved review
threads. Earlier CI caught the indexed discovery assertion and two exact
chat intent-binding regressions; all three were fixed and pass locally
and in CI. A separate chat shard lost contact with its self-hosted
runner without a product assertion result. A later shared runner
shutdown interrupted several jobs; one infrastructure rerun passed
before the final test-and-worker follow-up triggered a fresh green run.
- No staging deployment is part of this PR. The original approval
remains saved; deployed recovery still needs acceptance after rollout.

## Risks

- More recovery writes now wake a bounded maintenance duty. Completed
history must not keep it active. Idle and deadline tests cover this
boundary.
- Resolved files-only obligations must remain eligible without granting
replay. The startup regression checks that path.
- Provider stop proof, copyback ownership, source binding, retry
budgets, and normal execution admission remain required. There is no
schema or public API change.

## Model Used

- OpenAI Codex based on GPT-6, with code execution and browser tools.
The exact serving model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-26 10:28:12 -05:00
1 parent 338bf75761
commit 0373f9f261
13 files changed
+527 -63

No files matched your search

+7
View File
@@ -232,6 +232,13 @@ Gaps: An acknowledged retry can remain pending or fail; report actual useful res
Capability ID: `recovery/workspace`.
Committed authorized Retry and workspace-recovery requests notify the internal
preservation worker. Startup reconstructs its source-bound obligations and saved
retry deadlines. A duplicate request during physical preservation does not
replace the owner's generation or revision. Resolved files-only recovery retains
its no-replay restriction. The local PostgreSQL regression checks the producer,
worker, and physical source receipt; it is not deployed provider acceptance.
The internal preservation resume path grants no board or agent execution API.
It requires the exact source action, company, run, lease and environment, a
held physical workspace owner, and its current stop-only cleanup claim. The
+11
View File
@@ -257,6 +257,17 @@ gap.
### `operator-retry`
The [workspace recovery regression](../../server/src/services/workspace-recovery.test.ts)
uses the real Retry producer and a quiet delivery coordinator. It checks that a
committed request starts one physical source preservation, duplicate Retry keeps
the active owner's receipt valid, and startup reconstructs a resolved files-only
obligation without provider replay. The [execution-control lifecycle suite](../../server/src/__tests__/execution-control-work.test.ts)
checks independent finalization, stored deadlines, and a bounded pass through
completed obligations followed by quiet paused input.
The physical export regression holds one export while another completes, then
verifies a third Retry can use the free slot and shutdown joins the held work.
These local checks do not prove recovery of a deployed allocation.
Automated: [disposition notices](../../ui/src/components/DispositionRecoveryNotice.test.tsx)
tests pending/error/gated states and waiting for the real retry acknowledgement.
The recovery-card suite covers workspace divergence, reissue, reconcile, and