Files
PaperClipAI/server/src/services
DottaandClaude Opus 5.5 482ec0aea9 fix(native): run saved copy-back retries on schedule behind a slow export (#19145)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs copy their sandbox workspace back after the provider
finishes. A failed copy-back saves a retry 30 seconds later, with a
bound of 3 attempts.
> - One shared delivery queue runs these retries. A pass of that queue
ends only when its slowest check ends.
> - A legitimate sandbox export can take 10+ minutes. While it runs, all
other due retries wait.
> - On staging, a run active across a deploy showed "Reconnecting" with
an overdue retry for about 24 minutes. Its retries ran at +24 min and
+3.5 min instead of +30 s. It settled only after about 32 minutes.
> - This pull request lets idle check slots admit entries that come due
during a pass. It does not overlap a running check's resource and does
not start work after abort or drain.
> - The benefit is that the existing 3-attempt bound settles a failing
copy-back in about 1 minute, not after the slowest unrelated export.

## Linked Issues or Issue Description

**What happened?**
A native run whose result is durable but whose workspace copy-back
failed stays in `running` with the label "Reconnecting" for a long time.
Its saved retry time is in the past. The retry does not run until an
unrelated slow copy-back in the same queue pass finishes.

**Steps to reproduce**
1. Start two native runs on different tasks with accepted results.
2. Let the copy-back of run A take many minutes.
3. Let the copy-back of run B fail. It saves a retry 30 seconds later.
4. Watch run B. Its retry does not run until run A's copy-back ends.

**Expected behavior**
Run B retries on its saved schedule. After 3 failed attempts it settles
with `native_workspace_sync_out_retry_exhausted` and a recovery action,
while run A continues.

**Paperclip version or commit**
master at a0c0851766. Observed on a staging deployment of 69dea878.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Observed with the Paperclip runner.

**Relevant logs or output**
In-run copy-back failed at 23:44:57 (retry saved for 23:45:28). The
deploy restarted the server at 23:49:45. One queue check exported
another task's workspace from 23:53:54 to 00:06:15. The overdue retry
ran at 00:09:06, then 00:12:39, and the run settled at 00:12:40 with
`native_workspace_sync_out_retry_exhausted`. About 20 other runs
followed the same timeline. Their failures at 00:12–00:13 are this same
retry exhaustion. They do not come from a reattach sweep.

## What Changed

- `registerScopedDeadlineWork` has a new opt-in
`concurrentChecks.admitDueWhileActive` option. While a pass still has a
running check, idle slots admit entries that are due (saved deadline,
failure backoff, and minimum spacing). They wait for the next deadline
or the next check completion.
- Admission skips entries owned by the pass and entries whose resource
keys a running check holds. It stops on abort or when `canRun()` is
false. The pass still ends only when all running checks end.
- The native workspace finalization queue opts in. Its `check` already
rereads the durable coordinator row before it acts.
- Regression tests: a real-database test in
`native-workspace-finalization-work.test.ts`, plus two helper tests
(resource-key safety and shutdown).

## Verification

- `cd server && npx vitest run
src/__tests__/native-workspace-finalization-work.test.ts
src/__tests__/scoped-deadline-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts` passes (68 tests).
- Red before the fix: "keeps another task's saved retries on schedule
and settles it while a slow copy-back is running" failed because
`onWorkspaceSettled` was never called for the failing run. "admits saved
deadlines that come due behind a slow check without overlapping its
resource" failed too.
- Green after the fix: the failing run makes exactly 3 attempts, settles
as `terminal_failure` / `native_workspace_sync_out_retry_exhausted`, and
the slow copy-back is admitted once and then succeeds.
- `tsc --noEmit` shows no errors in the changed files.
- Not done: a live staging check. This change is not deployed.

## Risks

- Low to medium. Only the workspace finalization queue opts in. Other
queues keep the current behavior.
- An admitted entry is not reconstructed first. The finalization `check`
rereads durable state, so a stale in-memory entry cannot start a second
export. The advisory lock and owner receipt still prevent two copy-backs
for one run.
- This change does not fix the cause of the copy-back failures on
staging (`native_session_lease_lost` in-run, then
`workspace_sync_out_failed` with no detail). That needs a separate
change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 19:26:02 -06:00
..
…
…
…
…