Files
PaperClipAI/doc
Devin FoleyandPaperclip 6293e581f6 fix(recovery): resume cancelled workspace retries from verified waits (#19276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A shared workspace can make a run wait before it calls a provider.
> - The scheduler saves a retry for that wait.
> - The control plane can cancel the retry while it is still queued.
> - Recovery then asks for a process stop receipt for a run that never
claimed execution.
> - This pull request verifies the retry and its exact workspace-wait
parent before it accepts a later user instruction.
> - The benefit is that Retry and saved messages can continue through
normal admission without replaying old work.

## Linked Issues or Issue Description

Refs #13640 and #15483. Related fixes: #18838 covers dispatcher
cancellation codes; this change covers a system-cancelled scheduled
workspace retry. Related open PR #15383 covers claimed
conversation-adapter preparation. This change requires an unclaimed
retry with no run profile or owner.

**What happened?**
A scheduled workspace retry was cancelled before runtime selection. The
run retained an expected system cancellation and a
`startupCancellation.beforeNativeSelection: false` marker. It had no
start timestamp, execution stage, run profile, controller, process,
lease or provider events. Recovery recorded an unknown-action hold. A
later user message and Retry could not pass that hold.

**Expected behavior**
A later authorized instruction can start a fresh turn when the saved
retry and its exact parent prove that no provider work started. Missing
or conflicting evidence must retain the hold.

**Steps to reproduce**
1. Record a terminal workspace wait before provider dispatch, with
`executionRecovery.kind: workspace_wait` and `providerWorkStarted:
false`.
2. Save a scheduled retry for that same company, task and agent.
3. Cancel the retry through the control plane before it claims
execution.
4. Send a new user message or request Retry after the cancellation.
5. On the base commit, admission returns no continuation. With this
change, normal admission creates one fresh successor.

**Paperclip version or commit**
Reproduced in source based on master at
`499712c68b3ca95801b6114d5f2f0f0ea6759318`.

**Deployment mode**
Source checkout with isolated embedded PostgreSQL. Hosted rollout and a
real provider run remain to be verified after merge.

## What Changed

- Extend the existing cancelled-startup proof with one source-bound
workspace-retry case. Require the canonical retry parent and reason to
match the saved context.
- Verify the parent's company, task, agent, time order, immutable Runner
claim and typed workspace-wait record. Reject runtime, process, lease,
coordinator, result and output evidence on either run.
- Keep validation of the full saved result in the database predicate.
The bounded execution projection can add empty receipt fields without
changing eligibility.
- Reuse the existing admission, Retry and deferred-message paths.
Preserve cancelled history and unknown action outcomes. Keep permission,
Stop, budget and recovery-limit gates.
- Add 23 regression cases, including heartbeat message and Retry
admission, concurrent saved-message recovery, and 19
conflicting-evidence cases. Document the contract.

The cancelled-startup service owns this proof. No provider protocol,
schema, UI or shared contract changes. Reads use exact run IDs and
return only narrow metadata. The new reads occur only for a cancelled,
unstarted Runner retry candidate.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/explicit-native-continuation.test.ts
src/services/legacy-execution-recovery.test.ts
src/services/execution-blocker-read.test.ts
src/services/task-retry.test.ts` — 452 passed.
- `pnpm -r typecheck` — passed.
- `pnpm build` — passed.
- `node scripts/check-module-boundaries.mjs` and `git diff --check` —
passed.
- Local diff review and secret/private-context scan — passed. Test
identities are synthetic.
- Regression history: the original positive case failed on unchanged
source while 11 negative guards passed. Initial test fixtures then
exposed ownership/contract constraints and a missing saved-wake
envelope; the fixtures now use valid persisted shapes. The heartbeat
Retry test exposed extra keys in the bounded evidence projection.
Full-record validation now runs in SQL, and both assembled heartbeat
paths pass. The unknown action outcome stays unchanged.
- Platform: macOS arm64, Node 25.6.1, pnpm 9.15.4. Tests use real
isolated PostgreSQL. They stop at admission and do not call a live
provider.
- Exact-head GitHub CI: 54 successful checks, 3 skipped, no unfinished
or failed checks. Greptile scored
`cfbf55cea68e3c69f65f50afcba98f775fa6fa3d` 5/5, with no actionable
findings and no review threads.
- The full repository `pnpm test:run` was not run locally. A previous
full run in this repair effort stalled and was stopped after 2h45. The
four complete affected suites pass; full PR CI must pass before merge.

## Risks

- False admission could allow work before a predecessor is safe. The
proof fails closed on missing parent records, different ownership, a
changed retry binding, later parent completion, restore obligations, any
execution resource, or any provider/output evidence.
- This deliberately covers only system-cancelled, unclaimed workspace
retries. Other cancellations retain their current reconciliation path.
It does not grant automatic replay, certify unknown external writes, or
reset exhausted recovery limits.
- No migration. Verify the serving version and an existing saved-message
continuation after deployment.

## Model Used

- OpenAI `gpt-6.1-sol`, xhigh reasoning, through the Codex harness in T3
Code. Used code editing, shell tools and test execution. The harness
does not expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 09:13:46 -08:00
..
…
…
…
…
…