mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude
233f615644
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs agent turns and recovers them after a
server restart
> - A graceful restart can stop a legacy run (for example a `process`
adapter agent) while the run is still in setup, before the adapter
starts
> - That run was parked with `legacy_execution_requires_reconciliation`,
so automatic recovery stopped
> - A `process` or `http` adapter cannot continue that hold through
Retry, so the accepted Retry waited forever and the task stayed blocked
> - This pull request proves "no adapter work started" for that case and
resumes the run automatically
> - The benefit is that a restart during setup no longer strands the
task, and Retry starts a run for holds that an older server saved
## Linked Issues or Issue Description
**What happened?**
The restart-chaos harness ran workload Q3 (a board approves an agent's
app action late) with a `process` adapter agent. A graceful restart
(fault K2) fired right after the approval. The agent's follow-up run was
interrupted with `server_shutdown_interrupted` 0.2 s after it started,
while it was still in setup. Automatic recovery stopped with
`legacy_execution_requires_reconciliation`. The board clicked Retry. The
server accepted the Retry (HTTP 202), but no run started. The task
showed "Retry is waiting for execution checks" (`retryWait.reason =
execution_recovery`) forever.
**Expected behavior**
A run that a graceful shutdown stops before any adapter work must resume
automatically. An accepted Retry must start a run.
**Steps to reproduce**
1. Run a `process` adapter agent on a task.
2. Send SIGTERM to the server while the run is still in setup (before
the adapter starts).
3. Restart the server. The task is blocked with
`legacy_execution_requires_reconciliation`.
4. Click Retry. The request is accepted, but no run starts.
**Paperclip version or commit**
master at 2c2186706e
**Additional context**
The graceful-shutdown drain marks a legacy run `interrupted` without any
bootstrap evidence. The legacy classifier then treats the run as
"outcome unknown" and creates a no-replay hold. Only conversation
adapters can continue that hold through Retry
(`admitExplicitNativeContinuation` returns null for `process`/`http`
adapters), so the Retry intent stays deferred with no owner. The
hard-restart reaper already uses the durable stage proof
(`executionStage = 'preparing'`, no process identity) to retry such
runs. The graceful path did not.
## What Changed
- New `server/src/services/legacy-shutdown-before-dispatch.ts`: one SQL
proof that a legacy run interrupted by graceful shutdown never reached
the adapter. The proof requires: `executionStage = 'preparing'` with a
controller boot ID, no `budgetDispatch.handedOffAt` handoff receipt, no
process identity, no execution-start event, and no native or
workspace-recovery obligation.
- `terminalizeLegacyExecution` applies that proof in the same
transaction that commits the shutdown interruption, under the run lock.
A matching run gets `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. The existing process-loss retry then
schedules it in the interrupted-preparation lane (no delay, no failure
budget spent), and no hold is created.
- `executionBlockerPredicate` ignores a
`legacy_execution_requires_reconciliation` hold whose source run carries
the same proof. This releases holds that an older server saved for the
same case, so Retry starts a run (upgrade-safe for old rows).
- Tests in `server/src/__tests__/heartbeat-process-recovery.test.ts`.
## Verification
- Red before, green after:
- `resumes a legacy run that a graceful shutdown stopped before adapter
dispatch (stage=preparing)`
- `lets Retry start a run for a pre-dispatch shutdown hold saved by an
older server`
- Control (passes before and after): `... (stage=dispatching)` keeps the
reconciliation hold and schedules no retry.
- `cd server && npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/interrupted-preparation-hold-settle.test.ts
src/__tests__/interrupted-preparation-restore-retry.test.ts` (609
passed).
- `npx tsc --noEmit -p server` passes.
- Live proof: restart-chaos `--env local --adapters codex --workloads Q3
--faults K2 --matrix --concurrency 2 --seed 109`. Results are added in a
comment.
## Risks
- The proof is narrow. A run that reached `dispatching`, recorded a
handoff, a process identity or any execution event keeps its
reconciliation hold. Rows from a controller without a lease (no boot ID)
keep their hold.
- The stage proof relies on the `dispatching` stage write that the lease
renewal makes before adapter admission. That write and the `preparing`
claim were added in the same change, so old rows with `preparing` are
reliable.
- Not changed: a `process` or `http` run that really started and has
unknown outcomes still keeps its hold. Retry for that hold is still
accepted but cannot continue it. That is a separate product decision.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>