mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
4398a723c0
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs send provider tool calls to the server as durable
semantic inputs. The server runs the tool and saves the result in the
controller journal
> - Close must prove that every admitted tool call has a saved result.
Otherwise it reports `native_session_settlement_incomplete` and the
runner holds a cleanup gate (`native_session_cleanup_quarantined`)
> - A server restart in the middle of a tool call leaves its input
without a result. The new controller asks the tool authority for a
receipt. Tools without a receipt (assigned app tools,
`connections_search`) return nothing, so the input stays pending forever
> - Runs that already failed this way kept no-replay holds for a turn
that had completed. The task stayed blocked with no Retry and no
continuation
> - Restarts are normal. Work that a restart interrupts must settle by
itself, not wait for a person
> - This pull request settles such a callback as interrupted, with an
unknown outcome, when its owner is provably gone, and settles the holds
that earlier releases left once their controller is gone. It never runs
the tool or the turn again
> - The benefit is that close drains after a restart, and tasks that
earlier releases blocked this way continue on their own
## Linked Issues or Issue Description
Refs #20572 (that PR fixed the presentation of this quarantine. Its
Risks section names this runner drain rule as a separate follow-up.)
Refs #20641 (it settles a resolved no-replay hold when a sibling action
saved the run's files. This PR covers a different case: the hold's own
recovery saved the files, and its run's turn completed. Both use the
same recovery receipt, `completedWorkspaceRecoveryCondition`.)
**What happened?**
A remote native run started an assigned app tool call. The server
restarted 3 seconds later. The new server adopted the runner, and the
provider turn completed about 9 minutes later. The tool call never got a
saved result. Close reported `native_session_settlement_incomplete` with
`runnerSuspended: true`, `providerDrained: false` and one pending
callback. The session quarantined, and the same-run recovery was refused
at admission. A second run showed the same shape with a
`connections_search` callback after a later restart.
The first run then had two resolved-but-blocked holds with
`preserve_without_replay_v1` and an unknown action outcome: the
cleanup-quarantine hold, and a continuation hold whose workspace
recovery saved the files. The task view said "Saved work recovered. Send
a message to continue." with no Retry and no continuation. The task
waits on a child task and stays blocked.
**Expected behavior**
After a restart, a tool call without a saved result settles
automatically. A durable receipt is replayed. Without a receipt, the
agent is told that the outcome is unknown. Close proves the drain, and
no cleanup gate is left. Holds that earlier releases left for a
completed turn settle without a person, and the task continues when its
own wait ends.
**Steps to reproduce**
1. Start a native run and call an assigned app tool.
2. Restart the server while the tool call runs.
3. Let the adopted turn complete, then close the session. Close reports
`native_session_settlement_incomplete` and quarantines. The task stays
blocked by the no-replay holds.
**Paperclip version or commit**
master at fd6e71a7ff.
## What Changed
- Root cause: `DurablePrpControlPlane.#recoverPendingSemanticInputsOnce`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
keeps an input pending when `recoverSemanticToolInput` returns null.
`PaperclipRunnerToolAuthority.recover` returns null for every tool
without a durable receipt. Nothing else could settle the input, so
`semanticToolResultsSettled()` stayed false and `#closeOnce` in
`runnerd-codex-transport.ts` threw
`NativeSessionCloseUnrecoverableError`.
- The control plane now passes `admission` to
`recoverSemanticToolInput`: `this_controller` when this instance
dispatched the callback live, `earlier_controller` when it restored the
input from an earlier controller's journal. The field is not written to
the result command or the journal. A live dispatch leaves this record
when its result is journaled.
- New
`packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts`:
- `dispatchToolCallback` records the callbacks that this process runs,
and the ones it saw fail.
- `recoverToolCallback` replays a durable receipt first. Without one, it
settles an `earlier_controller` callback that this process does not run
as `tool_call_interrupted_by_restart` (error result, `outcome:
"unknown"`, `retryable: false`). A `semantic_tool_outcome_unknown`
rejection from the authority settles the same way, only under the same
owner-gone proof.
- `native-session-executor.ts`: `SessionAuthorityEpoch.execute` runs
through `dispatchToolCallback`. `SessionAuthorityEpoch.recover` runs
through `recoverToolCallback`. The three recovery call sites pass
`admission`.
- New ownership kind `native_completed_turn_cleanup_hold`
(`server/src/services/native-runtime/completed-turn-cleanup-hold-settlement.ts`,
registered in `recovery/ownership-registry.ts`). It lists a native run
that failed with `native_session_cleanup_quarantined` after its saved
checkpoint recorded a completed turn, and that has a system-recorded
no-replay hold. After `proveOwnerGone` proves the closing controller
gone (its in-process cleanup gate went with it), it settles each such
hold as `restored`, removes its replay block, and logs
`issue.recovery_action_resolved`. A hold that still protects files not
verified as saved stays for workspace recovery. Operator decisions are
never reopened. It starts no turn. `heartbeat.ts` registers the ordinary
resume path (`resumeRemoteStopComments`) for saved user input.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.
## Verification
- Red before the fix, green after:
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`
"host-tool callback orphaned by a server restart". It uses the real
control plane and an authenticated runner. A callback starts 3 seconds
before the restart, the old controller retires with no result, the new
controller adopts the runner, the turn completes, and then recovery
runs. `paperclip_call_assigned_tool` and `connections_search` settle as
`tool_call_interrupted_by_restart`, the runner accepts the late result,
`semanticToolResultsSettled()` becomes true (the condition close
requires), and the tool is not dispatched again. Before the fix, 4 of 5
cases failed: recovery received no admission and the callback stayed
pending.
- `server/src/__tests__/completed-turn-cleanup-hold-settle.test.ts` uses
the staging shape of both holds (real Postgres, rows as earlier releases
saved them). Before the fix, the ownership reconciler did not list them
(4 of 4 failed), and the execution blocker was the files hold with
`canRetry: false` and `canContinue: false`. After the fix, both holds
settle as `restored`, the execution blocker is null, no run is created,
the task keeps `blocked`, and the next pass lists nothing.
- Safety cases:
- A callback that this process still runs stays pending. After it
finishes, the next recovery pass settles it.
- A callback that this controller dispatched and saw rejected stays
pending, as before.
- A durable receipt is replayed and not replaced.
- Holds stay while the closing controller is alive (its gate may still
exist).
- A hold whose files are not saved stays; the run's other hold settles.
- Holds of a run whose turn did not complete are never listed.
- Focused suites pass: `durable-prp-control-plane.test.ts` and
`runnerd-codex-transport.test.ts` (506 tests); runner adoption,
release-resume, semantic readiness, ACPX reconciliation and external
driver tests; `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` (1496 tests);
`ownership-reconciliation-report.test.ts` and
`interrupted-preparation-hold-settle.test.ts`.
- `tsc --noEmit -p packages/paperclip-runner/tsconfig.test.json` passes.
`pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` returns exit 2 (a global doc input changed), so required
CI supplies the broad coverage.
## Risks
- Previous-release story: no persisted shape changes. `admission` is
computed at recovery and never saved. A journal that an earlier release
saved with a pending callback restores as `earlier_controller`, so that
session settles at its next adoption. Holds that earlier releases saved
settle on the reconciler's first pass after deploy. No migration.
- No double execution: recovery never calls the tool again. It replays a
durable receipt or reports an unknown outcome. The agent is told to
check the current state before it repeats the call. Hold settlement
starts no turn and replays nothing.
- The owner-gone proof for a callback: the new controller holds the
run's checkpoint fence, and the earlier controller retired its journal
writer. If the earlier process is still alive during an overlapping
deploy, it can still finish the tool, but it can never record a result
in this session. The unknown outcome is accurate in that case.
- Hold settlement requires `proveOwnerGone` on the closing controller's
stamp, because the cleanup gate lives in that process. On a server that
has not restarted since the quarantine, the holds stay.
- The in-process failure memory is bounded to 4,096 entries. Eviction
only lets a much older failed callback settle as interrupted later. It
never replays it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>