Files
PaperClipAI/server
DottaandPaperclip 60e03c002d fix(grok): hold new runs while connection needs recovery (#16445)
## Thinking Path

> - Paperclip manages agents and their work.
> - Grok Bot assignments use a managed Runner and a verified connection.
> - Stop can leave the connection held for remote-work inspection.
> - The queue started new runs before it checked that connection.
> - Those runs allocated Runners, failed, and caused more preparation
attempts.
> - This pull request checks readiness at queue admission.
> - Pending input waits for recovery without using managed compute.

## Linked Issues or Issue Description

Refs #16427 and #16422. The open Grok changes cover the coding adapter
and do not duplicate this fix.

**What happened?**

A real Stop test held the Grok connection for inspection. A second task
was pending. Four new preparation attempts acquired managed Runners,
then failed because the connection was held. No new external assignment
or webhook was sent. Pause stopped the loop, and all leases completed
cleanup.

**Expected behavior**

Keep fresh input in the durable queue until its connection is ready. Do
not allocate a Runner to discover a known connection hold. Keep existing
session reconciliation under its recorded authority.

**Steps to reproduce**

1. Connect Grok Bot and start an assignment.
2. Queue another task for that agent.
3. Stop the first assignment and leave the inspection hold in place.
4. Run queue recovery. The pending task repeatedly starts preparation
and fails.

**Paperclip version or commit**

Observed on packaged Core 5a9e6b6315 in
Cloud staging. This branch includes current master
afe5b3418d.

## What Changed

- Reuse the broker's current connection and authority checks before
fresh Grok run claims.
- Let valid connection-pool routing select a native provider before
applying any unused Grok connection hold.
- Leave pending input queued while readiness is denied. Recheck at the
claim boundary if Stop races the queue scan.
- Preserve already-started reconciliation and all later
runtime-selection and dispatch checks.
- Propagate database failures. Treat only explicit connection-authority
errors as a hold.
- Add queue, OAuth, broker and assembled heartbeat regressions.
- Update the Grok guide and existing release evidence plan.

## Verification

- Two queue admission assertions fail before the correction.
- Six focused suites pass all 813 checks, including current master's
native ownership coverage and a real healthy Codex pool member with a
held Grok connection. The routing test reaches native runtime selection
and then injects a controlled stop before provider start.
- The assembled test uses real OAuth and broker state, repeats public
queue recovery, and checks zero leases, assignments and extra webhooks.
- Repository typecheck and build pass on final head
83664a91ce. Its full local suite is still
running. The earlier full attempt was stopped as superseded after the
valid router review.
- The full suite uses a clean tool PATH. On the earlier baseline, mocked
Gemini tests reached the installed host CLI and exceeded their
five-second deadline. Both tests pass with that host CLI absent. Test
source and deadlines are unchanged.
- The existing native cleanup and queue claim-isolation tests pass. The
process-agent Stop browser journey passes locally. Its first CI attempt
queried the queue 67 ms after comment creation, before asynchronous
publication. The test now polls for the same durable entry with its
existing deadline.
- One server CI worker shut down after 1,047 tests passed. The failed
attempt remains recorded. No routing flags or gates were changed. No UI
component or schema changes.
- The final head passes 55 CI checks, with two optional skips. Greptile
reviewed that exact head at 5/5, with the earlier thread resolved and no
new actionable finding.
- Actual packaged recovery admission still needs a live repeat. The
provider currently rejects routine starts with an external rate-limit
error. This patch does not remove that provider limit.

## Risks

Readiness can change after preflight. Runtime selection and dispatch
retain their current authority checks. Already-started runs bypass this
fresh-work check so they can reconcile their original session. Other
providers keep their current queue behavior. Expected connection denials
hold input; unexpected database errors still propagate. No migration is
needed.

## Model Used

OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools
and computer use. The exact serving model ID, snapshot and context
window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 11:32:46 -05:00
..