mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 19:28:46 +01:00
DottaandPaperclip
0b87308d5d
## Thinking Path > - Paperclip manages agent work and preserves its saved results. > - Native warm sessions can continue on a retained remote Runner. > - A completed predecessor can still have a live finalization lease. > - The continuation must wait for that owner before it can start work. > - Two admission checks treated that temporary lease as ambiguous state. > - This pull request uses the existing deadline-based wait for both checks. > - The same business run can continue once, with its saved context and gates. ## Linked Issues or Issue Description **What happened?** A compatible remote warm continuation can fail before provider work starts. Its predecessor has completed, but a finalizer still holds a live lease. The warm-session check reports `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The earlier retained-sandbox check has the same problem on the command transport. **Expected behavior** Keep the unstarted continuation on its existing run. Wait until the recorded owner deadline. Recheck ordinary ownership, Stop, credentials and checkpoint evidence before execution. **Steps to reproduce** 1. Complete a remote warm turn and retain its controller records and provider session. 2. Give that completed predecessor a live finalization lease. 3. Admit a compatible follow-up with full local controller projections. 4. Before this change, admission fails permanently. It must instead persist a bounded pre-provider wait. **Paperclip version or commit** The PostgreSQL regression failed before production edits on `adb49a4434`. The current head is `5762dfbe4e`, based on master `e491b23eb5`. **Deployment mode** Native remote sandbox execution. No database migration or provider artifact change is required. Related: #18718 refreshes stale warm-controller projections. #18343 adds controller replacement recovery. #18554 restores suspended remote sessions. This change handles a live predecessor finalizer after those source checks. The shared historical error does not identify its failed predicate. This regression proves the finalizer race; it does not prove that every historical instance of that error had this cause. ## What Changed - Defer a compatible warm continuation while its completed predecessor has a live finalization lease. - Use the same wait in the preceding retained-sandbox read check. - Preserve the current run, provider attempt count and checkpoint. Reuse the existing resumption scheduler. - Add PostgreSQL cases for both finalization phases, lease expiry, company scope, incompatible configuration, unfinished commands, Stop and duplicate resumption claims. - Add an opt-in three-turn Linux Runner case. Update the child-controller fixture to own and register a local API listener. - Document the waiting behavior and acceptance case. - Repair two readiness barriers in the refresh-worker integration fixture after a required CI shard failed twice: wait for paginated startup and for active worker promises to settle before checking idle SQL. ## Verification - **Native regression RED:** before production edits on `adb49a4434`, the PostgreSQL boundary case received the permanent identity error instead of `NativeControllerRecoveryWaitingError`. - **Regression GREEN on the current head:** all **71** focused recovery, settlement, heartbeat safety and refresh-worker cases pass on `5762dfbe4e` with Node 24.21.0 and PostgreSQL 18. They cover incompatible identities, company/configuration mismatches, expired or missing deadlines, unfinished commands, Stop, retained files and duplicate resumption claims. - **Real process acceptance on the current head:** the three-turn `finalizer-wait` case passes in 58.01 seconds. PostgreSQL, TLS, a rebuilt Linux Runner and the deterministic Codex app-server fixture confirm zero attempts and no provider turn during the wait, one claim resuming the same run, retained history, a healthy third warm turn and final suspension. The sibling `warm-suspended` case from master also passes, in a fresh container, in 46.02 seconds. Both runs use `5762dfbe4e`; the artifacts were rebuilt from that checkout with the repository's pinned Rust 1.97.1 toolchain. - Acceptance used Node 24.21.0 and glibc 2.41. Earlier probes exposed the missing fixture API listener and the preceding retained-sandbox guard; both are corrected. A stale disposable listener and an incompatible libc image failed setup before the successful fresh-container run. These setup failures are not the original regression's RED. - Repository typecheck/build passed locally before the rebase. Server typecheck passes after the rebase. **Full CI passes on the current head:** 48 successful jobs and one expected scope skip, including typecheck, build, Runner checks, all serialized-server shards, general tests, E2E and the final aggregate. Module-boundary and whitespace checks also pass. - Broad local inventories are incomplete: six chat shards passed (1,834 tests), but untouched server/UI/adapter cases hit timeouts. The exact shared-base expiration case passes alone; a full-file rerun passed all 89 assertions but timed out in teardown. A Node 24 rerun timed out in another lifecycle case and teardown. A later PostgreSQL recheck could not allocate shared memory; this task's redundant broad controllers were stopped and their logs retained. - CI attempt 1 failed a Cursor Cloud cancellation deadline, an E2E cleanup cancellation with a PostgreSQL lock timeout, and the refresh-worker startup settling limit. One failed-only retry passed Runner and E2E; the same refresh startup case failed again. Two aggregate jobs had previously stalled in the hosted-runner queue and blocked the retry; that obsolete attempt was cancelled. No repeated blind rerun is being used. - **Readiness repair RED/GREEN:** real PostgreSQL 18 behind a loopback proxy with 20 ms delay per direction reproduces the exact startup error before the test change; the same probe passes afterward. An initial controlled probe hit the dedicated-database naming guard before tests ran and is excluded from RED evidence. The full-file check then exposed a separate false-idle barrier: a worker could be running while its scheduler had no timer. The fixture now also checks the existing active-work tracker. All **24 cases pass** with both repairs. The at-most-six scoped SQL reads and zero recurring idle-SQL assertions are unchanged; startup alone has an explicit bounded condition wait. No production behavior or timeout changes. - Greptile and Paperclip Review are both **5/5 on the current head**, with no open findings. Both reviews used source inspection; the runtime evidence above was obtained separately. The previous review workspace was recovered on the first production attempt after repairing its partial-clone export. Its preserved result and real follow-up both finalized successfully, saved snapshots and released their writers and finalization leases. Three current-head CI shards received hosted-runner shutdown signals and were cancelled. All three passed their one justified retry. The queued aggregate had to be cancelled to schedule that retry; the final aggregate also passed. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41706617909). - This acceptance does not qualify a live Codex model or Daytona API run of the native repair. No deployment or recovery of the original task is claimed. ## Risks - A wait must not grant a second executor. The current attempt stays at zero until the existing scheduler claims it. Expired or missing deadlines retain their existing hold. - Saved state must survive the wait. Admission still requires the same company, task, session, configuration and physical sandbox evidence. - Current Stop, budget, credential and permission gates remain with ordinary resumption. No polling loop is added. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact serving model ID or context-window size. Capabilities: reasoning, repository inspection, code editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>