mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
DottaandPaperclip
aaea05e158
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps provider history when a warm session ends. > - Object storage must preserve that history before the sandbox is released. > - A completed turn releases its live controller lease. > - The backup owner lookup rejects that released lease and stops the next task preparation. > - This pull request uses the existing completed-run checkpoint fence for backup storage. > - The next turn can prepare after the previous provider state is saved. ## Linked Issues or Issue Description Refs: #18932, #18011, #18984. **What happened?** A warm Codex turn completed and kept its provider session alive. The next turn stopped during preparation with `native_harness_storage_owner_missing`. The backup lookup required a live lease after normal completion had released it. **Expected behavior** Save the completed session under its retained controller generation and committed result. Keep the existing owner and successor checks. Do not start another provider turn during the save. **Steps to reproduce** 1. Run a warm native session with object-backed provider backups. 2. Commit the first turn and release its controller lease. 3. Start the next task preparation and retire the previous session. 4. Observe the backup owner error before provider work starts. **Paperclip version or commit** The regression is present at `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** A hosted server with a remote native Runner and S3 storage. ## What Changed - Accept a released backup owner only when its exact controller record has a committed result. - Reuse the checkpoint owner policy for same-controller preparation and native successors. - Recheck the controller generation, committed result, and released state after upload. - Preserve live-owner rejection and the rule that a late upload cannot replace a newer conversation backup. - Add real PostgreSQL storage and restore regressions. Document the completed-run backup rule. ## Verification - On 2026-11-04, two new regressions failed on the unchanged source with `native_harness_storage_owner_missing`. These covered an idle completed session and a same-controller preparing successor. - After the fix, 54 selected tests passed: 30 durable backup tests and 24 existing early-retirement checkpoint tests. After the fixture-only upstream sync, all 58 selected storage, checkpoint, process, and affected fixture cases passed again. - The storage cases use real PostgreSQL, encrypted archives, object transfer, and restoration. They cover invalid completion evidence, foreign or expired successors, active predecessors, changed generations, changes during upload, and newer backup publication. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'released|backup|early retirement|durably retires an accepted warm Runner'`. - An independent source review found no outstanding code issue. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs` before the upstream fixture-only sync. Production source is unchanged across that sync. - An assembled process probe passed with the Runner binary built from this head and a fake Codex protocol peer. It started real processes, committed a result, released the source lease, claimed the next controller, and completed the source stop, drain, and suspend commands. It recorded one provider turn start and kept the task bound to its successor. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'durably retires an accepted warm Runner'`. - CI exposed the old allocation-readiness test setup in three cases. The branch now includes the merged #18984 fixture correction. The same three cases passed locally after that sync. - An earlier CI test shard was cancelled after its runner received a shutdown signal. Its log was retained before one rerun. Another old-head shard reported failure with no retrievable job log or annotations; its cause remains unknown. Current-head CI must pass independently. - Local `pnpm build` passed at this head. All 1,904 chat cases passed across the supported local shards; the earlier shards used byte-identical source before the fixture-only sync. The interrupted default command is retained and is not reported as a pass. - The local bulk server run had separate enrollment acknowledgment, environment-run status, monitor-settle, idle-history teardown, and repository-rehome failures. In isolation, idle-history passed all 75 tests and the unchanged repository cases passed with their original limits (4.9s large files, 6.4s refs). The remaining four local failures reproduced on unchanged master `04a24f4e06` under the same environment and limits. Their causes remain unproven. Logs and the baseline comparison were retained; the local full suite is not claimed as passed. - Current-head CI first hit the unchanged repository-rehome large-file case's 15s timeout. The same isolated case passed in 4.9s. After source/call-path comparison, one bounded failed-shard retry passed all 115 executed suites; no assertion or timeout was changed. The original timeout remains recorded. - Local verification uses macOS 26.5.2 arm64, Node 26.4.0, pnpm 9.15.4, PostgreSQL 18.1, and Cargo 1.97.1. The current tested head is `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`. - The full unfiltered [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/41868587974) passed at `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`: all 12 general server, 3 chat, 9 serialized, workspace, native Runner, build/typecheck, E2E, and final verify lanes. The file selector was skipped, so this was the complete fallback matrix. - Greptile reports 5/5 on this exact head with no open review threads. Disposable local test processes were cleaned up and the source tree is clean. - The real hosted Codex and S3 workflow has not been rerun because the local verification environment has no remote provider credentials. The original task remains unchanged. ## Risks The main risk is a stale controller publishing provider history after a new owner starts. Both storage transactions keep the company, run, boot, and generation fence. A released owner must keep its committed result. Existing successor checks and the newer-backup publication guard remain in force. The live checkpoint writer still requires a non-null lease. No migration or API change is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks; separate baseline broad-suite failures are disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>