mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
DottaandPaperclip
e3fb4bc73a
## Thinking Path > - Paperclip manages AI agents and their work. > - Native recovery must preserve saved work and respect current task authority. > - A closed local API listener can stop startup before a provider starts. > - #19462 gives new failures a bounded bootstrap retry. > - Older saved failures still have a generic terminal code and a recovery hold. > - This pull request verifies the old producer outcome and uses the existing safe replacement path. > - Eligible tasks continue once. Unknown provider outcomes and user holds remain blocked. ## Linked Issues or Issue Description Refs: #19462 **What happened?** A restart can leave an old listener startup failure in `terminal_failure` with `native_session_interrupted`. Startup recovery does not claim it after #19462. A completed file recovery can also retain its no-replay hold. **Expected behavior** Continue automatically when durable records prove that no provider started, the source stopped, and the current files were saved. Keep the existing attempt budget and normal admission checks. **Steps to reproduce** 1. Close the actual API listener after a fresh native coordinator claim. 2. Persist the old executor's generic failure. Save the workspace and confirm source termination. 3. Restart recovery. The baseline schedules no successor. The repair schedules one successor after all receipts pass. **Paperclip version or commit** Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`. **Deployment mode** Built from source. Local macOS arm64, Node 26.4.0, real embedded PostgreSQL, independent filesystem execution. No external AI service is used by the controlled acceptance. ## What Changed - Recognize the exact saved local-listener guard through one proof predicate shared by discovery and the locked transaction. - Require a fresh first attempt, matching execution bindings, no prior provider lineage, no provider events or tool calls, no checkpoint or admission receipt, and no cancellation. - Reuse current snapshot, exact stop, source block, attempt budget, and successor idempotency checks. - Wait for completed source file-recovery receipts. Retire only that same source's verified preservation hold in the successor transaction. - Bound discovery to 100 candidates per pass, keep completed history quiet, and wake through existing transactional producer notifications. - Retain retries after transient database and final ownership-proof errors; advance the bounded cursor and force a rescan so a failing page cannot hide later work. - Let only the replacement duty discharge a proved-empty notification during drain through its existing read-only readiness check, once per notification generation. Keep real work and concurrent changes outstanding without running replacement during drain. - Wake replacement at canonical local-stop proof writes and the existing dispatch eligibility boundary; these hints grant no new execution authority. - Add database regressions, scheduler quietness checks, real controller handoff coverage, and an opt-in controlled Runner continuation. ## Verification Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`. Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest 5.0.3, frozen-lockfile dependencies. Tests execute freshly; `--fsModuleCache` reuses only content-validated module transforms. - Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18 seconds), including clean per-suite database disposal. All 62 assertions also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook timed out; that invocation is not counted as a pass. Docker's disk was full, so the disposable PostgreSQL 17 data used a size-limited RAM mount, with no host mounts. The owned container was removed afterward. - Actual stopped-process producer suite: 31/31 PASS. A real exited child produces the stop receipt, and its replacement notification settles only after the outer transaction commits. - Workspace and separate-process drain regressions: 33/33 PASS (75.90 seconds). - Shared coordinator and execution-control lifecycle checks: 39/39 PASS (16.43 seconds). Pending work, concurrent changes, cold startup, shutdown, and passive drain boundaries remain covered. - Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds), using the unchanged test and timeouts. Its matching local RED reproduced CI's 462-query, zero-pending-query, one-idle-hold timeout after an admitted write committed during drain (69.46 seconds). - Actual historical controller handoff: PASS (48.40 seconds). A separate controller loads the original executor from `2f35eb1080`, writes its generic listener failure, and exits. Recovery at the final head admits exactly one successor. The real Rust Runner and controlled Codex process complete governed `paperclip_finish`; attempt two commits and the saved file remains intact. Only the child loads the old artifact, and both isolated artifacts are restored afterward. Rust source subtree `77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No external AI service is used. - Seven corrected existing fixtures pass: transactional stop-proof calls (5 cases) and notification inventory (2 cases). Their earlier CI failures exposed fixture calls outside the documented transaction and an outdated expected queue set. - Server typecheck: PASS. Red/green proofs also cover the original saved failure on baseline `0252421d5c` (zero successors instead of one), held-history quietness, bounded forward scanning through 100 parked rows and concurrent wakes, real snapshot/cleanup producer notifications, actual issue-row lock timeout, and task/history eligibility notifications. An obsolete prior-run question can be skipped to request another proof check; actual skip while the task is paused or the source owns provider tool activity schedules no successor. The failed-page regression starts with 25 persistent verifier errors and a healthy candidate 26. Baseline RED repeats the first page. Repair GREEN advances the healthy candidate, revisits the original failures after their fault clears, creates all 26 successors exactly once, and becomes quiet. Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21 UTC). Final-head CI is GREEN: 56 successful checks, three intentional skips, one neutral result, and no pending or failing checks. [Attempt two and results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2) include passing build, typecheck, Rust, native server, package, browser E2E, serialized server, general server/chat tests, and final verification. Five jobs in the first attempt were interrupted together at 17:13:12 UTC; the native-server log records an explicit runner shutdown signal. The single infrastructure retry reran only those five jobs and dependent verification, preserving the 49 successful checks on the unchanged head. No timeout was weakened, and setup failures or interrupted invocations are not counted as passes. Focused commands from the repository root: ```sh ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false ``` The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1` and uses an original-executor child artifact with a `try/finally` restore. Normal regressions require no Rust binary. ## Risks - Compatibility is deliberately limited to positively proved fresh remote startup failures. Other old ambiguous failures remain held. - Existing provider work, a changed task block, incomplete preservation, or revoked authority must prevent execution. Normal retry admission remains in control. - This PR does not deploy the change or mutate a hosted task. Hosted recovery still requires the repaired revision and all current proof gates. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact backend model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>