mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
DottaandPaperclip
db11d899cf
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents keep remote processes and conversation history across warm turns. > - A completed turn can leave handoff authority only in controller memory. > - Losing that controller can block a follow-up even while its sandbox survives. > - This pull request verifies durable ownership and reconnects the exact prior process before handing it to the next run. > - Healthy follow-ups stay warm. Recovery preserves saved work without replaying a completed turn. ## Linked Issues or Issue Description **What happened?** A remote warm turn can finish without suspended backup authority. A replacement controller rejects that prior owner even when its exact process and completed checkpoint remain. A single unavailable identity probe can also stop an otherwise valid reattachment. **Expected behavior** Keep healthy follow-ups warm. Recover a completed session after proving that its controller stopped and that the successor holds current authority. Reuse compatible processes; otherwise save and resume the conversation. Retry unavailable identity reads within one shared deadline. **Steps to reproduce** Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute real turns through PostgreSQL, TLS ingress, and the Rust runner. They kill a separate controller process, recover old checkpoint metadata, and change process configuration. The provider is deterministic. Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds recovery diagnostics. #14695 covers provider process retention. ## What Changed - Preserve warm process reuse between healthy remote turns. - Verify a committed prior result, stopped controller, matching checkpoint and sandbox, and current task/coordinator ownership before recovering an orphaned warm session. - Reconnect through the prior run route, then use the existing authenticated run attachment to change authority. - Persist a process-configuration digest. Close and save recovered sessions when old or changed configuration cannot support reuse. - Retry unavailable state, marker, and process reads within one 30-second deadline. Keep signals and definitive identity failures single-attempt. - Record closed recovery diagnostics, including unreadable or malformed saved state. - Retire the recovery-only ownership check after verified attachment, so later warm turns use their current authority. - Add real-process Linux regressions and refusal controls. - Expose the existing source-only file-recovery form for retained native holds, with the same board authority, evidence, and exact stop-receipt requirements as legacy holds. Recording recovered files does not change task status or replay a run. ## Verification - `pnpm -r typecheck` passed; the server check was repeated for the third-turn fix. - `pnpm build` passed after the final UI change. The UI typecheck, token gates, and map checks also pass. - Four Linux continuity cases passed: warm reuse, controller loss, legacy checkpoint metadata, and changed configuration. Each creates one conversation and starts exactly three turns, including a healthy follow-up after recovery. The fixture uses expired prior leases; the third turn keeps the runner and provider processes. - Greptile caught the retained recovery gate; the third-turn regression now covers it. An initial test omitted the heartbeat’s saved input and left a test process after failure; the fixture was corrected and its disposable container reset before the passing run. - The current focused executor/restart group passes all 749 tests, including the original failure classifications. - Native file-recovery and UI tests pass (52 tests), including blocked tasks and both native and legacy holds. - After merging current master and retaining its equivalent CI expression check, the CI policy tests pass (15 tests) and all four three-turn Linux continuity cases pass again on `dec418ae55ff21b4fb067132613658cce445f181`. - `pnpm test:run` completed its general server group with 20,572 passes, 103 skips, and three failures: the skill-audit child process and two HTTP test timeouts. The same three suites then passed in isolation (89 tests) without code changes. That local command stopped before later groups; the complete final-head CI, including general, serialized, Runner, build, and browser gates, is green. - Greptile reviewed the combined head `dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable findings; both earlier findings are resolved. All final-head CI gates and the security scan passed. Staging deployment and retained-incident disposition remain deployment acceptance work. ## Risks - Recovery requires exact durable evidence. Live conflicting controllers, replacement sandboxes, Stop/pause, and uncertain effects must remain blocked. - After controller loss, tool-contract or process-configuration changes can restart the provider. Normal healthy follow-ups retain their processes. - The new idle-recovery path has Linux fake-provider evidence. Live Codex/OpenCode and retained quarantined-state recovery still need staging qualification. - No database schema, telemetry, or runner protocol changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution, and browser tools. The session does not expose a more specific model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks and isolated reruns; full-run limitation recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `dec418ae55ff21b4fb067132613658cce445f181` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
68 lines
3.1 KiB
Markdown
68 lines
3.1 KiB
Markdown
# Native remote continuity regression
|
|
|
|
The opt-in case in
|
|
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
|
|
runs the real executor, PostgreSQL, TLS ingress, Rust runner, remote file transfer,
|
|
and backup validation against a deterministic fake Codex process in Linux. It
|
|
checks the same runner process identity and one provider process across three warm
|
|
turns, provider history reuse, and verified suspension at idle close. Separate
|
|
cases kill the actual controller process and recover the next turn. They reject
|
|
a still-live controller and a replacement sandbox before reconnecting. Legacy
|
|
checkpoints and changed process configuration require verified suspension and
|
|
replacement, while preserving the conversation and starting each turn once.
|
|
Recovery can restart a provider to refresh its tool contract; process reuse is
|
|
required for ordinary healthy follow-ups.
|
|
Each recovery case includes another healthy turn after the recovered turn and
|
|
asserts that it does not start another runner or provider process.
|
|
It does not qualify a live model or a sandbox provider API.
|
|
|
|
Use a disposable Linux container with Node.js, `sh`, `tar`, `base64`, `ps`, and
|
|
`setsid`. Build `paperclip-runnerd` and `fake-codex-app-server` from the checkout's
|
|
`packages/paperclip-runner/runner/Cargo.toml` for that container's architecture.
|
|
Keep the Linux runner binary on the host as well: the product hashes and stages
|
|
that artifact. Put the fake provider at `/opt/continuity/fake-codex-app-server`
|
|
inside the container. Do not put credentials in this container.
|
|
|
|
Start the container with `--init` and publish container port 43127 on localhost.
|
|
Read its assigned port with `docker port CONTAINER 43127`; an automatic host port
|
|
can change after a container restart. The test opens a local TLS tunnel to this
|
|
port. Create a test-only certificate and key:
|
|
|
|
```sh
|
|
openssl req -x509 -newkey rsa:2048 -nodes \
|
|
-keyout /tmp/continuity-test.key -out /tmp/continuity-test.crt \
|
|
-days 1 -subj '/CN=localhost' \
|
|
-addext 'subjectAltName=IP:127.0.0.1,DNS:localhost'
|
|
```
|
|
|
|
Create an executable wrapper on the host at `/tmp/continuity-fake-codex`:
|
|
|
|
```sh
|
|
#!/bin/sh
|
|
if [ "$1" = '--version' ]; then
|
|
echo 'codex-cli 0.999.0'
|
|
exit 0
|
|
fi
|
|
exec /opt/continuity/fake-codex-app-server \
|
|
--state-file-in-codex-home --durable-turn-ids --record-process-start \
|
|
--call-log "$PWD/provider-calls.log" "$@"
|
|
```
|
|
|
|
Run the test with the container name, current host port, and Linux artifact path:
|
|
|
|
```sh
|
|
NODE_EXTRA_CA_CERTS=/tmp/continuity-test.crt \
|
|
PAPERCLIP_TEST_RUNNER_TLS_KEY=/tmp/continuity-test.key \
|
|
PAPERCLIP_TEST_RUNNER_CONTAINER=CONTAINER \
|
|
PAPERCLIP_TEST_RUNNER_PORT=HOST_PORT \
|
|
PAPERCLIP_TEST_LINUX_RUNNER_BINARY=/path/to/linux/paperclip-runnerd \
|
|
PAPERCLIP_TEST_FAKE_CODEX_WRAPPER=/tmp/continuity-fake-codex \
|
|
pnpm exec vitest run \
|
|
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts \
|
|
-t 'preserves remote warm continuity' --maxWorkers=1
|
|
```
|
|
|
|
The caller owns the disposable container. Remove it after the run, including
|
|
after failure; failed cases can retain remote processes and files for inspection.
|
|
The test cleans up its database and host session directories.
|