mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
814fe5d20f5ea4cabb35cdd4ef19a16543f3e4df
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
814fe5d20f |
fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox start `runnerd` in the sandbox, and Paperclip connects to it through provider ingress and authenticates it with a one-use bootstrap ticket > - The ticket lives for 60 seconds, but Paperclip issued it before the remote launcher staged its files in the sandbox > - Under load, staging took 60 to 81 seconds, so `runnerd` presented an expired ticket and every connection was closed. The run failed with `runner_ingress_auth_failed`, and the task stopped behind a manual Retry > - This pull request issues a fresh ticket right before dispatch, retries these failures automatically when the controller checkpoint proves no runner ever authenticated, and adds spans that attribute the slow startup window > - The benefit is that a slow but healthy start authenticates, a failed start recovers on its own, and the remaining startup delay becomes measurable ## Linked Issues or Issue Description **What happened?** On a hosted instance, 9 runs of one Codex agent on Daytona failed in about 70 minutes with `native_session_interrupted` and the message `runner_ingress_auth_failed: Runner ingress did not complete PRP authentication before the deadline.` In the same window only 6 runs succeeded company-wide. The run log showed Paperclip reconnecting for 60 seconds while `runnerd` reported that its peer closed each WebSocket. In 294 remote runs with timing spans, the time between the bootstrap ticket issue (end of `runner.transport.connect`) and the process dispatch (start of `runner.process.launch`) decides the outcome without exception: - 60 s or more: 23 of 23 runs failed authentication. - Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s. The tasks did not recover by themselves, because the failure was classified as ambiguous state. **Expected behavior** A runner that starts slowly but correctly authenticates. If ingress authentication still fails before any runner authenticated, Paperclip retries the start automatically. The time between the start request and the runner connection is visible in run spans. **Steps to reproduce** 1. Run a Codex agent on a Daytona sandbox environment that uses provider ingress. 2. Make remote staging (`runner.runtime.stage`, inside the remote launcher) take 60 seconds or more. Many concurrent runs of the same agent do this on staging. 3. See that `runner.prp.authenticate` fails after 60 seconds and the run ends with `runner_ingress_auth_failed`. 4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts` reproduces this locally with a 75-second staging delay. **Paperclip version or commit** `4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`. **Deployment mode** Hosted (cloud), controller on Railway, sandboxes on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`), remote sandbox target, provider-ingress transport. ## What Changed - `RunnerProcessLaunchSpec` and `spawnRunner` accept `refreshBootstrapTicket`. A restart does not inherit it. - The runnerd Codex transport passes a refresh that issues a new bootstrap ticket and flushes the controller checkpoint. It does not do this for a warm-transition launch. - The remote sandbox launcher calls the refresh after `ensureArtifact` and immediately before the launch RPC, and puts the new ticket in the launch environment. - `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller checkpoint that no runner authenticated: no connections, no leases, no used tickets, no events, and only pending commands. - `nativeSessionFailureSourceCode` classifies the outbound ingress startup failures as `runner_ingress_startup_failed`. `nativeProviderRecoveryEvidence` treats it as pre-session (`bootstrap_retry`) only when that proof holds for the exact run identity and there is no provider session or provider event. Restart recovery and failed-run retry accept the new code with the same rules as similar startup codes. - Remote startup now records `runner.control_plane.open`, `runner.launch.prepare`, `runner.checkpoint.register` and `stage.asset.codex_skills.verify`. The 6 to 11 minute gap between `native.process_start_requested` and `runner.transport.connect` had no spans before. ## Verification - Red/green: `packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts` starts a real `runnerd` with a 75-second staging delay. Without the fix it fails with `runner never connected`. With the fix it opens the session. - Red/green: `native-session-executor.test.ts` "launches runnerd with a bootstrap ticket issued after remote staging". Without the fix the launch RPC gets the stale ticket. - Red/green: `native-controller-recovery.integration.test.ts` "retries a fresh remote start whose launched runner never authenticated through provider ingress" (embedded Postgres). Without the fix the failure is `native_session_interrupted` and `ambiguous_state`. - `durable-prp-control-plane.test.ts`: the new checkpoint proof, its negative cases, and the restart rule for the refresh. - `native-run-trace.test.ts`: remote startup spans are recorded without duplicate launcher spans. - Focused suites pass: runner package 5 files and 235 tests; server 9 files and 1,607 tests, including `native-session-executor`, `native-runner-execution-driver`, `remote-launch-assets`, `native-restart-recovery`, `runner-prp-outbound` and the recovery integration tests. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - A launch now issues two bootstrap tickets. The first one is never sent to `runnerd` and expires in 60 seconds. Bootstrap counters already allow more than one issued credential. - The new retry path needs a checkpoint proof that no runner authenticated, so a run that did provider work cannot be retried this way. The existing attempt budget still bounds the retries. - This change does not remove the 6 to 11 minute controller-side delay before `runner.transport.connect`. The new spans make it attributable for a follow-up change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
baa4a2e64e |
fix(runner): restore Codex auth before source recovery (#18612)
Restore disposable Codex authentication and safe provider configuration before cold source recovery. Preserve source receipts and conversation state until reconciliation, and rebuild assigned tools from the admitted launch binding. Validate remote launch destinations, clear stale authentication when no selected credential exists, and atomically publish SSH file transfers without following retained links. Red/green recovery regressions, a real two-turn Codex/MCP probe, and all current PR checks pass. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f8484a9f8d |
fix(native): deliver selected skills to remote Codex homes (#18080)
## Thinking Path > - Paperclip coordinates company agents and preserves their assigned instructions and approval rules. > - Native runs pin an immutable skill inventory for each execution. > - Built-in remote Codex launches prepare that inventory on the controller, but transfer only authentication, configuration, and the separate context assets. > - Codex needs the selected skills under its isolated home. An explicit skill path does not load a skill outside its registered discovery roots. > - This pull request transfers the sealed selected inventory to the path already used by explicit skill inputs. > - Each launch replaces stale assignments inside the authorized workspace while preserving provider history. ## Linked Issues or Issue Description Refs #18025. The corrected first-task instructions exist in the selected bundle, but remote Codex must receive those bytes before it can follow them. The legacy skill resolver work in #16646 and #9323 uses different paths and does not repair native remote launch staging. **What happened?** A fresh remote native Codex run could not read its assigned first-task skill. It produced an approval question instead of the required confirmation. The Product E2E journey correctly rejected that question before child creation. **Expected behavior** Codex receives the selected immutable skill at the same provider-local discovery path used by explicit skill input. Removed or unassigned skills do not remain available after a new launch. **Steps to reproduce** 1. Prepare an isolated Codex home with one assigned skill. 2. Run the built-in remote launch staging path against a fresh remote filesystem. 3. Resolve an explicit invocation of that skill. Before the repair, its provider-local file is missing. ## What Changed - Transfer only the prepared selected skill inventory into the isolated remote Codex home before other launch writes. - Reject writable files, symlinks, missing entries, and extra unassigned packages in the controller projection. - Validate each provider-home path component below the authorized workspace. Reject redirected retained paths before deleting stale skills. - Create the private discovery root before native sandbox transfers, which omit the source-root metadata and can ignore the requested directory mode. - Replace stale or empty inventories without changing provider conversation history, preconfigured homes, other providers, or plugin-provided launch preparation. - Exercise the real home materializer and turn resolver through the existing test-only Runner surface. Document the remote discovery contract. - Synchronize an existing Telegram attachment-failure test with the durable provider-effect commit, preserving its required final states. This CI repair is a separate test-only commit. ## Verification - The new production-staging regression failed before the repair because the resolved skill file was absent. - All 23 focused staging checks pass. They cover delivery, file modes, empty/revoked inventories, history retention, extra packages, writable files, source symlinks, target symlinks, redirected ancestors on sandbox/SSH, and an authorized workspace alias. The filesystem fixture packs and extracts actual tar bytes without applying the requested root mode; it preserves exact projected-file digests and executable bits. - The unchanged materializer and prepared-launch contracts passed 26 tests. Eight selected native caller checks passed; other cases in that filtered run were not selected. - Independent probes used official Codex 0.160.0, the real materializer, production staging, the real resolver, and a local model endpoint. The original staging supplied no skill message. The final staging supplied the complete selected skill body with its exact SHA-256. A final probe also ran production Daytona `performSyncIn`, backing only its SDK exec/upload/chmod edge with the local filesystem under umask `022`: the pre-review source reproduced root `0755`, and the final source retained root `0700`, package `0555`, file `0444`, and the exact selected digest. All model requests returned local 403; there were no upstream requests or credentials. - Full local `pnpm -r typecheck` and `pnpm build` passed on verification commit `e7d4e7a9`, based on `60b8089a`. The full local `pnpm test:run` on that frozen tree was not completed: later embedded PostgreSQL startup failed because the host could not allocate semaphore sets (`semget`, SEMMNI/SEMMNS), and the owned attempt was stopped with its log preserved. Its earlier four caller-fixture failures and one workspace-conflict failure are recorded separately below; the later resource error is not an explanation for them. Delivery commit `567d838d` replayed the initial six-file patch onto `ee67d2ec`; production was last changed in `23456f51`, followed by test-only commits `002fa4a7` and `6e8fa595`. All 24 staging checks, all 26 materializer/prepared-launch checks, and the complete native executor file (968 tests, zero skipped) pass locally. Server typecheck also passed after these test edits. The complete CI matrix passed on `002fa4a7`; current-head CI is tracked separately. - Initial-head CI exposed four caller fixtures that manually wrote auth/config without preparing the skills inventory. The same four failures reproduced locally. Those fixtures now use the real home preparer for their existing empty inventory and assert its transfer; all 34 retained/replacement scenarios and the complete 966-test caller file pass. A sandboxed full-file attempt could not bind local PostgreSQL (`EPERM`); it was stopped with its log preserved, then the identical command passed with local fixture access. Initial-head CI also recorded timeouts in workspace recovery, approval-queue backoff, and heartbeat attribution. Their three complete files pass unchanged locally (102 tests); the next CI head also passed those cases. The original slowdown remains unexplained. The consequent approval cleanup foreign-key failure is a timeout teardown race already recorded in the owning worker documentation. - The frozen full local run also observed an existing workspace-conflict test fail. The test and owning workspace code are unchanged between the verification and delivery trees, and the isolated delivery case passes. The original failure remains unresolved: stopping the later resource-blocked run prevented the final assertion detail from being emitted. That isolated pass does not replace the earlier result. - CI on `23456f51` caught a Telegram test reading `provider_effect=processing` after observing the provider mock call. Production commits `processed` later, so the test now waits for the unchanged complete durable-state assertion. All 18 selected Telegram attachment/media cases pass (the other 1,670 cases were not selected). The same CI attempt also recorded an E2E runner shutdown followed by cancellation, without a failed Playwright assertion; the necessary new test-only head reruns the matrix. Earlier failures remain preserved and are not reported as passes. - Review requested an executor-level preconfigured-home negative test. Retained and replacement workspace cases now drive the actual launcher, assert successful context staging, and exclude the isolated home's skills/auth/config transfers. A counterfactual that removes only the executor exclusion makes both tests fail; the exact production bytes were restored before the complete 968-test pass. Flat and nested selected package delivery is also covered. Optional arbitrary remote-stderr expansion is deferred; the repair keeps the existing closed error code and report surface. - Final head `6e8fa595` passes the complete CI matrix, including typecheck, build, Runner, browser, and aggregate gates. Two first-attempt jobs were interrupted by runner shutdown/lost communication; one unchanged-head rerun of those jobs and their aggregates passed. The underlying host interruption cause remains unknown. Both current-head automated reviews are 5/5 with the implemented finding resolved. - The original checked-in lockfile fails a fresh frozen install on the baseline. The repository's maintained resolution-then-frozen sequence passes. No lockfile change is included. - Independent source review found and then verified the retained-ancestor guard. This PR does not claim that the full live Cloud onboarding campaign has passed. It changes no journey assertion or approval gate. ## Risks Launch preparation adds an asynchronous inventory walk, one bounded remote cleanup command, and one transfer of the selected skill tree. Those costs grow with selected files and bytes and occur only at provider launch. A failed validation, cleanup, or transfer prevents launch. Remote Node is already required by the shared SSH/sandbox launcher. The selected packages remain read-only; the private discovery root retains Codex's normal writable mode. No database, API, credential-routing, or deployment change is included. ## Model Used OpenAI Codex based on GPT-6, with reasoning, source inspection, tool use, and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c91f7a6e5f |
perf(runner): overlap verified sandbox launch assets (#16585)
Native sandbox launch preparation stages fresh harness files and immutable runtime context in sequence. Overlap them only under the verified concurrent-sync capability, settle every started transfer before launch or cleanup, and preserve serial SSH behavior and error priority. A matched real-Daytona comparison reduces staging from 1,552 ms to 1,017 ms. The larger active-run difference includes model variation and is not attributed to this change. Add an explicit single-attempt diagnostic and synchronize stale Pi evidence metadata with its shipped profile. Validation: 764 native runner/tracing cases, 2,764 support cases, repository typecheck/build, fresh exact-head full CI/E2E/security and Apex 5/5 pass. Observability remains a silent no-op when no endpoint is configured. Co-Authored-By: Paperclip <noreply@paperclip.ing> |