mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
dfd2a6efcab84bc19106a2dbf720eb53cb2282d3
2447
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicky LeachandPaperclip
|
38501b1170 |
Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents open input images with a file reader. > - Plugin Runner tasks use an authenticated Runner connection. > - Runner needs a way to fetch an authorized file from the tenant app. > - The tenant grants access to the exact attachment for the active run. > - Runner downloads and verifies it before it returns a local path. ## Linked Issues or Issue Description Refs #19740. Builds on the attachment authorization and command transport added in #20939. ## What Changed - Add the PRP v3 `runner.attachment.download` command for native workspace providers. - Issue short-lived grants for exact authorized wake and historical attachments. Recheck the active run, source, size, and SHA-256 at the tenant endpoint. - Download current-wake inputs before provider dispatch. Download historical files when the authorized chat reader asks for them. Preserve SSH and sandbox command staging as a fallback. - Run HTTPS transfers off the Runner control loop. Report completion with one PRP event per transfer. Drop pending results on cancellation and clear files at turn settlement, authority rotation, and restart. - Grant restricted Codex sessions read-only access to the attachment directory. ## Verification - 69 targeted server attachment tests passed. They cover wake and historical downloads, authorization failures, and command fallback. - 22 focused historical attachment tests passed, including grant redemption under a held database lock. - Six focused Rust attachment tests passed, including a source-pinned Codex sandbox check. The sandbox could read the staged file but could not modify it or read adjacent Runner state. - The real Runner transport test confirmed one durable command and one completion event for a transfer. Restart-continuation tests confirmed fresh attachment paths appear before dispatch. - Server and Runner TypeScript checks, Rust formatting, and PRP generated schema checks passed. - Required GitHub CI passed on `7a33752bd7`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43122967837) - Greptile scored the same revision 5/5 with no unresolved review threads. [Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134) ## Risks - Saved provider and Runner identities keep their existing formats. A previous Runner falls back to command staging where the target supports it. A restart continuation refreshes temporary attachment paths before dispatch. - Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS access to the configured tenant origin. - Live agent-computer qualification remains a release check. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell tools, and local test execution. Exact variant and context window are unavailable in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #123` / `Closes #123` / `Refs #123` or (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1880526603 |
fix(runner): never start an OpenCode session without its Paperclip tools (#20964)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode agents use Paperclip tools through a local MCP bridge in the runner > - OpenCode 1.18 connects each MCP server one time per process. It does not retry > - One failed `initialize` or `tools/list` at startup marks the Paperclip server "failed" for the full session > - The agent then works with only built-in tools. In one chat it told the person "Switched the working design to blue", but no saved state supported this claim > - This pull request makes the runner confirm the Paperclip tools while the session loads, reconnect them, restart OpenCode if necessary, and fail the session by name if the tools stay unavailable > - The benefit is that an OpenCode turn never starts without its Paperclip tools, so the agent cannot silently claim actions that it could not do ## Linked Issues or Issue Description Refs #20886, #20921. **What happened** In chat eval `steering.steer-active` (private paid run 43086418488, on a commit with #20886 and without #20921), two of three OpenCode sessions started without Paperclip tools. One was the CEO chat (fresh session). One was the worker task after its saved session resumed in a new OpenCode process. The CEO agent ran `opencode mcp list` and saw `paperclip ✗ failed — Failed to get tools`. It then sent a request to the bridge without the bearer token and got HTTP 401. That 401 came from its own probe, not from OpenCode. The run still succeeded with the reply "Switched the working design to blue …", because a plain chat reply counts as the result. **Expected behavior** Every native OpenCode session starts with the Paperclip MCP tools. If the tools cannot be loaded, the session fails with a named reason, and the turn does not start without tools. **Steps to reproduce** 1. Start OpenCode 1.18.34 with a remote MCP server whose first `tools/list` request fails (for example, the server resets the socket one time). 2. `GET /mcp` returns `{"paperclip":{"status":"failed","error":"Failed to get tools"}}`. OpenCode keeps this status for the life of the process. 3. `POST /mcp/paperclip/connect` returns `true`, and then `GET /mcp` returns `connected` (about 8 ms). We measured this with the real 1.18.34 binary. This is the same class as #20886: `Connection: close` made the one-shot startup connect fail more often (about 9 of 25 sessions), and #20921 removed the trigger. Any other transient failure at startup still produced a toolless session. This change removes the class. ## What Changed - `opencode-server-driver.ts`: while a session opens (fresh, resumed or recovered), the driver reads OpenCode's `GET /mcp`. If the `paperclip` server is not `connected`, it calls `POST /mcp/paperclip/connect` up to two times. If the server is still not connected, it throws `paperclip_tools_unavailable`. The existing startup retry restarts OpenCode for this error (three attempts in total). After that, the session fails visibly. - The check runs in parallel with session creation or resume. The added work on a healthy start is one local HTTP request. - `fake-opencode-server.mjs`: the fixture now serves `GET /mcp` and `POST /mcp/paperclip/connect`. It connects to the real bridge with the configured token, and it can simulate failed connects. - New `paperclip-tools-startup.test.ts`: reconnect after one failed connect, restart after a process stays failed, reconnect on a saved session that resumes in a new process, and a named failure when the tools never load. All four tests fail on master (master opens a toolless session). ## Verification - `pnpm exec vitest run src/drivers/opencode/paperclip-tools-startup.test.ts` (in `packages/paperclip-runner`): 4 passed. Red on master: 4 failed. - All OpenCode driver, proxy, live runnerd OpenCode and native-runtime server suites that use the fixture: 29 files, 567 tests passed. - `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11 passed. `tsc --noEmit -p packages/paperclip-runner`: clean. - Real OpenCode 1.18.34 probe (failure, then `connect`, then `connected`) as in the steps above. - Paid chat eval run 43117909094 on this change (steering.steer-active and three other OpenCode chat cases). Result is in a comment on this PR. ## Risks - Low. A healthy start adds one local `GET /mcp` request, in parallel with session creation. - If OpenCode cannot load the Paperclip tools after reconnects and restarts, the session now fails with `paperclip_tools_unavailable`. Before, it ran without tools. This is intended. - No persisted run, session or pack state changes. A session saved by the previous release resumes through the same open path and gets the same check. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
23909077ad |
fix(retry): re-deliver the failed run's wake when the board retries it (#20970)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails, the board shows a Retry button. The Retry starts a
new run with the wake reason `task_retry`.
> - A failed run can serve a wake that happens once, for example a
scheduled task monitor check (`issue_monitor_due`). The monitor is
consumed when it fires.
> - The Retry run did not tell the agent which wake it retries. The
agent saw only `task_retry` and the earlier interruption.
> - In the restart-chaos harness, the agent followed its brief ("act
only on a board comment or a scheduled monitor check"), decided that the
Retry was not the monitor check, and changed nothing. The run succeeded,
but the task stayed `in_progress` with no run, no wake and no monitor.
The person clicked a Retry that worked mechanically but lost the work it
was for.
> - This pull request records the retried run's own wake reason on the
Retry's continuation trigger (`trigger.retriedReason`) and shows it in
the wake summary.
> - The benefit is that a Retry re-delivers the same wake that failed.
Dotta's rule: if a Retry button shows up, it still needs to work.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude, Daytona warm,
volume-free server mode, restart fault K1), task t03 (workload W2). The
phase-2 monitor wake started run `60d821af`. That run failed with
`native_session_interrupted`. The board Retry was accepted (HTTP 202).
Retry run `a5a3958e` started 2 s later and succeeded. Its context had
`wakeReason: task_retry` and `executionContinuation.trigger = { reason:
"task_retry", sourceRunId: "60d821af…" }`, with no reference to
`issue_monitor_due`. The agent's summary: "This wake was a task_retry
resuming the previously interrupted run, not a new board comment or
scheduled monitor check, so per instructions I changed nothing." The
issue stayed `in_progress` with `executionRunId = null`, monitor status
`triggered` and no `nextCheckAt`. The harness graded
`R3:retry_did_not_complete` and U1 `no_owner_no_action 182s`.
**Expected behavior**
A Retry of a failed run re-delivers that run's wake. A Retry of a failed
monitor check is handled as that monitor check.
**Steps to reproduce**
1. Give an agent a task that acts only on a scheduled monitor check.
2. Let the monitor fire, and make the monitor run fail (restart fault K1
during the run).
3. Click Retry on the task.
4. Before this change, the Retry run's prompt says only `reason:
task_retry`, and the agent does not do the monitor work.
**Additional context**
The same signature (`R3:retry_did_not_complete:W2C:K0`, Codex, monitor
rewake run `bac90594` failed `native_session_interrupted`, Retry run
`cfb24146` succeeded without the phase-2 marker) is in wave
`w-20261111T192658-s5201` task t06. That case was attributed to the
response-wake loop that #20935 fixed. Its shape matches this cause: the
Retry run served the monitor wake but did not do the monitor work. That
wave kept no database, so I could not read its run context directly.
Refs #20935
## What Changed
- `buildExecutionContinuation` adds `trigger.retriedReason` when the
current wake is a Retry (`task_retry` or `retry_failed_run`). The value
is the source run's own wake reason. A Retry of a Retry inherits the
first non-retry reason from the source run's saved continuation.
- `ExecutionContinuationEnvelope.trigger` gets the optional
`retriedReason` field.
- `renderPaperclipWakePrompt` adds `- retried wake: <reason> (this Retry
re-delivers the wake the failed run did not finish; handle it as that
wake)` to the wake summary for fresh and resumed sessions.
- New test in `server/src/services/execution-continuation.test.ts` on
embedded Postgres: a failed `issue_monitor_due` run, a Retry of it, a
Retry of that Retry, and a non-retry wake.
## Verification
- Red on master: `cd server && npx vitest run
src/services/execution-continuation.test.ts -t "carries the failed run"`
failed with `trigger` = `{ reason: "task_retry", sourceRunId }` and no
`retriedReason`.
- Green: the same file passes (45 tests).
- `tsc --noEmit` passes for `packages/shared`, `packages/adapter-utils`
and `server`.
- `pnpm verify` exits 2 (a shared type changed), so required CI supplies
broad coverage.
- Chaos: the existing cell W2 × K1 (Claude, Daytona warm) reproduces
this when the fault lands in the monitor rewake run. W2C (Codex) shows
the same shape. I did not rerun a wave for this PR.
## Risks
- Low risk. The change adds one optional field and one prompt line. It
changes no admission, lock, status or authority path.
- The field is computed only for Retry wakes. Other wakes are unchanged.
- Runs saved by an older version have no `retriedReason`. A Retry built
from them reads the source run's `wakeReason`, which older runs already
saved.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
d1e4109df1 |
fix(runner): keep a connected adopted Runner through a failed liveness probe (#20965)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, the controller adopts the remote Runner
that kept running in the sandbox. A monitor then checks the Runner's
liveness every 250 ms through the environment plugin.
> - #20940 and #20945 made a stalled plugin probe "unknown" for the
launched Runner's monitor, reattachment and launch. The adopted Runner's
monitor still failed the run on the first probe that failed without an
answer.
> - A short plugin stall after adoption therefore failed a healthy run
with `native_adopted_runner_identity_unverifiable`.
> - This pull request uses the evidence that is available: while the
adopted Runner holds its authenticated connection to the controller, a
liveness check that fails without an answer does not fail the run, for
at most the reconnect grace.
> - The benefit is that a healthy adopted Runner survives a short plugin
stall. A check that reports the process gone still fails the run at
once.
## Linked Issues or Issue Description
Refs #20940, #20945
**What happened?**
`#watchAdoptedRunner` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts` called
`failTransport("native_adopted_runner_identity_unverifiable: runner
liveness could not be revalidated")` on the first `isAlive()` rejection.
For a remote adopted Runner, `isAlive()` is a plugin
`environmentExecute` RPC. A plugin stall makes it reject, even while the
Runner holds its authenticated connection and works normally.
**Expected behavior**
A rejection is not an answer. While the Runner holds its authenticated
connection, the monitor keeps checking. The bound is the reconnect
grace. A check that returns `false` (the process is gone) still fails
the run at once.
**Steps to reproduce**
1. Adopt a remote Runner after a controller restart.
2. Stall the plugin worker, so that the liveness probe rejects.
3. Within 250 ms the run fails with
`native_adopted_runner_identity_unverifiable`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
d9d5958d09 |
fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path - Paperclip's controller and the sandbox Runner exchange numbered commands; the Runner refuses a command whose number is not exactly one past the last it processed. - Staging's most common "Runner gone" failure was a follow-up controller whose journal was behind the Runner. #20928 removed the cause on the server (a warm follow-up now launches from the journal its retired predecessor saved) and added a defence that reconciles a journal behind only by the predecessor's settled close. - The Runner still treated an out-of-order command as fatal: `durable/state.rs` returned an error and the process exited. A controller with a stale journal from any other cause (state saved by an older release, a mid-run deploy) therefore lost the Runner process and the run could only fail as "Runner gone" (`runner_remote_process_identity_unavailable`). - This pull request makes the Runner refuse such a command with a precise result instead of exiting, and makes the server rest that run with a precise code and a Retry whose next run reconciles the journal (the #20928 defence) before it opens the session. ## Linked Issues or Issue Description - Refs #20928 (the server-side root-cause fix and the journal reconciliation this relies on) - Refs #20907 (makes runnerd exit text visible in the run log) - Duplicate search: searched PRs for "controller sequence contiguous", "controller_sequence_mismatch" and "out-of-order command"; none changes the Runner's handling. ## What Changed - `runner-core/src/durable/state.rs`: a new command whose `controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap) now yields `CommandDisposition::Reject` with `status: "rejected"` and `{ code: "controller_sequence_mismatch", message, expectedControllerSeq, receivedControllerSeq }`. The command is not executed or journaled and the last sequence is unchanged; the process keeps running. Replay of an already-processed command and the compacted-history rejection are unchanged. - `native-session-executor.ts`: a failure carrying `controller_sequence_mismatch` is classified as `native_controller_journal_behind_runner`, a permanent (terminal) failure with a Retry action that says no provider work began and that Retry reconciles the journal from the saved checkpoint or stops with a precise error. - Tests: - Rust `tests/durable_recovery.rs`: out-of-order (behind and gap) commands are rejected with the precise code, the journal is unchanged, and the next contiguous command executes. The existing `command_gaps_and_identifier_reuse_fail_closed` unit test now asserts the rejection instead of an error. - Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a follow-up controller with a journal missing the predecessor's close now gets `"code":"controller_sequence_mismatch"`, the Runner process stays alive and its durable sequence is unchanged; with the current journal it resumes. - Server: the classification and terminal disposition of the new code. ## Verification - Red (master Runner): the Rust test panics on `expect("an out-of-order command is not fatal")`; the real-runnerd test receives `runnerd exited unexpectedly with code 1: paperclip-runnerd: controller sequence must be contiguous: expected 6, received 4`. - Green: `cargo test -p paperclip-runner-core --test durable_recovery` (4 passed); `cargo test -p paperclip-runner-core` lib suite (the only other failure, `task_memory::tests::monitor_bounds_notices`, is a timing test that passes alone); `npx vitest run src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built from this branch (2 passed); server `native-session-executor.test.ts` focused tests (71 passed). - `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`: profiles match the release manifest. No ACPX profile, attestation, pin or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the change is Runner source behaviour with an existing command-result shape, so no profile version or `commandDigest` changes. ## Risks - A controller that is *ahead* of the Runner (a gap) is now rejected instead of fatal. Neither case executes the command; the Runner journal is unchanged, so this is strictly safer than exiting. - The rested run needs a Retry. Before this change the same run failed after its retries as "Runner gone" with no precise cause. - Runner source changes the Daytona image content ID; a new image is needed for chaos waves on this revision. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
6a43890326 |
Fix image attachments in remote native workspaces (#20939)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs must receive the files that users attach to their messages. > - Remote wake preparation returned file metadata without readable bytes. > - The historical chat reader also rejected every remote target. > - Command-backed sandboxes and SSH targets already provide a bound command transport. > - This pull request stages verified files through that transport and clears them after the turn. > - Agents can then open attached images with their normal image reader. ## Linked Issues or Issue Description Refs #19740. This change covers SSH and command-backed sandbox execution, including Daytona. Plugin Runner tasks without a command transport still need a separate versioned file-transfer contract. This PR does not close that broader issue. The open attachment-recovery PR #20778 concerns provider-session attachment journals, not input files. **What happened?** A user uploaded an image and sent it to an agent. The remote native run received its filename and size, but no readable file. Wake preparation reported `remote_workspace_staging_unsupported`. The historical read fallback also refused remote execution. **Expected behavior** Copy the exact admitted attachment into the bound remote workspace. Let the provider read the image. Preserve company, task, source-comment, size, and hash checks. Clear temporary file contents when the turn ends. **Steps to reproduce** 1. Select a native agent on a command-backed sandbox or Linux SSH environment. 2. Attach a PNG to its current task or in-app chat message. 3. Ask the agent to inspect it. 4. Check that wake preparation supplies a readable workspace-relative image path. ## What Changed - Add a remote staging owner with bounded commands, stdin transfer, hash verification, opaque paths, and safe image extensions. - Confine Linux filesystem operations through directory descriptors. Refuse symlinks and replacement files during cleanup. - Bind current wake files and authorized historical external-chat reads to the selected lease transport. - Join active writes during cancellation. Retain failed cleanup work until scope close. - Tell providers to open image files with an image-capable reader. - Handle stdin stream errors in the shared SSH transport. Failed uploads return failure instead of raising an unhandled EPIPE or reporting success. - Add database, transport, cancellation, integrity, cleanup, executor-wiring, and Linux filesystem regressions. Document support and limits. ## Verification - Prepared an isolated source checkout with `node scripts/agent-preflight.mjs --skip-rust-debug`. - Focused suites: `pnpm exec vitest run server/src/services/native-runtime/remote-attachment-staging.test.ts server/src/services/native-runtime/native-runner-file-handoff.test.ts server/src/services/native-runtime/chat-attachment-read.test.ts` — 74 passed, 4 Linux-only tests skipped. - Executor checks: `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t 'attachment staging to the remote|stages from the authenticated|verifies frozen staged admission'` — 7 passed. - Server typecheck: `pnpm --filter @paperclipai/server typecheck`, followed by the server layered check after the final changes. - `git diff --check` passed. The local secret, PII, and private-reference diff scan found no matches. - `pnpm verify` returned exit 2 because the documentation change requires broad CI. No full local suite is claimed. - The four Linux filesystem tests were skipped locally because this macOS host has no available Docker daemon. Linux CI executed all 17 remote-staging tests successfully, including those four confinement tests: [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646266). - SSH regression: the real pipe test reproduced two unhandled EPIPE errors before the fix, plus false success when the peer exited with code zero. The fixed SSH tests and existing argv tests pass (12 tests). The remote-staging transport suite also passes (13 tests; 4 Linux tests require CI). - `pnpm --filter @paperclipai/adapter-utils typecheck` passed after the SSH change. - Final revision `1c5a9288b8d5f267ddeac84638ff6075dbbbf8ec`: all applicable CI jobs passed, including typecheck, build, general and serialized tests, Runner checks, canary dry run, and all eight browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43102035096). Linux CI also passed all 6 native SSH tests: [server shard 5](https://github.com/paperclipai/paperclip/actions/runs/43102035096/job/130796646349). - Greptile reviewed that final revision at 5/5 with no remaining findings. The review thread is resolved. GitHub reports the PR mergeable with a clean merge state. - No live model turn or deployed release is claimed. ## Risks - Remote staging requires Linux, Node, and a command transport bound to the selected lease. Plugin Runner tasks without that transport retain their unavailable result. - Each file uses bounded independent commands. Slow transports can delay preparation or cleanup. File bytes stay out of command arguments and provider-facing error messages. - An abrupt controller or remote-host failure can bypass cleanup. Ephemeral sandbox disposal remains the backstop. Retained files are never blindly overwritten or deleted. - Previous-release provider sessions and checkpoints retain their format and identity. New turns receive newly staged input files. Existing restart-admission checks still apply. No migration, runner-image change, or account-credential transfer is required. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, code execution, and tests. No subagents were used. The context-window size is not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
48d23d1530 |
fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.
## Linked Issues or Issue Description
Refs #20909 (chaos cell CR3, which reproduces this failure)
**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.
**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.
**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
884a9e19ea |
fix(native): launch a warm follow-up from its predecessor's saved journal, and restart a Runner that exited before session.open (#20928)
## Thinking Path - Paperclip runs agent work through a controller (server) and a runner (`paperclip-runnerd`) that exchange numbered commands; the runner refuses any command whose number is not exactly one past the last it processed. - Staging's most common failure is "Runner gone" (`runner_remote_process_identity_unavailable`, `native.remote_runner_recovery_failed` with `process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`). Chaos wave `w-20261111T192658-s5201` (Daytona, warm, Codex, W2/W2Q/W2C re-wake cells) captured it live: the new runner wrote `controller sequence must be contiguous: expected 13, received 11` (and `expected 5, received 3`) and exited 0.15 s after authenticating. - In all three captured failures the follow-up run started 1–2 s before the predecessor's warm session was closed (`transport_close_requested` for the predecessor arrives after the follow-up's `run started`). The close is issued by the follow-up's own admission. - Admission first materializes the controller projection for the session root. With an idle retained warm owner the root only holds the external-persistence marker, so `restoreWarmRunnerdController` copies that owner's saved checkpoint into the root. Later in the same admission, `executePaperclipNativeSessionWithinScope` finds the owner cannot be reused (configuration, process or GitHub access changed) and retires it. Retirement sends `runner.drain` and `runner.suspend`; both are saved to the owner's checkpoint before they are sent, but the root still holds the projection taken before the close. - The follow-up then opens its controller from that projection. Its journal is two commands behind the runner's durable `lastControllerCommandSeq`, so its first command repeats sequence 11; the runner treats this as fatal and exits, and every later attempt only reattaches to a process that is gone. - So the saved controller journal was persisted correctly; it was not selected: the projection is never refreshed after the retirement that extends it. ## Linked Issues or Issue Description - Refs #20874 (the runner-exit proof this extends from a rejected `session.open` to a Runner that exited before answering it) - Refs #20907 (makes the runnerd exit text, including `controller sequence must be contiguous`, visible in the run log; independent of this fix) - Refs #20853 (related: resumes a remote Runner that died at a settled `ready` cut; this PR covers the unsettled `ready` state left by an exit before `session.open`) - Duplicate search: searched open and closed PRs for "controller sequence contiguous", "runner_remote_process_identity_unavailable" and "Runner gone"; none fixes the stale controller projection. ## What Changed - `server/src/services/native-runtime/warm-controller-journal.ts` (new): - `refreshRetiredWarmOwnerProjection` reloads the retired owner's canonical checkpoint into the session root after retirement, only when the root still projects that owner and the saved journal extends the projected one. A diverged journal holds admission with `native_runner_checkpoint_projection_held:retired_owner_journal_diverged`; a projection changed underneath holds with `...:journal_changed`. - `reconcileControllerJournalWithRunner` is the defence for state already saved in the bad shape (old-saved/new-resumed, mid-run deploys). When the remote runner state shows its journal ahead of the controller projection and the runner's last command is a completed `runner.suspend`, it reconciles only if every extra sequence is a completed `command_close_drain_*`/`command_close_suspend_*` in the runner's processed-command journal and the predecessor's canonical checkpoint holds exactly those commands. Anything else refuses before launch with `native_controller_recovery_missing_evidence:controller_journal_behind_runner` instead of guessing. A runner whose last command is not a completed suspend is left to the existing live-adoption checks. - Defence in depth for the server side of the same failure (`native_session_recovery_failed: runnerd exited while waiting for session.open`, then `runner_remote_process_identity_unavailable` on every same-run retry): a Runner proven to have exited while this attempt's opening command (`session.open`, or the recovery probe) was still unanswered is a failed fresh start, not a live Runner to reattach. - `runnerd-codex-transport.ts`: `runnerExitConfirmed()` (#20874's proof) now also holds for a resumed session whose provider identity was preloaded, when the attempt's opening command is still pending at the proven exit. - The Codex driver's recovery result, `HarnessDriverBackend` and `native-session-runtime` carry that proof on the `native_session_recovery_failed` error. - The executor records `runnerExitedBeforeSessionOpen` on the settled failure; restart recovery puts `exitedBeforeSessionOpen` on the `reattach_remote_runner` claim; `verifyRemoteRunnerOrSettledExit` and `verifyStagedRemoteRunner` then start a new Runner when the process is gone and the saved Runner state has no pending terminal delivery, provider cleanup or warm transition (any lifecycle other than an active turn). Attempt budget and identity checks are unchanged. - `native-session-executor.ts`: calls the refresh right after the configuration-changed retirement of a warm owner, and the reconciliation in `restoreRemoteSuspendedRunnerCheckpoint` with the runner state it already reads (no added remote read, no wait before `turn.start`). - Tests: - `native-session-executor.test.ts`: new `changed-external-checkpoint` scenario of "rotates an owned warm codex remote runner before checking changed configuration" with a real `DurablePrpControlPlane`: the retired owner's close saves drain/suspend, and the journal the follow-up launches from must end with them. - `warm-controller-journal.test.ts`: refresh, divergence hold, untouched marker/other run, reconcile of a settled close, refusal when the canonical journal cannot prove the close, refusal for any other extra command, and no action for a live runner. - `packages/paperclip-runner/src/live/runnerd-stale-controller-journal.test.ts`: real `paperclip-runnerd` + fake Codex. A follow-up controller on the predecessor's suspended runner with a journal that lacks the close commands reproduces `controller sequence must be contiguous` and loses the runner; with the current journal it resumes. ## Verification - Red for the added part: `verifyRemoteRunnerOrSettledExit` with the staging state (process gone, lifecycle `ready`, unsettled outbox) rejects with `runner_remote_process_identity_unavailable` without the change; with `exitedBeforeSessionOpen` it resolves to a fresh start, and stays fail-closed for a pending cleanup, terminal delivery, active turn or exhausted budget. Without the propagation, the backend recovery result lacks `runnerExitConfirmed`; the real-runnerd stale-journal test asserts the transport's exit proof after the contiguity exit (`expected false to be true` on the old transport). - Red on master (executor unchanged, new scenario): `AssertionError: expected [] to deeply equal [ 'runner.drain', 'runner.suspend' ]` — the follow-up launched from the projection taken before the predecessor's close. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "rotates an owned warm codex remote runner before checking changed configuration"` (14 passed), `npx vitest run src/services/native-runtime/warm-controller-journal.test.ts` (7 passed). - Real runner: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-stale-controller-journal.test.ts` (stale journal → `controller sequence must be contiguous`; current journal → resumes). - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript`; `pnpm verify` selected broader verification (a new runner test file is outside the resolved graphs), so broad coverage comes from required CI. ## Risks - The refresh replaces the root projection only with a checkpoint whose identity matches and whose journal extends the projection; otherwise it holds admission with an existing `native_runner_checkpoint_projection_held:*` code, which is safer than launching a runner that will exit. - The defence refuses launch when a runner is ahead by anything other than a provable settled close. Before this change that launch always ended in a runner exit and a failed run, so the refusal only changes the error code and the moment of failure. - This PR does not change the runner. The runner still exits on an out-of-order new command; a follow-up runner PR will turn that into a rejected command result. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
d7b04a8b6d |
refactor(runner): rename codex-runtime-adapter to acpx-runtime-adapter (#20938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner opens agent sessions through one ACPX runtime adapter > - The adapter began as the Codex ACPX adapter, so its file is `codex-runtime-adapter.ts` > - The adapter now opens sessions for every ACPX agent: Claude, Codex, Pi, Cursor, Grok, Kimi, Copilot and Antigravity > - The Codex name makes readers look for Claude and other agent behavior in the wrong place > - This pull request renames the module to `acpx-runtime-adapter.ts` and changes no behavior > - The benefit is a file name that tells readers what the module does ## Linked Issues or Issue Description No issue exists. Refs #20933, which adds Claude restore behavior to this module. **What existing behavior does this improve?** The name of the shared ACPX runtime adapter module in Paperclip Runner. **Subsystem affected** Paperclip Runner, ACPX driver (`packages/paperclip-runner/src/drivers/acpx/`). **Current behavior** The module that opens ACPX sessions for all agents is named `codex-runtime-adapter.ts`. Its test file, which holds Claude, Pi, Grok, Cursor and Copilot tests, is named `codex-runtime-adapter.test.ts`. **Proposed behavior** The module is `acpx-runtime-adapter.ts` and its test file is `acpx-runtime-adapter.test.ts`. All imports, scripts and docs use the new name. **Reason and benefit** The name matches the module's scope. Reviewers and contributors find agent-specific session behavior faster. **Breaking changes** None. No exported name, behavior or contract changes. ## What Changed - Rename `src/drivers/acpx/codex-runtime-adapter.ts` to `src/drivers/acpx/acpx-runtime-adapter.ts`, and its test file in the same way. Git records both as renames with no content change, except the test file's import path. - Update the imports in `acpx-runtime-sidecar.ts`, `codex-acpx-driver.ts` and four test files. - Update the `dist` import in `scripts/grok-native-smoke.mjs` and the test file key in `scripts/vitest-lane-durations.json`. - Update the links and file names in `doc/RUNNER-INTEGRATION-PLAYBOOK.md`, `doc/architecture/runner-cursor-capabilities.md` and `doc/architecture/runner-rich-acp-capabilities.md`. - Do not change the historical profile identity fixtures under `packages/paperclip-runner/test/fixtures/`. They record the sources of earlier releases with the old path, and tests check them by hash. ## Verification - `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`: no errors in changed files. The only errors are in `src/eval/workflow-harness.ts`, because `@paperclipai/paperclip-eval-kernel` was not built in the local worktree. - `npx vitest run src/drivers/acpx src/cli/acpx-runtime-sidecar.test.ts`: 93 files, 1859 passed, 9 skipped. - `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11 passed. The pack contract digest reads protocol schemas, the Rust crate and the ACPX patch, not TypeScript sources, so the rename does not change it. - `node --check scripts/grok-native-smoke.mjs`, and `scripts/vitest-lane-durations.json` parses. - `git grep codex-runtime-adapter` finds the old name only in the historical fixtures. - Merge check with #20933: a local merge of that branch into this one completed with no conflicts. Git applied #20933's changes to the renamed files. The merged adapter tests passed (133 passed). ## Risks - Low risk. The change renames files and updates references only. - An open PR that edits the old file path must merge or rebase after this PR. Git rename detection handled #20933 with no conflicts. A PR that adds a new import of `./codex-runtime-adapter.js` fails typecheck until it uses the new path. - `openCodexAcpxRuntime` keeps its name. It is an alias of `openQualifiedAcpxRuntime`, and this PR does not change exported names. - `acpx-runtime-adapter.ts` and its test file were already longer than 1,000 lines. This PR does not add lines to them. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c1edb9ec02 |
fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - On staging, `runner_remote_process_identity_unavailable` was the most frequent failure. The tasks stayed blocked until someone clicked Retry > - In 31 of the 34 failed runs the reattachment found the Runner gone with an unsettled `ready` journal (`runnerSettled: false`). The Runner was not lost by accident. The controller's own close stopped it with `SIGKILL`: close could not prove suspension within its 10 s budget, so it force-stopped the Runner in the middle of a turn > - The server then scheduled a same-run reattachment that could never succeed, because the close had just removed the Runner it needed > - This pull request keeps an unreachable remote Runner alive when a failed run closes. A Stop, a cancelled turn and a `stop_cleanup` recovery still stop it > - The benefit is that the same-run reattachment finds the live Runner. The Runner then settles into a `suspended` journal, which the existing relaunch path resumes. The task continues without Retry, and no provider work is replayed ## Linked Issues or Issue Description Refs #20853, #20907 **What happened?** Remote native runs failed with `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. The tasks were left blocked. In the failed runs the first attempt failed with a recoverable transport error, for example `provider_transport_failed: PRP command turn.start timed out`, `native_session_interrupted` or a server restart. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The second string comes from `waitForProcess`, which sends `SIGKILL` when its deadline passes. The same-run reattachment ran 20–60 s later and reported `process_verification` / `process_not_running` with `runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs had done provider work before the failure (turns and tool calls), so the automatic Retry correctly refused them (`provider_turn_unsettled`). **What did you expect to happen?** When a run fails for a reason that the server resumes on the same run, the close does not destroy the Runner that the reattachment needs. The reattachment adopts it, and the run continues without Retry and without a repeated tool effect. **Steps to reproduce** 1. Start a remote native run and let a turn start. 2. Make the Runner unreachable (a broken transport or a slow event pump), so that the turn fails and the close cannot prove suspension within its budget. 3. The close sends `SIGKILL` to the Runner. The same-run reattachment then fails with `runner_remote_process_identity_unavailable`, and the task is blocked. **Relation to open PRs** - #20853 covers the 3 runs whose journal was a settled `ready` cut (`runnerSettled: true`). This PR does not change that path. - #20907 makes the Runner's exit reason visible. It does not change recovery. This PR does not overlap with it. ## What Changed - `runnerd-codex-transport.ts`: `close(reason, { retainUnsettledRemoteRunner })` accepts a new option. When the option is set, the Runner is remote (a process launcher or an adopted Runner), the close is not a `stop_cleanup`, the Runner is still alive and suspension is not proven, the close does not stop the Runner. It skips the `waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It still releases the controller route, writes the unsettled checkpoint and throws `NativeSessionCloseUnrecoverableError`, so the session stays quarantined and is never reused. The settlement diagnostic records `runnerRetained: true`, and a new diagnostic records `remote runner left alive unsettled for same-run reattachment`. - `HarnessNativeSession.close` asks for retention only when the session was not explicitly cancelled. After `cancel()` (Stop), it asks for the old behavior, so Stop still stops the Runner. - `CodexHarnessSession.close` passes the option to the transport. `HarnessSession.close` and `CodexAppServerTransport.close` accept the optional field. Callers that do not set it keep the old behavior. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md` describe the rule and the `runnerRetained` field. How the run continues: the reattaching controller adopts the live Runner. The Runner then completes the `runner.drain` and `runner.suspend` that the failed close queued, and exits with a settled `suspended` journal. That attempt fails as `native_runner_process_exited`, which the server resumes on the same run. The next reattachment finds the process gone with a settled `suspended` journal and relaunches the Runner over it (the existing path from #19745). If the Runner was only slow, it suspends by itself and the first reattachment relaunches. ## Verification - Red, on `master` with only the tests added: `runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5 (`expected 'gone' to be 'alive'`): the close killed the Runner. `harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2. - Green with the fix: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-remote-retention.test.ts src/backends/harness-driver-backend.test.ts`. - The staging shape: a real `paperclip-runnerd` started through a process launcher, a turn in progress, the Runner frozen with `SIGSTOP` during close. The close throws `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`, the Runner stays alive, and its journal stays `ready`. - A second controller attaches through `adoptExistingRunner` (no duplicate launch). The Runner settles into `lifecycle: suspended` with an empty outbox and no pending terminal delivery, then exits. - An explicit cancellation, a caller that does not ask for retention, and a `stop_cleanup` close still stop the Runner. - Also run: `src/backends`, `src/drivers/codex`, `src/drivers/acpx/codex-acpx-driver.test.ts`, `src/drivers/acpx/codex-runtime-adapter.test.ts`, `src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/` (files that timed out under full-parallel load pass at `--maxWorkers=2`). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The fix is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Medium. A failed close now leaves an unreachable remote Runner alive instead of killing it. This is the same state a restart detach leaves today, and the same-run reattachment or a Stop cleanup owns it. - If the server does not resume the run on the same Runner, the Runner stays alive until something reattaches or the sandbox is released. Remote Runners have no reconnect deadline (`runnerProcessReconnectGraceMs: null`). Before this change they were killed. The close still quarantines the session, so it is never released for reuse. - Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and `stop_cleanup` close without retention, so they still stop the Runner. - Previous-release story: this change adds and changes no persisted state. State that an older release left (a killed Runner with an unsettled `ready` journal) keeps its current handling: the named refusal `runner_remote_process_identity_unavailable` and a working Retry. Tasks already blocked on staging keep their Retry button. A deploy during a run changes nothing, because the decision is made in the closing process. - No wait is added before `turn.start`. The change only affects the failure path of close. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
e8e548e856 |
feat(db): classify migrations for application under the running release (#20913)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - `packages/db` owns the schema and its migrations. There are 467
migrations, and several ship each day
> - A zero-downtime deploy starts the new release while the previous
release still serves the same database
> - Core does not say whether a migration is safe to apply under the
previous release. A deployer must treat each pending migration as a
reason to pause, or a person must review it
> - On stacks that may not pause, releases wait for hours
> - Core owns the schema, so Core must own this knowledge and publish it
as data
> - This pull request adds a conservative classifier, an in-file
override, the published result, and a guard for new migrations
> - The benefit is that a deployer can apply routine compatible
migrations automatically, and incompatible changes become visible at
review time
## Linked Issues or Issue Description
**Subsystem affected**
Database migrations (`packages/db`), the cloud migrator artifact
(`scripts/cloud-migrator-artifacts.mjs`), and the pull-request policy
job.
**Problem or motivation**
Core has no machine-readable statement that a migration can be applied
while the previous release still reads and writes the database. A
deployer cannot tell an added nullable column from a dropped column. It
must pause for both, or refuse the release.
**Proposed solution**
Classify each migration as `online_compatible`, `requires_staging`, or
`unknown`. Publish the result in the migrator manifest. Fail a pull
request when a migration it adds is `unknown` and has no reviewed
override in the file.
**Alternatives considered**
- A hand-kept list in the deployer. It goes stale and it is not reviewed
with the SQL.
- A sibling JSON file in the artifact. It needs a workflow change, and
the attestation of `manifest.json` would not cover it.
**Roadmap alignment**
This is release infrastructure. It adds no product feature.
## What Changed
- `packages/db/src/migration-compatibility.ts`:
`classifyMigrationSql(sql)` returns `{ compatibility, classified,
reasons, override, overrideError }`. It parses statement by statement
(breakpoints, `;`, comments, string and dollar quoting, guarded `DO`
blocks). A statement that it does not recognize is `unknown`. A
migration takes the worst class of its statements.
- Lock and duration hazards have their own reason codes
(`lock_hazard_*`) and classify as `unknown`, not as compatible with a
warning.
- Override: `-- paperclip:migration-compat online_compatible:
<justification>` (or `requires_staging`) in the leading comment block of
the migration file. The justification is mandatory (10 characters or
more). A malformed, repeated, or misplaced directive is an error and
never improves the verdict.
- A table counts as new after a plain `CREATE TABLE`, or after `CREATE
TABLE IF NOT EXISTS` when no earlier migration created that table.
`classifyMigrationSequence` classifies migrations in apply order with
that knowledge.
- `packages/db/scripts/migration-compatibility.ts` (run with `pnpm exec
tsx` in `packages/db`): classify files, print the distribution, print
JSON, or run the guard (`--check`). No new package script.
- Published data: `manifest.json` of the cloud migrator artifact has a
new additive field `migrationCompatibility`. `version` stays `1`. No
existing field changes. The build reads the SQL from the packed
`db.tgz`, so the data describes what the migrator applies. `validate`
(publisher) and `verify` reject data that is malformed or does not match
the packed migrations. The workflow file does not change.
- Guard for new migrations:
`.github/scripts/check-pr-migration-order.mjs` (the existing "Validate
migration ordering against target branch" policy step) now also
classifies the migrations that the pull request adds. `check:migrations`
runs the same guard against the merge base with `origin/master`, so the
`@paperclipai/db` typecheck and build report it locally. `unknown`
without an override fails. `requires_staging` prints a notice and
passes. A new migration that uses `CONCURRENTLY` also fails when it is
not in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`
(`packages/db/src/client.ts`), because the migrator would run it inside
a transaction.
- `doc/DATABASE.md`: new section "Migration compatibility with the
running release" under "Schema change workflow".
Published shape:
```json
"migrationCompatibility": {
"version": 1,
"migrations": [
{
"idx": 468,
"tag": "0468_busy_wendell_rand",
"sqlSha256": "56a6e272…",
"compatibility": "online_compatible",
"classified": "online_compatible",
"override": false,
"reasons": [
{ "code": "add_column", "compatibility": "online_compatible", "message": "Adds a column that is nullable or has a constant default.", "statements": [1] }
]
}
]
}
```
`sqlSha256` is the hash that the migrator records in the migration
history. `overrideJustification` is present only when `override` is
`true`. Entries are in apply order (journal `idx`, then file name; the
shipped journal repeats `idx` 178).
### Distribution over the existing migrations
No existing migration is rewritten or annotated.
| Class | All 467 | Newest 100 |
| --- | --- | --- |
| `online_compatible` | 228 (48.8%) | 67 |
| `requires_staging` | 114 (24.4%) | 12 |
| `unknown` | 125 (26.8%) | 21 |
Migrations per blocking reason (one migration can have several):
| Migrations | Class | Reason |
| --- | --- | --- |
| 74 | `requires_staging` | `create_unique_index_existing_table` |
| 70 | `unknown` | `lock_hazard_create_index` |
| 66 | `unknown` | `lock_hazard_validated_foreign_key` |
| 56 | `unknown` | `lock_hazard_data_backfill` |
| 54 | `unknown` | `drop_constraint` |
| 41 | `requires_staging` | `add_validated_check` |
| 33 | `unknown` | `drop_index` |
| 16 | `unknown` | `do_block_procedural` |
| 15 | `unknown` | `replace_function` |
| 13 | `unknown` | `create_trigger_existing_table` |
| 12 | `unknown` | `data_change_existing_table` |
| 10 | `unknown` | `drop_object`, `lock_hazard_set_not_null`,
`set_default` (each) |
| 8 | `requires_staging` | `drop_column` |
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/migration-compatibility.test.ts`: 113 tests pass. They cover a
positive and a negative case for each rule, each reason code, the
override (missing, short, misplaced, and repeated justification are
rejected), unknown by default, the guard, the shipped migrations, the
`CONCURRENTLY` list, and the command line.
- `node --test scripts/cloud-migrator-artifacts.test.mjs
.github/scripts/tests/check-pr-migration-order.test.mjs`: 21 tests pass.
- Real bundle at the first commit of this branch: `preview-artifacts.mjs
pack`, then `cloud-migrator-artifacts.mjs build`, `validate`, and
`verify-install` pass. The manifest lists 467 migrations and is 312 KB.
This run found the repeated journal `idx`, which the second commit
handles.
- `pnpm --filter @paperclipai/db run check:migrations` passes. With
`--check --base HEAD~150` the guard fails on three `unknown` migrations
and prints notices for two `requires_staging` migrations.
- `pnpm verify` exits 2 because `.github/scripts/**` is a global CI
input. Required CI supplies the broad coverage.
- Not run: the publish job (it needs the master-only AWS role), and no
deployer reads the new field yet.
## Risks
- The manifest grows from about 1 KB to about 312 KB. A consumer that
rejects unknown manifest fields would fail. The verifier in this
repository ignores unknown fields; consumers outside this repository
were not inspected.
- From this merge, an open pull request that adds an `unknown` migration
fails the policy step and the `@paperclipai/db` typecheck until the
author rewrites the migration or adds the override. `drizzle-kit` emits
validated foreign keys and plain indexes for existing tables, so this is
common. The error text and `doc/DATABASE.md` give the fix.
- `ADD CONSTRAINT ... NOT VALID` classifies as `online_compatible`.
PostgreSQL still enforces it on new writes, so the author must make sure
that the previous release does not write rows that violate it.
- The classifier reads SQL only. It cannot see that new code writes
values that the previous release cannot read (for example a new enum
value). The `requires_staging` override exists for that case.
- `master` cannot fail the guard: the guard only judges migrations that
are not on the merge base.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
64afd22d82 |
fix(runner): keep the OpenCode MCP bridge on keep-alive so OpenCode keeps its Paperclip tools (#20921)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode agents call Paperclip tools through a local MCP
bridge in the runner's OpenCode proxy
> - #20886 made the bridge send `Connection: close` on every response,
as hardening for one stalled parallel tool call that was never
reproduced
> - After #20886 merged, many native OpenCode runs started without any
Paperclip MCP tools
> - Agents then said the Paperclip tools were missing, worked only with
bash and file tools, and never called paperclip_finish
> - This pull request removes the `Connection: close` header and keeps
the delivery diagnostics from #20886
> - The benefit is that OpenCode agents get their Paperclip tools again,
and a recurrence of the original stall still shows in the run log
## Linked Issues or Issue Description
Refs #20886.
**What happened**
In chat eval runs on CI, every OpenCode session before #20886 sent a
first model request of about 34k–36k input tokens (prompt plus the
Paperclip tool schemas). After #20886, about 9 of 25 sessions sent only
about 8k–15k tokens on the same prompts (for example 14,711 for
questions-and-approvals.necessary-question, 15,481 for
cases.cases-disabled, 14,096 for cases.update-case-facts). The prompt
envelope size did not change (about 32.7k characters). Those sessions
made no Paperclip MCP calls: zero tool-delivery records, no
paperclip_finish. The agents' own reasoning said "The paperclip tools
aren't in my toolset". Sessions on a commit without #20886 (
|
||
|
DottaandClaude Opus 5.5
|
6dc36700b1 |
fix(costs): mark unpriced usage in the agent, model and project cost reports (#20889)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cost reports show what agents and projects spent
> - A run can have usage with no known price; Paperclip saves it with
costStatus "unpriced" and costCents 0
> - The summary and the provider, biller and user reports count those
events, but the agent, agent-model and project reports did not
> - An agent that read the by-agent-model report saw a native run at
costCents 0 and told the person it cost $0.00
> - This pull request adds unpricedEventCount to those three reports and
tells agents what it means
> - The benefit is that missing charges are reported as unpriced, not as
free
## Linked Issues or Issue Description
**What happened**
In the chat eval budgets-costs.spend-breakdown, the person asked "How
much did Launch cost this week, broken down by agent?". The week
included a real native run whose usage had no known price (data-quality
reason missing_price). The agent read /costs/by-agent-model and
/costs/by-project. Both returned costCents 0 for that usage and no
unpriced count. The reply showed "1 native run ($0.00)" and gave the
total as complete.
**Expected behavior**
The reports say which rows include unpriced usage. The agent says that
usage is unpriced, marks the total incomplete, and links the record.
**Steps to reproduce**
1. Record one priced and one unpriced metered cost event for an agent in
a project.
2. Call GET /api/companies/{companyId}/costs/by-agent, /by-agent-model
and /by-project.
3. See costCents counts only the priced event, and no field shows the
unpriced event.
## What Changed
- `costService.byAgent`, `byAgentModel` and `byProject` return
`unpricedEventCount` (same filter as the summary: unpriced and not
subscription_included).
- Shared types `CostByAgent`, `CostByAgentModel` and `CostByProject` add
`unpricedEventCount`.
- The OpenAPI catalog entries for the cost reports that count unpriced
usage now say that costCents counts priced usage only, that unpriced
usage must be reported as unpriced (never $0.00), and that
/costs/data-quality lists those runs.
- Tests: embedded-Postgres service test (red on master, green here) and
an OpenAPI description test. A UI test fixture adds the new field.
## Verification
- `npx vitest run src/__tests__/costs-service.test.ts` (19 passed). The
new test fails with the master service.
- `npx vitest run src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2146 passed).
- `tsc --noEmit` for packages/shared and server.
## Risks
- Low risk. One added aggregate column per report and an added response
field. The UI does not read it yet.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
951d1b0b1b |
fix(opencode): ask for the empty-response correction with reasoning off (#20890)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner drives OpenCode turns for agents that use OpenCode models, such as DeepSeek through OpenRouter > - Sometimes the model ends a turn inside its reasoning. It reasons that it will reply and call `paperclip_finish`, and then it stops with no answer and no tool call > - The driver sends one correction prompt. The correction used the same reasoning variant, and in the failed runs the model stopped inside its reasoning again > - The run then failed with `opencode_empty_response`, and the person saw "The OpenCode run failed before it completed." > - This pull request sends the correction prompt with the no-reasoning variant, so the final answer comes in the answer channel > - The benefit is fewer failed chat replies, with no extra prompts and no change to the normal turn ## Linked Issues or Issue Description **What happened** A chat reply failed with `opencode_empty_response` in eval run gha-43069861174 (`agent-chat.conference-room`, OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`). The first completion and the one correction each held only a reasoning item (403 and 411 output tokens). Both reasoning items said the model would reply and call `paperclip_finish`. No answer text and no tool call followed. In retained eval evidence, 14 of 879 attempts needed this correction and 3 of those 14 failed. Each failed correction was another reasoning-only completion. **Expected behavior** After an empty completion, the correction gets a final answer or a completion report. **Steps to reproduce** 1. Start an OpenCode turn with reasoning enabled on an OpenRouter model. 2. Let the provider return a completion that holds only reasoning. 3. See that the correction prompt uses `paperclip-default` (reasoning on), so the model can end inside its reasoning again. ## What Changed - `OpenCodeServerDriver` sends the empty-response correction with the `paperclip-no-reasoning` variant. Normal turns keep their own reasoning mode. - `empty-response.test.ts` starts the `reply` and `empty-again` cases with reasoning on. It checks that the first prompt uses `paperclip-default` and the correction uses `paperclip-no-reasoning`. ## Verification - `pnpm exec vitest run src/drivers/opencode/empty-response.test.ts src/drivers/opencode/reasoning-mode.test.ts src/drivers/opencode/interrupt-pending-tool-call.test.ts` (in `packages/paperclip-runner`): 23 passed. Without the driver change, the `reply` and `empty-again` cases fail (red/green). - `node --test tests/runner-e2e/native-completion-source-contract.test.mjs`: 29 passed. - A paid eval run with this change and the related eval fixes is in progress. The PR comments will have its result. One run cannot prove the rate change, because the failure is rare. ## Risks - Low risk. Only the correction prompt changes. A model with no reasoning can give a shorter final answer, but the correction asks only for the final answer and the completion report from work already done. - Providers other than OpenRouter do not get a variant. Their behavior does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c02de922af |
fix(chat): deliver the person's time zone to chat agents (#20898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to agents in chat, and agents create routines and other scheduled work from that chat > - Paperclip does not record the person's time zone, so a chat agent cannot read "every weekday at nine" or "today" in the person's zone > - The agent then saves schedules in UTC, and the person gets the work at the wrong hour > - This pull request saves the browser time zone with the person's personal preferences and gives it to chat agents in the task prompt > - The benefit is that schedules and dates from chat match the person's local time ## Linked Issues or Issue Description **What happened** A person in America/Chicago wrote in chat: "Every weekday at nine in the morning, summarize Launch for me." The agent saved the routine schedule `0 9 * * 1-5` with the UTC time zone. It replied that it used UTC because the person's time zone is not recorded. **Expected behavior** The agent knows the person's time zone. It saves one enabled `0 9 * * 1-5` schedule with the time zone America/Chicago. Its reply names the zone. **Steps to reproduce** 1. Open the board in a browser with the time zone America/Chicago. 2. Open a chat with an agent. 3. Send "Every weekday at nine in the morning, summarize Launch for me." 4. Look at the routine trigger that the agent creates. Its time zone is UTC. ## What Changed - Database: add the nullable `time_zone` column to `user_workspace_preferences` (migration `0468`). This is an instance-wide personal preference by user, as AGENTS.md section 5 permits. - Shared: `WorkspacePreference` has `timeZone`. `updateWorkspacePreferenceSchema` accepts an optional IANA `timeZone` (max 64 characters, checked with `Intl.DateTimeFormat`). `coordinatorWorkspaceEnabled` is now optional. The request must contain at least one field. - Server: the preference service returns `timeZone` and updates only the fields in the request. The activity log entry stays the same. The OpenAPI entry uses the shared schema. - UI: a new `useBrowserTimeZoneSync` hook in `Layout` saves the browser time zone one time when it is different from the saved zone. It is silent and best-effort. It uses the selected company as the audit context. - Prompt: for a conversation (any adapter), the task context reads the chat owner's saved zone. Recovery keeps the zone that the first turn admitted. The chat block adds one line only when the zone is known: `The person's time zone is America/Chicago (from their browser). Read times and dates they give, such as "nine in the morning" or "today", in that zone, and save schedules such as routine triggers with that timezone; say the zone when you report a schedule.` ## Verification - `cd server && npx vitest run src/__tests__/workspace-preferences.test.ts src/__tests__/workspace-preferences-routes.test.ts src/services/heartbeat/task-markdown.test.ts src/__tests__/heartbeat-task-context.test.ts src/__tests__/openapi-routes.test.ts src/services/native-runtime/runner-api.test.ts` - `cd ui && npx vitest run src/hooks/useBrowserTimeZoneSync.test.tsx src/hooks/useCoordinatorWorkspaceEnabled.test.tsx src/components/Layout.test.tsx` - Red/green: without the task-context and task-markdown changes, the two new prompt tests fail. With the changes, they pass. - `pnpm check:token-gates` is clean. `tsc --noEmit` passes for `packages/shared` and `packages/db`. - Manual: open the board in a browser with a non-UTC zone. Then ask a chat agent to create a weekday routine at nine in the morning. The trigger uses the browser zone. ## Risks - Low risk. The migration adds one nullable column. - The PUT body is now more flexible: `coordinatorWorkspaceEnabled` is optional. An empty body still gets a 400 response. - The prompt grows by one line only for conversations with a known zone. - The saved zone changes when the person uses a browser in a different zone. The latest browser wins. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
683fd1e9a5 |
test(restart-chaos): cells for staging's no-restart failures (re-wake, Nth recovery step, long turn, K15 without volume) (#20893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) tests that agent work survives control-plane faults > - In one 8-hour staging window, 20 of 25 failed runs happened with no restart at all, so the harness did not see them > - The harness had blind spots: one short task per agent, no wake right after a successful run, no fault in a later recovery step, short turns, and a redeploy that always keeps its instance home > - This pull request adds cells for those shapes, so the harness reproduces these failure classes on master > - The benefit is that fixes for these classes can be proven red and green with the harness, without staging ## Linked Issues or Issue Description **What happened?** On staging, an agent ran 10–13 tasks at once on reused sandboxes. 30–60 s after a long run succeeded, an ordinary wake (a comment or an issue monitor) started the next run of the same task. That run or its recovery failed before any provider work, with errors such as `runner_remote_process_identity_unavailable`, `Controller journal drain timed out` and `journal_tool_input_unanswered`. Other failures happened in a later recovery step (a held agent-directory lock, a missing runtime asset after a deploy without a volume). **Expected behavior** The chaos harness can reproduce each of these shapes on demand, and grades them: no failed attempt, every task `done`, and no task left blocked without a working Retry. **Steps to reproduce** See the commands in the README sections "Re-wake after success", "Fail the Nth recovery step", "Long turn" and "Redeploy without a volume, with overlap". Related: #20844 (session fault SR1). W2C is the configuration change after a successful run; SR1 is the same change while a question is parked. This PR does not add or change `session-faults.ts`. ## What Changed - New workloads W2, W2Q, W2C (`rewake.ts`): phase 1 ends with the task left open; the harness, as the board, wakes the same task the seeded 0–60 s after its run settled. W2 uses a comment or a board-scheduled issue monitor (by nonce). W2Q parks on an unanswered `request_human_input` question and is woken by a comment before the answer. W2C changes the agent's `adapterConfig.idleTimeoutMs` before the wake. Oracle: phase 1 succeeded, the wake was served by a run that succeeded, `no_failed_attempt`, `done`, markers. - `--faults K0`: cells with no fault, for workloads whose shape is the fault. - `--fail-recovery-step <step>[:<nth>]` (`recovery-step-faults.ts`): when the wave's first restart fault starts its replacement, the harness arms one crash point in a new fail mode; the nth hit throws the error of the real step, once. Steps: `runnerd-start` (`spawn paperclip-runnerd EAGAIN`), `teardown` (provider destroy did not finish), `directory-lock` (`Agent directory operation is already active`). Crossed tasks get `recovery_step_failed_once` and `no_failed_attempt`. - Crash points (`packages/shared/src/crash-points.ts`): fail mode in the control file (`<token> <name>:<nth> fail [pid=<n>]`), with an optional call-site error; `pid=` binds the line to one server process. The harness arms the step for the replacement only and disarms it once its receipt lands. New point `agent_directory.operation_lock.before_acquire`. The teardown point moved into its `try`, so fail mode behaves like a failed provider destroy; kill mode is unchanged. All points stay disarmed under `NODE_ENV=production`. The runner package's own crash-point copy is not changed (runner image content ID unchanged). - `--long-turn-minutes <n>` (`long-turn.ts`, 15–60, opt-in): every task's first turn runs rounds of real tool work for n minutes before its workload. Checks `long_turn_rounds` and `long_turn_run_observed`. - K15 `--k15-fresh-home` (Daytona) and `--k15-overlap-seconds <min>-<max>` (`redeploy.ts`, `supervisor.ts`, `edge-proxy.ts`): each deploy generation gets a new, empty instance home and no volume variables; the next generation boots beside the old one behind an edge proxy on the wave's port, takes traffic once ready, and the old one gets SIGTERM after the overlap. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed shared crash-point files. - README: one section per new cell, with what it covers. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (48 files, 581 tests after rebasing on master, includes the new `shape-cells.test.ts`). - `pnpm exec vitest run packages/shared/src/crash-points.test.ts server/src/__tests__/crash-point-call-sites.test.ts` (fail mode, call sites). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`. - Chaos waves on master `1d67e309e0` server code (Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:99fc2671fc320816407f63cffd3acbd27471621febcd26494fe25bb5d7c5dd77`, content ID `1836c515…`), all RED unless noted: | Cell | Wave | Result on master | Staging class | | --- | --- | --- | --- | | W2/W2Q/W2C, 10 tasks on one Codex agent, `--daytona-warm`, K0 | `w-20261111T163206-s5201` | 9/10 fail. 3 tasks (W2 t02, W2C t03, t06): first fault `native_runner_process_exited`, same-run reattach, `native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`, terminal `runner_remote_process_identity_unavailable`, "Workspace repair required". 3 tasks (t04, t08, t10): `native_restart_recovery_blocked` (`native_retry_source_superseded`) while its successor ran (R4 `concurrent_runs`). 3 W2Q tasks: the answer's continuation was cancelled `issue_not_in_progress` (status `blocked`), and the task needed Retry. 2 W2C tasks then looped: 60+ succeeded runs in 20 minutes on an `in_progress` task. | D′/T terminal step reproduced with no restart, no governed wait and no PRP timeout (same error code and refusal fields). C+ retry outcome (ownership ambiguous) reproduced as `native_restart_recovery_blocked`. Not reproduced: `Controller journal drain timed out`, `did not durably suspend`, `journal_tool_input_unanswered`. | | Same, local `--lifecycle warm` | `w-20261111T160103-s5101` | 4/10 fail. 3/3 W2Q: continuation cancelled `issue_not_in_progress`, Retry needed. 1 W2C: the same `in_progress` run loop (39+ runs). | The remote-runner classes need a remote runner; locally they do not show. | | `--fail-recovery-step runnerd-start:1`, K1 at 10 s, local A1 | `w-20261111T162432-s5303` | Fail. After K1 the first runnerd start of recovery failed (`spawn paperclip-runnerd EAGAIN`); the run ended `native_provider_terminal_failed` ("previous turn is no longer active", "Automatic recovery stopped") and needed Retry. Control, same seed and K1 without the step fault (`w-20261111T171114-s5303`): pass. | A later recovery step that fails once ends the run instead of trying again (the A-chain shape). | | `runnerd-start:2`, local | `w-20261111T170909-s5304` | Step never reached: recovery started the runner once and succeeded. Grade fails `recovery_step_failed_once` (tested nothing). | — | | `teardown:1`, K1, Daytona A1 | `w-20261111T172048-s5312` | Pass. The teardown failed once and the sweep settled it. | Not reproduced. | | `directory-lock:1`, K2, Daytona A1 | `w-20261111T171247-s5311` | Step never reached by A1 after K2. | Not reproduced; A1 recovery does not collect remote agent files. | | K15 `--k15-fresh-home --k15-overlap-seconds 30-150`, `--storage s3`, Codex and OpenCode, T1/C1 | `w-20261111T173618-s5322` | 1/4 fail (OpenCode T1): `native_continuation_requires_reconciliation` with "Agent directory belongs to a different execution environment"; the Retry runs failed `adapter_failed` "Configured instruction entry is missing"; the task rests `blocked` on a board recovery action, and two sandboxes were left (R6). One Codex T1 task finished before its fault (`fault_crossed_tool`). | A-deploy: a controller-local file missing after a deploy without a volume. Here the missing file is the agent's instruction bundle, not a runtime-context asset; whether staging keeps instruction bundles outside the instance home is not verified. | | `--long-turn-minutes 15`, K1 at 600 s, local A1 | `w-20261111T175505-s5401` | Task pass (10/10 rounds, 992 s span, run 1019 s, K1 crossed mid-turn). The wave exited 2 only because I switched branches in the worktree during the wave. | Shape check only. | ## Risks - Product code changes are test instrumentation only: three `crashPoint` call sites take a fail-mode error, and one new call site was added. Disarmed (always in production) each call is one comparison. Moving the teardown point into its `try` does not change kill mode: the process dies at the same instant. - The new cells are opt-in. Default waves plan the same tasks as before for the same seed. - The W2 cells depend on model behaviour (phase 1 leaves the task open). The oracle records the task status at the wake. - `--k15-fresh-home` drops everything under the instance home. Whether staging keeps agent instruction bundles outside the instance home is not verified (see the K15 result). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test and chaos runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
854b4bcac4 |
fix(opencode): report a provider 401 from the hello probe as missing authentication (#20878)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent has a runtime readiness test ("test environment") that
runs a short hello probe through its harness
> - The OpenCode probe sorts failures into model, timeout,
authentication and generic errors
> - OpenRouter rejects an empty or unknown key with HTTP 401 and the
messages "Missing Authentication header" or "User not found."
> - The probe matched authentication only by wording, so these 401
replies became a generic `opencode_hello_probe_failed` error
> - The server maps `*_hello_probe_auth_required` to the missing-account
state, so the user saw a generic failure instead of "connect an account"
> - This pull request classifies the structured APIError status 401 as
`opencode_hello_probe_auth_required`
> - The benefit is that a member without a working account shows the
missing-account state and its fix
## Linked Issues or Issue Description
**What happened**
An OpenCode (native runner) agent with an empty OpenRouter key failed
its readiness test with the generic error `opencode_hello_probe_failed`.
The chat eval `teams.configure-team` could not form its "one member is
missing an account, one is ready" fixture because of this.
**Expected behavior**
The readiness test reports `opencode_hello_probe_auth_required` (warn)
when the provider rejects authentication with HTTP 401, whatever the
provider's wording.
**Steps to reproduce**
1. Configure a native runner OpenCode agent with model
`openrouter/deepseek/deepseek-v4.1-flash-1017` and `OPENROUTER_API_KEY`
set to an empty string.
2. Run the agent's environment test.
3. OpenCode 1.18.34 prints
`{"type":"error","error":{"name":"APIError","data":{"message":"Missing
Authentication header","statusCode":401}}}`.
4. The result is `opencode_hello_probe_failed` (error), not the
missing-authentication check.
## What Changed
- `parse.ts`: add `openCodeErrorStatusCodes`, which reads the HTTP
status from OpenCode error events.
- `test.ts`: treat a 401 status on the hello probe as
`opencode_hello_probe_auth_required`, in addition to the existing
wording match.
- `test.remote.test.ts`: regression tests with the two recorded
OpenRouter 401 outputs, and a negative control (500 and 429 stay generic
failures).
## Verification
- `pnpm exec vitest run
packages/adapters/opencode-local/src/server/test.remote.test.ts
packages/adapters/opencode-local/src/server/parse.test.ts` passes (28
tests).
- Red check: without the classifier change, the two new 401 tests fail.
- `npx tsc --noEmit -p packages/adapters/opencode-local` passes.
- I recorded the probe output from the pinned OpenCode 1.18.34 binary
against OpenRouter with an empty key and with an unknown key.
## Risks
- Low risk. Only a 401 status changes classification. A 401 means the
provider rejected the credential. Other statuses keep their current
result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandPaperclip
|
5dfbb04b85 |
Poll asynchronous workspace provider preparation (#20865)
## Thinking Path > - Paperclip starts agent runs through a workspace provider when a project uses remote storage. > - The provider must prepare the workspace before the agent starts. > - One long worker call ties the checkout to its transport. > - Some checkouts take longer than the transport allows. > - This change lets an opted-in provider accept work and report its deadline. > - Core waits for completion through short status calls and stops at the provider deadline. ## Linked Issues or Issue Description **What happened?** A workspace provider can lose its reply when preparation takes longer than a single transport request. This stops run bootstrap before agent execution. **Expected behavior** Core should wait for a provider-confirmed workspace through short requests. The provider's deadline must bound the wait. **Steps to reproduce** Use a workspace provider that takes longer to prepare a checkout than its transport can remain open. Start a run that requires a new checkout. ## What Changed - Add an optional workspace preparation method to the plugin SDK. - Validate pending receipts and keep the original binding checks on completed replies. - Poll only during active preparation. Retry a lost status reply within the advertised deadline. - Preserve the original operation for providers that do not offer asynchronous preparation. - Reject deadline offers that exceed the existing lifecycle lock budget. Honor accepted offers without shortening them. - Checkout verification uses the existing synchronous RPC with a 10-second deadline and a 12-second transport timeout. [Core verification PR #20876](https://github.com/paperclipai/paperclip/pull/20876) ## Verification - All 63 workspace SDK and Core integration tests passed with temporary local PostgreSQL. The tests cover lost status replies, deadline expiry, oversized offers, and synchronous verification with an async-capable worker. - All 58 runner SDK input, evidence, and public type tests passed with the regenerated source pins. The local server typecheck passed. ## Risks - This change activates only when the provider advertises the method. Existing providers keep their current behavior. - Polling uses one-second intervals during active preparation. It stops at completion, failure, or the provider deadline. - No database migration is required. ## Model Used - OpenAI GPT-6 Codex with local code execution and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e601f9d2fc |
fix(runner): answer each OpenCode MCP tool call on its own connection (#20886)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip runner runs native OpenCode agents. OpenCode calls Paperclip tools through a local MCP bridge in the OpenCode proxy. OpenCode runs on Bun and uses Bun `fetch` for each MCP request. > - In one chat eval, the model made two parallel tool calls. The bridge produced and wrote both results in about 0.55 s. OpenCode received only one of them. > - The other call waited until Bun's default fetch timeout fired, about five minutes later. The chat turn lost five minutes, and the eval missed its deadline. > - The most likely trigger is the reuse of an idle keep-alive socket. We could not reproduce the loss locally, so this pull request is hardening that is based on a hypothesis. > - This pull request makes the bridge answer every request on its own connection. It also records how each tool result left the bridge, so a recurrence can be attributed from the run log. > - The benefit is that a pooled socket can no longer strand a tool result. If a call is stranded again, the run log shows whether the bridge wrote the answer. ## Linked Issues or Issue Description No public issue exists. Related: Refs #18751 (draft native client acceptance diagnostic), Refs #19546 (abandoned queued calls), Refs #20862 (transport event delivery; it does not touch this path). **What happened** In CI run `gha-42925998380` (chat-features, runner-opencode, `cases.create-case`), the CEO chat run used OpenCode 1.18.34 with an OpenRouter DeepSeek model. At 00:01:12.16 the model issued `list_workflows` (MCP id 3) and `list_projects` (MCP id 4) in parallel. The runner emitted the dynamic tool results for id 4 at 00:01:12.468 and for id 3 at 00:01:12.714 (ok, HTTP 200). The bridge writes the HTTP response directly after that event. There is no queue between them. OpenCode completed `list_projects` at 00:01:13.113. It failed `list_workflows` only at 00:06:01.761 with `The operation timed out.` We read the bundled OpenCode source. Its MCP tool call has no timeout that produces that message: the MCP request timeout gives `-32001 Request timed out`, and the configured value is 1 h for durable calls. With the same binary in Bun mode (Bun 1.3.14), a fetch on a reused keep-alive socket that never gets an answer fails after 298 s with `TimeoutError: The operation timed out.` So Bun's default fetch timeout ended the call, and the client never consumed the written answer. The previous call (`list_agents`) answered at about 00:01:06.89, so its socket was idle for about 5.27 s when the parallel pair started. Our thesis is that call 3 reused that pooled socket, and call 4 opened a new one. **Expected behavior** When the bridge has the result of a tool call, the MCP client gets it. The client must not wait for a five-minute transport timeout. **Steps to reproduce** We did not reproduce it locally. This is the only occurrence in the retained evidence. A macOS stress test (Bun 1.3.14 client, Node 26 server with the bridge's default keep-alive, one call, then an idle time of 4.8–7.2 s, then two parallel calls, 8 pairs in parallel) completed 343 trials and about 1,029 requests with 0 lost results. The failing run was on Linux CI with Node 24. ## What Changed - `mcp-bridge.ts`: every bridge response sends `Connection: close`. The MCP client cannot reuse a pooled keep-alive socket for a later request. A loopback connect per request costs much less than the failure mode. - `mcp-bridge.ts`: each `tools/call` records one closed delivery diagnostic. This includes rejected calls (unsupported tool, invalid input, duplicate identity), which record the outcome `error` (`paperclip.opencode.tool-delivery-diagnostic/v1`) when its response closes. The fields are the call id SHA-256, the socket request index, the socket idle ms, the handler ms, the outcome (`result`, `error` or `unanswered`) and `written` (`writableFinished`). It holds no tool names, inputs or results. The limit is 256 records per bridge. - `opencode-proxy-finish-diagnostics.ts`: the proxy forwards these records to the run log as `warning` frames after session binding. It validates the exact key set and value types and drops anything else. - Tests: a new `mcp-bridge-delivery.test.ts` uses a pooling keep-alive client with the incident's call shape (one call, then two parallel calls). New forwarder cases cover the record validation. The existing finish diagnostic assertion now filters by its own schema. ## Verification - Red: with the `Connection: close` line removed, `mcp-bridge-delivery.test.ts` fails with `expected 'keep-alive' to be 'close'`. The pooling client then reuses the socket. - Green: `cd packages/paperclip-runner && pnpm exec vitest run src/drivers/opencode/mcp-bridge-delivery.test.ts src/drivers/opencode/mcp-bridge-diagnostics.test.ts src/drivers/opencode/mcp-bridge.test.ts src/cli/opencode-proxy-finish-diagnostics.test.ts src/cli/opencode-app-server-proxy.test.ts src/cli/opencode-proxy-abandoned-tool-call.test.ts` gives 6 files and 45 tests passed. - The new test asserts that all three results arrive with the correct JSON-RPC ids, that no socket is reused, and that each call records `socketRequestIndex: 1`, `outcome: "result"`, `written: true` without tool names or raw ids. A second test asserts that a client that leaves before its answer records `outcome: "unanswered"`, `written: false`. A third test asserts that a rejected call records `outcome: "error"`, `written: true`. - `tsc -p tsconfig.json --noEmit` for `packages/paperclip-runner` reports no errors in the changed files. Seven existing errors in `codex-runtime-adapter.ts` and `eval/workflow-harness.ts` come from this worktree's linked dependencies. Required CI typecheck is the source of truth. - This does not prove the production fix. A recurrence on CI will show socket reuse, idle time and `written` for the stranded call in the run log. ## Risks - Low risk. Each MCP request now opens a new loopback TCP connection. That adds well under 1 ms per call, and there is no TLS. - The `notifications/cancelled` handling and the abandoned-call abort do not change. The abort still reads `writableFinished` on response close. - One extra run-log warning for each tool call (at most 256 for each bridge). The record holds only a hash, counters and timings. - Previous-release story (upgrade-safe run state): no persisted run, session, launch or pack state changes. A bridge from the previous release keeps its behavior until the proxy restarts. - If the thesis is wrong, the five-minute stall can recur. The new records will then show `written: true` on a fresh socket and point the next investigation at OpenCode or Bun. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
f9171ef621 |
fix(native): retry a throttled post-turn checkpoint for real; wait out storage throttling while saving the provider session (#20871)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native turn in a remote sandbox, closing the session saves
the provider session (the harness backup) to object storage. The next
attempt resumes from it.
> - On staging, object storage throttled that save ("Please reduce your
request rate", S3 `SlowDown`). The close failed, was retried 3 times 1 s
apart, and the session was quarantined. The next attempt of the same run
was refused with `native_session_cleanup_quarantined`.
> - The retries could never succeed: the Runner transport memoizes
`close()`, so every retry returned the first rejected promise and never
ran the checkpoint again. Quarantine recovery and admission recovery
call the same `close()`.
> - Also, the backup upload did not wait out a throttle or outage longer
than one request's own retries.
> - This pull request makes a close that failed only in its checkpoint
run that checkpoint again on the next `close()`, and makes the backup
upload wait out storage unavailability inside the existing bounded
storage wait.
> - The benefit is that a transient storage throttle no longer ends in a
quarantine that refuses the next attempt of the same run.
## Linked Issues or Issue Description
Refs #20772 (made the backup upload resendable after `SlowDown`). Refs
#20165 (per-request throttle backoff). Refs #20833 (earlier steps of the
same failure chain).
**What happened?**
1. A staging Codex run on Daytona finished its turn. The post-turn close
ran the remote checkpoint.
2. The checkpoint failed: `remote checkpoint failed: Please reduce your
request rate.`
3. The Runner retried the close 3 times 1 s apart (`native session
cleanup recovery after close failure (1..3)`), then 3 quarantine
recoveries. All failed within about 5 s.
4. The next attempt of the same run was refused:
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.
**Expected behavior**
A throttle slows the save down. A retry of the close saves the session
again. The next attempt of the same run proceeds once the save succeeds.
**Steps to reproduce**
1. Make a transport close whose checkpoint throws once.
2. Call `close()` again.
3. On master the second call returns the first failure without running
the checkpoint again.
4. Separately, answer each harness-backup `PUT` with 9 `503 SlowDown`
responses. On master `persistNativeHarnessBackup` throws `SlowDown:
Please reduce your request rate.`
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandPaperclip
|
b2edea5754 |
fix(workspaces): verify existing worktrees before fresh retries (#20876)
## Thinking Path > - Paperclip manages work for AI agents. > - Retry checks whether the task can keep its existing workspace. > - A provider can complete checkout before Core receives the result. > - Recreating a completed worktree costs time and can change its base commit. > - Core uses the same ownership and writer checks for local and provider workspaces. > - The provider verifies its checkout before Core creates a fresh workspace. ## Linked Issues or Issue Description **What happened?** A retry can allocate a new provider worktree even though the original checkout completed. **Steps to reproduce** Complete workspace checkout through a storage provider. Fail Runner startup or lose the prepare reply. Retry the task. **Expected behavior** Reuse the same worktree when the provider confirms checkout and the prior run can no longer write to it. Related: #20865 addresses timeouts while the original prepare is in progress. This PR checks an earlier checkout when the user retries. ## What Changed - Check the prior task binding and writer state before asking a storage provider to verify a checkout. - Extend the existing prepare operation with an optional verification mode. Providers opt in through their manifest. - Return through normal workspace reuse after verification. Keep the fresh retry fallback when verification cannot establish checkout. ## Verification - Core workspace preparation, plugin workspace ownership, retry continuity, and SDK workspace-provider suites passed locally (213 tests). - Focused cases pass for provider verification, retry through heartbeat dispatch, and real local Git checkout verification with no plugin worker. The heartbeat test confirms that ordinary preparation and adapter dispatch keep the original workspace, branch, and base commit. - Core server typecheck passed before the final audit and timing edits. GitHub CI covers the final revision and broader regressions. ## Risks Provider verification adds one bounded call before a fresh retry. Older providers keep their current behavior. Verification must preserve the workspace, branch, and recorded base commit. The normal execution preparation checks still run before admission. No database migration is needed. ## Model Used OpenAI GPT-6 (Codex), with source inspection and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a52f622be8 |
fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error
## Linked Issues or Issue Description
Refs #20808
Refs #20832
Refs #20820
**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.
**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Fable 5.2
|
76da2fec4f |
perf(runner): deliver transport events when they arrive instead of polling every 5 ms (#20862)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server runs each native agent session through a runner transport. The transport reads the events that the runner commits to the session's control plane and delivers them to the driver. > - The transport found those events with a timer. Every live transport ran a pass every 5 ms, for as long as the session was open, whether or not an event had arrived. > - A CPU profile of the staging server shows the cost. The main thread was at 96% of one core, and this pass used 5.6% of it with about a dozen live sessions. The cost grows with the number of sessions and never stops. > - The same timer also delayed every event. An event waited up to 5 ms for the next pass before the driver saw it. Agent boot time and time to first token are measured requirements. > - This pull request makes the control plane tell the transport when a reader can see a change, and the transport delivers in the next task. The fixed timer is gone. > - The benefit is that an open session with nothing to deliver costs nothing on the main thread, and an event reaches the driver sooner than before. ## Linked Issues or Issue Description No public issue exists. The problem is described here. **What happened?** The server's main thread on a hosted instance stayed at 96% of one core, so every request and every agent turn waited behind it. A 30 second CPU profile of the live server (Node 24, about a dozen live native agent sessions) attributes 5.6% of the main thread to `#pumpEventsSafely` and 4.7% self time to `#pumpEvents` in `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`. Three places started `setInterval(() => this.#pumpEventsSafely(), 5)`. Each pass scans the session's committed-event window, so the cost is `sessions x 200 passes per second x window size`, with nothing to deliver. **Expected behavior** An open session that has no event to deliver does no periodic work. An event that arrives is delivered at once, not at the next timer tick. **Steps to reproduce** 1. `cd packages/paperclip-runner` 2. `PAPERCLIP_TRANSPORT_PUMP_BENCHMARK=1 pnpm exec vitest run src/live/runnerd-transport-pump.benchmark.test.ts` 3. Read the `TRANSPORT_PUMP_BENCHMARK` lines. `stateReadsPerSessionPerSecond` counts how often an open, idle transport reads the control plane. `deliveryWaitMs` is the time from the moment a reader can see a committed event to the moment the transport delivers it. **Paperclip version or commit** Present on master at `c8ae02e5ce`. The profile is from `556b249897`. **Deployment mode** Hosted server with native agent sessions. The same code runs in local development. ## What Changed - `DurablePrpControlPlane` has a new `onReadableStateChange(listener)`. It fires when the state that a reader of `store` can see may have changed. The control plane calls it after every stored-state replacement or save, after a saved checkpoint releases events that were waiting for it, and after an authenticated runner moves the recovered cursor. - The transport subscribes when it creates its control plane. A change wakes one delivery pass in the next task (`setImmediate`). The pass never runs inside the control plane's own commit, and several changes in one task share one pass. This keeps the existing order: the control plane finishes its commit before a reader reacts. - The three `setInterval(..., 5)` timers are gone. `#startPump` and `#stopPump` replace them and keep the same lifecycle: the pump runs from session start, and it stops on release, controller retirement, failure, and close. - The delivery pass now reports what it left behind. Events that the transport holds back itself (a pending turn start, a pending recovery binding, a trace write that must be retried) are retried every 5 ms, as before, and only while they are held. This is the only timer that remains. - A pass that stops after it exposes `turn/started` (so the driver can bind the turn first) now continues in the next task. Before, the events after it waited for the next 5 ms tick. - `attachRun` wakes the pump when it moves the transport to the new run identity. - New regression test `src/live/runnerd-codex-transport-pump.test.ts`, new opt-in benchmark `src/live/runnerd-transport-pump.benchmark.test.ts`, a paced provider fixture, and a test-only probe that records when each committed event becomes readable. How this meets `rules/sleep-aware-background-work.mdc`: | Item | Answer | | --- | --- | | Trigger | The control plane's readable-state change. No timer. | | Source of truth | The control plane's committed-event window and the transport's delivery cursor. A notification is a hint to look at them. | | Pending-work predicate | The pass returns `held` or `next_task`, or a trace write is waiting. | | Next deadline or retry | 5 ms, only while the predicate is true. `next_task` uses one `setImmediate`. | | Quiet condition | Nothing is held. Then the transport has no timer and no scheduled task. Release, retirement, failure, and close stop it completely. | ## Verification **Reproduction and before/after measurement** Run from `packages/paperclip-runner`: ```sh PAPERCLIP_TRANSPORT_PUMP_BENCHMARK=1 pnpm exec vitest run src/live/runnerd-transport-pump.benchmark.test.ts ``` The benchmark opens 12 real transports against `runnerd` and a credential-free provider fixture. It measures the open sessions while nothing arrives. Then all 12 sessions stream one turn at the same time (40 events each, one every 15 ms). "Before" is the unchanged source at `e2cf9a670f`, which was the base of this branch when the measurements were taken. "After" is this change on that base. Each cell shows two runs, and the runs alternate between before and after. The host was a shared macOS arm64 machine with other work running (load average 8 to 10), so absolute numbers carry that noise. External checkpoint persistence. This is the mode that the server uses. | Measure | Before | After | | --- | --- | --- | | Idle: control-plane state reads per session per second | 364 / 365 | 0 / 0 | | Idle: event-loop utilization, 12 new sessions | 1.5% / 1.5% | 0.0% / 0.0% | | Idle: event-loop utilization, 49 retained events per session | 2.1% / 2.2% | 0.0% / 0.0% | | Idle: process CPU, 49 retained events per session | 12.8 / 11.8 ms per second | 0.05 / 0.04 ms per second | | Delivery wait, mean (480 events) | 3.48 / 4.22 ms | 1.91 / 1.81 ms | | Delivery wait, p50 | 3.16 / 3.97 ms | 0.83 / 0.58 ms | | Delivery wait, p95 | 7.78 / 8.49 ms | 6.36 / 7.47 ms | | Delivery wait, max | 8.84 / 9.23 ms | 9.07 / 8.86 ms | | Delivery wait of the turn's terminal event, p50 | 2.37 / 2.58 ms | 0.08 / 0.05 ms | The idle cost grows with the retained event window, as the profile predicts. With `PAPERCLIP_TRANSPORT_PUMP_DELTAS=400` (409 retained events per session, one run each): idle event-loop utilization 3.4% before and 0.0% after, idle process CPU 21.5 ms per second before and 0.08 after. Delivery wait in that run: mean 5.93 ms before and 3.55 ms after, p95 16.27 ms before and 8.03 ms after, max 66.94 ms before and 14.04 ms after. Delivery wait is the time from the moment a reader can see a committed event to the moment the transport hands it to the driver. It is equal or better at every point. The p95 and the max did not move much. They belong to the first events of each turn, which wait for the turn start response. That wait is a separate 10 ms poll (`waitForRunnerCommand`), and this pull request does not change it. File persistence (the local journal, which writes and syncs on the main thread): | Measure | Before | After | | --- | --- | --- | | Idle: state reads per session per second | 363 / 364 | 0 / 0 | | Delivery wait, 12 sessions, mean | 10.33 / 13.76 ms | 11.88 / 12.73 ms | | Delivery wait, 12 sessions, p50 | 6.04 / 6.87 ms | 6.34 / 6.20 ms | | Delivery wait, 4 sessions and 400 events each, mean | 4.75 ms | 2.39 ms | | Delivery wait, 4 sessions and 400 events each, p50 / p95 | 0.07 / 22.99 ms | 0.06 / 11.80 ms | With 12 sessions the file journal saturates the main thread with its own synchronous writes, and the wait is the time behind those writes. Three more before-runs of that case gave means of 9.0, 14.0 and 20.2 ms, so the two columns are the same within the noise. With 4 sessions the journal does not saturate and the after column is better. **Regression test (red, then green)** ```sh pnpm exec vitest run src/live/runnerd-codex-transport-pump.test.ts ``` - `stops looking for events while an open session has none to deliver` (file and external persistence). On the unchanged source: `expected 708 to be less than 100` and `expected 732 to be less than 100` state reads in a 2 second idle window. With the fix: pass. - `delivers each committed event in the next task, without a timer tick` (file and external persistence). The test freezes every interval timer, runs a full turn, and checks that each event is delivered before a task that the test queues at the moment the event becomes readable. This is an order check and does not depend on the speed of the host. On the unchanged source: `the turn's events were delivered without a periodic poll: expected false to be true`. With the fix: pass. - Result: 4 of 4 failed on the unchanged source, 4 of 4 passed three times with the fix, and 4 of 4 passed again after the rebase. **Other local checks** - `node ./node_modules/typescript/bin/tsc -p tsconfig.json --noEmit` and `-p tsconfig.test.json --noEmit` in `packages/paperclip-runner`: pass on the rebased commit. - `pnpm exec vitest run src/live/runnerd-codex-transport` (15 files, 455 tests). These suites cover attachment, recovery, warm transition, suspension, and release and resume on real `runnerd` processes. Before the rebase, on a quieter host: all passed. On the rebased commit, at a host load average of 15 to 20: 454 passed and 1 failed. The failure is the terminal-replay row that is described below. - `pnpm exec vitest run src/live src/control-plane src/native-session-runtime.test.ts src/backends src/drivers/dot/runnerd-dot-driver.test.ts src/drivers/muse/runnerd-muse-driver.test.ts src/drivers/codex/codex-protocol-integrity.test.ts --maxWorkers=3` (81 files): 1566 passed, 32 failed, 6 skipped. The failures: - 30 failures in `runnerd-acpx-permission-ordering.test.ts` and `runnerd-acpx-permission-forwarding.test.ts`. Both replace the control plane with a fixture that appended events in place and relied on the 5 ms poll to notice them. The fixture now implements `onReadableStateChange` and notifies its readers when it emits. Both files pass (30 of 30). - 1 failure in `runnerd-codex-transport.test.ts`, `stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`: `runner did not durably suspend before checkpoint`. It passed 3 of 3 when run again alone. - 1 failure in `runnerd-codex-transport-settlement.test.ts`, the terminal-replay row of `settles only retained control authority without starting another provider turn`: `native_cleanup_maintenance_unproven`, with events still in the runner outbox. It failed again in 2 runs of that test group and in the rebased run above, each time while the host load average was 11 to 20. At a load average near 8 it passed 3 of 3 alone and 1 of 1 as a group with the fix. The unchanged master source (`c8ae02e5ce`) gives the same result: it passed 2 of 2 alone and 1 of 1 as a group at a load average near 8, and the same row failed with the same error (`native_cleanup_maintenance_unproven`, 27 events still in the runner outbox) in a group run at a load average of 11 to 15. So this row is sensitive to host load with or without this change. The failing step is the retained-session maintenance path. That path builds its own control plane, has no transport pump, has no listener on the new notification, and has a fixed 30 second deadline. - `pnpm verify` returns exit 2 (broader verification required, because of the new fixture file). CI supplies the broad coverage. - Not run locally: the server test suites, the repository-wide typecheck, and the build. CI ran them on `4bce83f50a`: 59 checks passed and 4 were skipped (Storybook deploy, Storybook visual regression, Sandbox provider tests, Select Vitest files). ## Risks - **A change that does not notify.** With no periodic pass, an event is delivered only when something wakes the pump. Every wake comes from one of a small, closed set: the five places where the store replaces or saves its state, the checkpoint publication, the recovered-cursor update, and the transport's own identity and start points. A new code path that changes readable state without one of these would leave an event undelivered until the next change. The regression test runs a full turn with every interval timer frozen, in both persistence modes, so a missing wake on the normal path fails the test. The existing transport suites (attachment, recovery, warm transition, suspension, release and resume) now run with no timer that could hide a missing wake. - **Delivery in the next task, not later.** An event now reaches the driver up to 5 ms earlier. The pass still runs in its own task after the commit, and the same gates hold a terminal event until the turn start response is released. The order of events does not change. - **No protocol, schema, Rust runner, or provider pack change.** The contract fingerprint inputs are untouched. Saved state from an older release resumes the same way. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code. Tool use and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
cda23f08ac |
fix(db): preserve legacy project reads during rolling upgrades (#20860)
## Thinking Path > - Paperclip controls agent work while web servers change versions. > - The old and new servers can share a database during that change. > - Older servers still read and write the project environment column. > - Migration 0466 removes that column before those servers stop serving. > - A later repair alone leaves a committed missing-column window and loses values. > - This pull request applies the exact drop and repair in one transaction. > - Old reads and saved values survive while current placement behavior stays unchanged. ## Linked Issues or Issue Description Refs #20835 and #20852. **What happened?** Migration 0466 drops `projects.environment_id`. The previous server still selects that column and reads it during admission. Applying a separate repair afterward leaves a window in which those requests fail. The drop also removes saved values. **Expected behavior** A database that has not yet applied 0466 must keep the column and its values visible to concurrent old servers throughout the upgrade. Both published migration entries must be recorded truthfully. Current project responses must keep the new workspace placement contract. **Steps to reproduce** 1. Use a database before migration 0466 with nonempty project environment values. 2. Keep an old project writer transaction open and start the pending migrator. 3. Start an old project read, then commit the writer. 4. Without this repair, the saved values disappear. With this repair, the reader sees the preserved values and both migrations commit together. ## What Changed - Add migration 0467 to restore the legacy nullable UUID column and its original foreign key. Leave the published 0466 SQL unchanged. - Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock the project table, retain nonempty values in a temporary table, execute both real SQL files, restore values, and record both real journal entries before commit. - Apply the additive 0467 repair normally when 0466 was already committed. - Keep a deprecated database-only field and omit it from current project API responses. Current placement logic stays unchanged. - Add PostgreSQL regressions for concurrent old reads and writes, idempotence, rollback, and an already-applied drop. Document the overlap contract and retain the five newest generated snapshots. ## Verification - Node 24 cached agent preflight passed. - `pnpm exec vitest run packages/db/src/project-environment-overlap-migration.test.ts server/src/__tests__/project-environments.test.ts -t 'project environment rolling migration|saves storage, not'`: 4 passed in 32.57 seconds against real disposable PostgreSQL. - Before the owner fix, the regression had two failures: saved values disappeared and the restore-failure case committed. The fixed case proves that both DDL and journal entries roll back together. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/db check:migrations`: passed. - `git diff --check`: passed. Migration 0466 bytes are unchanged. The generated snapshot set is 0463 through 0467. - Review found a raw DELETE response that still exposed the database-only field. The delete projection now omits it; a real authorized HTTP DELETE passed with successful cleanup. The concurrency test now observes the old reader waiting inside PostgreSQL before releasing the old writer. All three migration cases passed with that barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as failed attempts; they were not product-regression passes. - Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh Greptile 5/5 with no open findings. [Required final-head CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612) passed, including build, typecheck, migration checks, all test shards and the aggregate verification gate. The separate Superagent security scan ended inconclusive/neutral; it did not produce a passed scan. Deployed acceptance remains pending. No hosted database or app deployment was changed by this branch. ## Risks The pair takes a table lock for its transaction. Operators must retain their normal lock and statement timeouts. Both SQL files and their exact hashes must match before the pair runs. A failed restore rolls back the drop and both journal entries. If 0466 was already committed, 0467 restores the column and foreign key. It cannot recover values that were already deleted by the earlier release. This repair does not claim to recover those values. ## Model Used OpenAI Codex (GPT-6), with tool-assisted implementation and real PostgreSQL qualification. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
a219dd7499 |
fix(runner): keep orphan-leaving tests apart from the verifying lineage reap (#20849)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner stops a dead runnerd's provider and tool processes from its process lineage journal (`process_lineage::journal::reap_journals`) > - The reap refuses its proof (`unattributed_orphan`) while any live orphan (parent PID 1) of this user started after the dead runnerd's last sample, whoever started it. This rule is deliberate > - `cargo test` runs the runner-core lib tests in parallel in one process, and several of those tests leave orphans of this user on purpose > - When one of those orphans starts during the reap in `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`, the reap correctly refuses, and `ci / Verify Paperclip Runner (rust)` fails > - This pull request keeps the orphan-leaving tests apart from that reap with a test-only gate, and fixes a related zombie race in the supervisor integration tests > - The benefit is that the Runner Rust job stops failing on unrelated pull requests ## Linked Issues or Issue Description Refs #20834. Refs #20770. **What happened?** `ci / Verify Paperclip Runner (rust)` failed intermittently in `process_lineage::journal::tests::reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` (journal.rs:649), for example in run 43044317310 on #20834. The panic was `ReapReport { verified: false, reason: Some("unattributed_orphan"), journals: 1, signalled: 4, remaining: [4178] }`. The test's own thread was TID 4142, so the leftover process started while the test ran. The same test failed once on #20770 and passed on re-run. **Expected behavior** The reap proves the test's provider and escaped tool job gone, and no other test can make it refuse. **Steps to reproduce** 1. Build the runner-core lib tests for Linux (`cargo test --release -p paperclip-runner-core --lib --no-run`). 2. Run the binary 40 times in `docker run --init --cpus 2 -e RUST_TEST_THREADS=8`. 3. See `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` fail with `unattributed_orphan` (3 of 40 runs here). ## What Changed - Root cause: the reap's orphan rule covers every process of this user on the host. The reaping test and the orphan-leaving tests run in the same test process at the same time. The test's own processes are always attributed through the recorded group and session, so the leftover was another test's orphan. - `process_lineage::host_orphans` (test-only): a gate. A test that can leave an orphan of this user holds `may_orphan()` until the orphan is gone or a zombie. A test whose reap must verify holds `no_foreign_orphans()`. Holders of `may_orphan()` never wait for each other, so a test can hold several without a deadlock. `wait_until_group_gone()` waits until a killed group has exited, because a signal takes effect after `kill` returns. - `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` holds `no_foreign_orphans()`. - These lib tests now hold `may_orphan()`: - `reaping_refuses_an_orphan_born_after_the_last_sample` (deliberate orphan) - the three `process_supervisor` `*_stops_a_tool_job_that_left_its_session` tests (escaped tool job) - `orphaned_startup::restart_keeps_the_fence_while_a_child_survives_its_exited_leader` (`sleep 30` outlives its leader) - `exited_child_reader_deadline_bounds_continuous_descendant_stdout` (flooding subshell outlives the Codex provider) - `a_timed_out_turn_start_never_establishes_a_resumable_rejection` (`sleep 10` can be orphaned for a moment during shutdown) - the five `provider_backend` tests that use `disconnected_receipt_limit_fixture` (`while :; do sleep 1; done`, killed through `fence_local_receipt_limit`). The fixture returns its guard first, so the guard is released after the executor stops the provider - the two `late_park_observation_*` tests (`cat >/dev/null` runs until shutdown) - `a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn` (shutdown during `sleep 2`) - New test `a_verifying_reap_waits_for_held_orphans_and_holds_new_ones_back` checks the gate's ordering. - `tests/process_supervisor.rs`: `forced_process_group_cleanup_stops_harness_and_worker` and `natural_harness_exit_also_cleans_up_workers` slept 20 ms and then used `kill -0` on a worker that was re-parented to init. `kill -0` succeeds for a zombie, and init reaps it on its own schedule. The tests now check that the process is not running (gone or a zombie), with no sleep. Cleanup already proves that before it returns. - No product code changes. The reap's orphan rule is unchanged. Audit of the other tests in the job (`check:runner`): - Only the runner-core lib binary can overlap the reap. `cargo test --workspace` runs test binaries one after another. - A second sweep, after review, checked every `/bin/sh` launch and every external command (`sleep`, `cat`, `perl`, `nohup`, `touch`, `mv`) in runner-core's `src/`, including `src/tests/*.rs`. The first sweep had missed tests that start a provider through `CodexProvider::start` or that live in `src/tests/`. - Other lib tests that start processes are not affected. They run shells that use only builtins (`configured_environment`, the other `process_supervisor` and `codex_provider` tests, `acpx_provider_session`), or run a short foreground `touch` that finishes long before teardown (`disconnected_codex_output_is_durable...`), or they start direct children that the test waits for (`oom_score`, `exited_writer`, `process_group_leader`). `publisher_crosses_real_codex_tool_boundary...` is `#[ignore]`. - The other `reap_journals` tests refuse before the orphan rule applies, or (the orphan test) require only their own orphan in `remaining`. - The `kill -0` checks in `tests/codex_provider.rs` and `tests/acpx_park_preparation.rs` target direct children that runner-core waits for before it returns, so they are deterministic. ## Verification Linux runs used Colima on macOS (linux/arm64), `rust:1.97.1-bookworm`, `--release`, and `docker run --init`, so orphans go to an init at PID 1, as on CI. - Full runner-core lib suite, `--cpus 2`, `RUST_TEST_THREADS=8`, 40 runs: before the fix, 3 runs failed with the CI signature. After the fix, 0 runs failed, and no other test failed. - The reaping test beside a temporary test that keeps leaving short-lived orphans (not committed), `--test-threads 2`, 10 runs: before the fix, 10 failed. After the fix, with the temporary test holding `may_orphan()`, 0 failed. - An orphan-leaving shell outside the test binary made the unfixed test fail in 9 of 10 runs with the same signature. This confirms the mechanism. - After the review fix: macOS `cargo test --release -p paperclip-runner-core --lib -- receipt_limit late_park_observation a_held_cancellation process_lineage process_supervisor orphaned_startup exited_child_reader_deadline a_timed_out_turn_start`: 65 passed. Linux full lib suite (same settings) ×20: 0 failures. - macOS (darwin-arm64): `cargo fmt --all -- --check` passes. `cargo test --release -p paperclip-runner-core --lib -- process_lineage process_supervisor orphaned_startup exited_child_reader_deadline a_timed_out_turn_start`: 51 passed. `cargo test --release -p paperclip-runner-core --test process_supervisor`: 7 passed. - Worker cleanup tests: with `tini` as init, 60 unfixed runs on 0.5 CPU with busy loops all passed, because `tini` reaps at once. In a container whose PID 1 never reaps (`sleep infinity`), the unfixed tests fail at `assertion failed: !process_exists(worker_pid)` because the killed worker stays a zombie. The fixed tests pass. The unfixed tests depend on how fast init reaps. ## Risks - Low risk. All changes are in tests or `#[cfg(test)]` code. - The gate covers the orphan producers that exist now. A new lib test that leaves an orphan of this user must hold `may_orphan()`. The `host_orphans` doc comment says so. Otherwise, it can make the verifying reap refuse again. - A process outside the test binary that leaves an orphan of this user during the reap still makes the test refuse. This is correct product behavior. CI runs no such process during `cargo test`. - `no_foreign_orphans()` waits until every `may_orphan()` guard is released, so the reaping test can wait a few seconds behind those tests. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, Docker). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
381b862135 |
fix(native): let a finish through after one stale-result refusal; rebuild a missing instruction bundle (#20833)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run ends its task with a Paperclip terminal call
(`paperclip_finish` or `paperclip_block`)
> - The Runner refuses a terminal call that the agent decided before it
saw earlier tool results (`terminal_call_before_results`). This protects
against stale calls held across a restart.
> - On staging, Codex cancelled a nested exec cell. Codex never sent
`item/completed` for one Paperclip tool call in that cell. The Runner
then refused every later finish and block in the turn, and the turn
ended with no semantic result.
> - The same-run recovery then resumed the run on a new controller
container. The run's admitted instruction bundle is stored on the
controller's local disk, so it was missing there, and runnerd failed
with ENOENT.
> - This pull request makes a refusal name a returned call once, and
rebuilds a missing admitted instruction bundle before the Runner backend
starts.
> - The agent-directory lock part of the same failure chain ("Agent
directory operation is already active") is fixed separately by #20829.
> - The benefit is that the agent can finish after one refusal, and a
resumed run does not fail because a deploy moved it to a host without
the bundle.
## Linked Issues or Issue Description
Refs #20829 (the agent-directory lock wait and dead-owner release; this
PR does not change that lock).
**What happened?**
Staging Codex runs on Daytona sandboxes failed after about 30 minutes of
work:
1. Early in the turn, the agent called `paperclip_search_assigned_tools`
from a nested exec cell. Codex cancelled that cell. The controller
returned the result, but Codex never sent `item/completed`, so the
Runner never committed `tool.execution.completed` for it.
2. At the end, the agent called `paperclip_finish` four times and
`paperclip_block` once. The Runner refused each one with
`terminal_call_before_results` ("Those calls have now returned"). The
old call stayed "returned but unseen" for the rest of the turn.
3. The turn completed with no semantic result
(`native_finalization_missing`). Same-run recovery scheduled another
attempt.
4. On the new controller container, the attempt failed with
`runner_ingress_unavailable: runnerd process failed: ENOENT ... scandir
'/paperclip/instances/default/runtime-context-assets/bundles/<sha>'`
(one run) or `ENOENT ... lstat` of the same kind of path (a run during a
rolling deploy). In both, `<sha>` is the run's admitted instruction
bundle. The asset root is controller-local, and the new containers had
no volume.
5. The last attempt failed with "Agent directory operation is already
active" (fixed by #20829).
**Expected behavior**
The agent finishes after one refusal. A resumed run rebuilds its exact
admitted instruction bundle on the new host.
**Steps to reproduce**
1. Run a Codex task that calls a Paperclip tool, then
`paperclip_finish`.
2. Drop Codex's `item/completed` for the first tool call (new chaos
fault `--faults PV2 --provider-fault-message tool_completed`).
3. On master, every `paperclip_finish` is refused and the run ends with
`native_finalization_missing`.
4. Separately, delete `runtime-context-assets/bundles/<instruction
digest>` and resume a run that pinned it: runnerd fails with ENOENT.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
57f1afdf19 |
fix(daytona): retry a sandbox create that timed out before any agent work (#20828)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a Daytona sandbox environment, every new task run first creates a fresh sandbox > - When one Daytona create did not start, the plugin waited the full 300 s acquisition budget > - The run then failed `setup_failed` with `lease_acquire_failed`. No agent work had started, but nothing retried it > - On staging this hit two agents in 15 minutes. Each attempt used about 5 minutes, and the task stopped until a person pressed Retry. The manual Retry then got a sandbox in 28 s > - This pull request ends a stuck create early, adopts a create that started late, and routes the timeout to the bounded bootstrap retry that already exists for transient provider refusals > - The benefit is that a stuck create costs about 90 s and recovers by itself. The task shows Retry only after the bounded attempts are used up ## Linked Issues or Issue Description Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait out a Daytona state change), Refs #20609 (force a stuck stop). **What happened?** A run on a Daytona sandbox environment failed after 307 s with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending`. The run recorded `errorCode: setup_failed`, `failureCause: lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. The task became blocked with Retry. No automatic retry was scheduled. **Steps to reproduce** 1. Use a Daytona sandbox environment with the default `timeoutMs` (300,000). 2. Start a run while Daytona keeps the new sandbox in `creating`. 3. See that the run waits 300 s, fails `setup_failed`, and is not retried. **Expected behavior** A create that has not started before provider work must not cost 5 minutes. The run must retry by itself within bounds. A late sandbox that has started must be used. A sandbox that was abandoned must be removed. **Root cause** - `packages/plugins/sandbox-providers/daytona/src/plugin.ts` (`onEnvironmentAcquireLease`): the only bound on the create step was the whole acquisition budget (`timeoutMs`, default 300 s). - `server/src/services/environment-provider-unavailable-error.ts` (`readTransientEnvironmentProviderFailure`): only an HTTP status or errno on the create call counted as transient. The timeout envelope carries neither, so the run fell through to generic `setup_failed`, which `isTransientWorkspacePreparationCode` does not retry. ## What Changed - Daytona: one fresh acquisition is now capped at 180 s. A create that has not returned after 90 s is looked up once by its creation name. A sandbox in `started` is adopted, and setup continues. A sandbox in `pending_build`, `building_snapshot`, or `pulling_snapshot` keeps waiting inside the cap. Any other state ends the attempt with the existing cleanup envelope and an acquisition diagnostic whose elapsed time reached the expired budget. - Daytona: a late SDK failure no longer runs the compensating delete for a sandbox that was adopted. - Host: `readTransientEnvironmentProviderFailure` reads the validated acquisition diagnostic. An acquisition whose first failure was its own deadline is transient. The run fails with `environment_provider_unavailable`, category `timeout`, and uses the bounded bootstrap retry (30 s, then 60 s). An earlier setup failure stays `setup_failed`. Message text is never parsed. - Host: the task reason for this case is plain: "The sandbox for ... did not finish starting in time. No agent work had started. Paperclip retries automatically ...". - Docs: `doc/execution-semantics.md` and `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`. ## Verification Red before the fix, green after: - `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh sandbox create that timed out through bounded bootstrap recovery" uses the staging error shape through the orchestrator. Red: `errorCode` was `setup_failed`. Green: `environment_provider_unavailable`, category `timeout`, bounded retry scheduled. The `started` and `early_failure` cases stay `setup_failed` with no retry. - `server/src/services/environment-provider-unavailable-error.test.ts`: deadline diagnostic is transient (red, then green). An early failure is not. The task message is plain. - `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends a create still pending at 90 seconds" (red: still pending at 90 s; green) and "adopts a create that started by 90 seconds" (red; green, and a late SDK rejection does not delete it). "keeps waiting for a create that is still preparing its snapshot" is a safety case. - Existing tests that encoded the 300 s acquisition budget now use the 180 s cap. Commands run locally: - `npx vitest run src/plugin.test.ts` (Daytona): 414 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The run-completion file reports an `afterAll` embedded-Postgres teardown timeout. The same timeout occurs on unchanged `origin/master`, so it is a local platform fault. - Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass. - `pnpm verify`: asked for broader CI coverage, because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. Upgrade safety: nothing new is persisted. A task already blocked by this failure keeps `canRetry: true`. Its Retry starts a new run with a new creation record, and that run uses the new path. On staging, the manual Retry of the affected task got a fresh lease in 28 s. The creation record of the earlier attempt stays with the creation reconciler. Not verified: a live Daytona create that stays in `creating`. That cannot be forced on demand. ## Risks - A legitimately slow create that stays in `creating` or `starting` for more than 90 s now ends the attempt and retries. Snapshot preparation states still wait inside the 180 s cap. Staging creates normally take about 20 to 30 s. - An abandoned create is removed by the creation reconciler. It can stay alive until that reconciler settles the record (the existing settle window). This PR does not change that window. - Other sandbox providers get the host classification only when they send the same acquisition diagnostic. ## Model Used Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code with tool use (shell, file edits, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
369e238295 |
fix(projects): make workspace storage the project's only placement choice (#20835)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A repository project gets its working files from core staging or from a workspace storage plugin > - #20459 reported that an installed storage plugin attached every new repository project, so sandbox agents could not use those projects > - #20587 fixed creation with an **Available in** choice that selects storage > - #20651 then saved an environment on every new project and used it before the agent default to place runs. That solved a problem that did not exist: the project needs storage, not a run environment > - Creation saves the first environment of the chosen group, so a project silently sends every agent's runs to that environment. Project Configuration also shows two controls (Environment and Workspace storage) for one decision > - This pull request removes the project run environment and makes storage the only placement choice a project owns. Storage is fixed at creation > - The benefit is that runs follow the agent again, and project settings show one clear, read-only fact about storage ## Linked Issues or Issue Description Refs #20459. Follow-up to #20587 and #20651. Refs #11644. That issue asks for a project-level run environment override. This pull request removes the override that #20651 added for it, because the override is not necessary to fix #20459. #11644 stays a separate feature request. Related: #18176 (open) repairs placement through the legacy workspace-policy field. This pull request does not restore project placement, so it conflicts with the direction of #18176. ## What Changed - **Remove the project run environment.** Migration `0466` drops `projects.environment_id`. Run placement no longer reads the project. Runs use the agent default, then the instance default, then local execution. The low-trust task sandbox and the enforced sandbox policies keep their precedence. - **Keep the #20651 recovery and binding guards.** A recovered run still uses its admitted environment lease (source `run`). Company-binding checks before admission and during lease creation stay. - **Keep the creation choice.** **Available in** in New Project still selects storage. It still shows only when the environments need more than one storage owner. That can only occur when a storage plugin is installed. `POST /projects` with `environmentId` selects the storage of that environment and does not save the environment. - **Fix storage at creation.** `PATCH /projects/:id` rejects any storage change with `409 project_storage_locked`, from core to a plugin and from a plugin to core. Before, a core project could move to a plugin. - **Drop legacy writes.** Older clients can still send `executionWorkspacePolicy.environmentId`. The service removes it on create and update. - **Simplify Project Configuration.** Remove the Environment and Workspace storage selectors. Show an advisory **Available in** list of the environments that can use the project's storage. Show it when a storage plugin is installed or when the project uses one. When the project's plugin is not installed, show **Storage plugin unavailable**. Without a storage plugin, show nothing. - **Issue workspace card.** Remove the "project default" environment label. - **Shared contract.** Remove `Project.environmentId`, the `environmentId` project field, and `expectedStorageProviderPluginId` from the update schema. The create schema keeps `environmentId` and `expectedStorageProviderPluginId` from #20587. - **Docs and maps.** Update `doc/project-environment-compatibility.md`, `docs/api/goals-and-projects.md`, and the project feature and agent-access maps. - **SDK source pins.** Refresh `tests/runner-e2e/source-sdk-public-inputs.json` for the two shared files. The shared file blobs return to their values before #20651. The refresh also corrects the `pnpm-lock.yaml` pin and `lockSha256`: `master` pins an older lockfile blob than the one it contains. ## Verification - Targeted server suites pass (227 tests): `project-environments`, `project-workspace-provider.integration`, `heartbeat/environment-selection`, `execution-workspace-policy`, `environment-selection-route-guards`, `environment-run-orchestrator`, `openapi-routes`. - Related server suites pass (349 tests): `environment-service`, `environment-routes`, `plugin-workspaces.integration`, `issues-service`. - Targeted UI suites pass: `ProjectProperties.concurrency`, `IssueWorkspaceCard`, `NewProjectDialog`, `NewProjectDialog.managed-sandbox`, `ProjectDetail`. - New tests: - Creation saves storage, not an environment. Core and plugin projects both reject a storage change with `project_storage_locked`. - A legacy policy `environmentId` is dropped on create and update. - A new run follows the agent default for any project. A recovered run keeps its admitted lease after the agent default changes. - Project Configuration has no storage or environment selector. The note is hidden without a plugin. It lists the core or plugin environments for the project's storage. It shows **Storage plugin unavailable** for a removed plugin. - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck`, and `pnpm --filter @paperclipai/shared typecheck` pass. - `pnpm --filter @paperclipai/db check:migrations`, `pnpm check:token-gates`, `node scripts/check-maps.mjs`, and `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` pass. - Not run locally: the full `pnpm test:run`, `pnpm build`, and browser suites. CI supplies that coverage. - Manual check: open Project Configuration on an instance with a storage plugin. Confirm that there is no Environment or Workspace storage selector, and that **Available in** lists the environments. ## Risks - **Migration.** `0466` drops `projects.environment_id` and its foreign key. The migration deletes the saved project environments. Apply it before you serve the new revision. An old server revision that reads `projects` fails until it stops. This is the same pattern as earlier drop-column migrations (`0229`, `0289`). - **Behavior change for existing projects.** Projects created after #20651 shipped have a saved environment that overrides the agent. After this change, their runs follow the agent default again. This is the intended fix. Before #20651, a project environment selection was not applied to runs (#11644), so no older behavior is lost. - **No path from core to a plugin.** A core project, including one created before a plugin was installed, cannot move to the plugin. To use plugin storage, create a new project. - **API.** A `PATCH` that sends `environmentId` or `expectedStorageProviderPluginId` no longer has an effect. The schema strips these fields. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with tool use (shell, file edits, test runs). Extended thinking was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0b8ba1ac92 |
fix(queue): show messages a booting run's first turn takes as sent (#20822)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task chat lets a person send follow-up messages to a running agent, and steer them into the live turn > - A sandboxed run (for example on Daytona) can take minutes to boot before its provider session exists > - A message sent during that boot cannot be steered, so it waits in the steering pane > - The runner then puts the message into its first turn, and the server removes it from the queue, but the view has no signal for this, so the message stays in the steering pane as "queued" > - That stale local entry also hides the queue id, so Steer and Discard fail for every later message > - This pull request tells the view which messages the first turn takes, and drops local queue entries whose run has settled > - The benefit is that a message sent while the sandbox boots shows as sent at once, and later messages keep working Steer and Discard ## Linked Issues or Issue Description **What happened?** A person sent a follow-up while the agent's sandbox was booting. The message stayed in the steering pane with Steer unavailable. After the agent started, the message stayed there. The agent had received it in its first turn. Later, Discard on that message failed with "Couldn't discard. Message is still queued.", and a new message also stayed in the steering pane. **Steps to reproduce** 1. Assign a task to a Paperclip Runner agent that runs in a Daytona sandbox. 2. While the sandbox boots, send a follow-up message. 3. Wait for the agent's first response. The message is still in the steering pane. 4. After the run hands off to a successor run, send another message and try to discard the first one. **Expected behavior** A message sent while the run is starting shows as a sent message. Later messages keep working Steer and Discard. **Actual behavior** The first message stays in the steering pane. Discard fails, and the new message cannot be steered until the page reloads. ## What Changed - `firstTurnQueuedCommentIds` (server, `first-turn-queued-input.ts`) returns the messages in a deferred queue that the target run's first turn already sealed, or will take while it is still starting. It uses the same eligibility rule as `claimFirstTurnQueuedInput`. It reads only lean columns, so the queue poll does not transfer run snapshots or large wake payloads. - The queue read path and the queue mutation path both put these ids in a new optional `firstTurnCommentIds` field on `IssueQueuedCommentQueue`. - The task view puts each of those messages in the conversation as input of that run, at its submission time, and not in the steering pane. - The task view no longer treats a saved message as a pending local queue entry after its target run is no longer live. A request that is still in flight keeps its original target. ## Verification - Red-green server test: `issue-queued-comments-routes.test.ts` "reports the messages a preparing/native run's first turn takes, before and after it seals them" and "reports no first-turn messages once the run's turn already started". Before the fix, `firstTurnCommentIds` was `undefined`. - Red-green UI tests: `IssueDetail.test.tsx` "shows a message the booting run's first turn takes as sent, not queued" (still starting, and sealed) and "keeps queue controls for a new message after the run that took an earlier one hands off". Before the fix, the queue held `['new-message', 'boot-message']` and the queue id was withheld. - Reproduced on staging first: the run's context held a first-turn receipt for the stuck message, and the server queue had no entries, while the view still showed it as queued. - `npx vitest run` for `issue-queued-comments-routes`, `queued-comment-consumption`, `queued-comment-postgres`, `queued-comment-use-cases`: 172 passed. The existing queue-poll byte budget tests still pass. - `npx vitest run src/pages/IssueDetail.test.tsx src/components/task-chat src/lib/issue-queued-comment-queue src/lib/optimistic-issue-comments`: 1288 passed. - `tsc --noEmit` for shared, server, and ui: no new errors. - `pnpm verify` exits 2 because the change touches `packages/shared`. Required CI supplies the broad coverage. ## Risks - Low risk. The new field is optional and only for display. Delivery still belongs to the first-turn claim and the deferred wake. - If a starting run fails before its first turn, the field goes away and the message shows in the queue again for the next run. This is correct, because the deferred wake still holds it. - The queue read adds one lean run read and one comment read only while a deferred queue targets a running run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4529dfb77f |
fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only
## Linked Issues or Issue Description
Refs #20808
**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.
**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.
**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.
## What Changed
- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.
## Verification
- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in
|
||
|
Nicky LeachandClaude Opus 5.5
|
ff8ec41e6a |
feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
34f2bb78ee |
feat(connections): connect Sentry with a long-lived user auth token (#20823)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use app connections to call provider tools, such as the
Sentry MCP server
> - The Sentry connection has only one method: browser sign-in through
the hosted MCP OAuth server
> - That grant expires. An unattended triage agent loses Sentry access
until a person signs in again
> - During that time, a scheduled triage routine completes each run but
reads no Sentry data
> - This pull request adds a second Sentry method that uses a user auth
token
> - The benefit is a Sentry connection that stays valid until a person
revokes the token
## Linked Issues or Issue Description
**Subsystem affected**
App connections: the Sentry entry in the connection catalog.
**Problem or motivation**
A Sentry connection that was created with browser sign-in on 2026-10-11
changed to "OAuth authorization expired. Reconnect this app to
continue." at 04:01 UTC on 2026-11-10. The agent that used it had no
Sentry tools after that time. Each hourly triage run reported the
blocker but could not continue. Unattended agents need a Sentry
connection that does not need a person to sign in again each month.
**Proposed solution**
The hosted Sentry MCP server accepts a Sentry user auth token in the
`Authorization: Sentry-Bearer <token>` header. It reserves `Bearer` for
its own OAuth tokens. Paperclip API-key methods already support a custom
header prefix. Add a Sentry API-key method that uses this header.
**Alternatives considered**
Reconnect with browser sign-in again. That restores access only until
the next grant expiry.
## What Changed
- Add the `mcp-user-token` method ("Use a user auth token") to Sentry in
`scripts/ingest-app-definitions.mjs`. It uses the same server URL and
sends the token as `Authorization: Sentry-Bearer <token>`.
- Regenerate `packages/shared/src/app-definitions/sentry.json`. I did
not include unrelated output that the content corpus drift caused for
other providers.
- Add a permission review entry for the new method in
`doc/connections/tool-method-permission-reviews.json`.
- Add a test for the new method and its header prefix.
- Document the token method, its required scopes, and its ownership risk
in `doc/connections/SENTRY.md`.
## Verification
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts`: 40
passed. The new assertion fails on `master` because Sentry has only one
method there.
- `pnpm exec vitest run src/__tests__/tool-access-service.test.ts -t
Sentry` (server): 2 passed.
- `pnpm exec vitest run ui/src/pages/apps/app-connect-policy.test.ts
server/src/__tests__/satisfied-connection-intents.test.ts`: 64 passed.
- `pnpm verify` returned exit 2 (a doc is a global CI input). Required
CI supplies the broad coverage.
- Live probe of `https://mcp.sentry.dev/mcp` with an `initialize`
request: `Authorization: Sentry-Bearer <invalid token>` starts an MCP
session (the token is checked on tool calls). `Authorization: Bearer
<same token>` returns `401 invalid_token`. This confirms the server
accepts the header scheme.
- Not done yet: a real agent tool call with a real token after
deployment.
## Risks
- Low risk. This adds a method. The existing OAuth method and saved
connections do not change.
- A user auth token acts as its creator and lasts until revoked. The
method guidance and docs say this. Rotate the token when that person
leaves.
- If Sentry changes the `Sentry-Bearer` scheme, connections that use
this method fail on tool calls. The OAuth method still works.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, web search).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude
|
fd1460bb24 |
fix(native): continue a governed-wait follow-up in a fresh session when its saved provider state is gone (#20809)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks and breaks the control plane on purpose, so failures show up before staging users see them > - A census of 48 hours of staging failures found runs whose result save-back failed because the sandbox disk was full (class F). The harness never filled a sandbox disk, so it missed that class > - #20758 made a full-disk save end recovery at once with cause `sandbox_disk_full` and a Retry. Nothing in chaos exercised that path end to end > - A real full disk cannot be made on demand. The owner chose a test-only hook that makes the save report a full disk, not a small disk > - This pull request adds the hook at the save's sandbox boundary and a DF1 fault that arms it, clears it, and grades the result > - The benefit is that every wave can now check the disk-full rest, the Retry offer, and that Retry works after space is freed ## Linked Issues or Issue Description Refs #20758 (product behaviour under test), #20678 (finalization hands a full-disk copy-back to recovery) **What is the problem?** The chaos harness had no fault for a full sandbox disk during result save-back. Staging saw this failure (census class F in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos could not reproduce it. **What is the expected behaviour?** A wave can schedule DF1. While DF1 is armed, the task's final workspace saves fail as a full disk. When DF1 is cleared, saves succeed again. The task must rest with `sandbox_disk_full` and a Retry, and that Retry must finish the task. **Why does it matter?** Without a fault, a regression in the disk-full path (no Retry, a wrong cause, or a Retry that does not work) reaches staging users before chaos sees it. ## What Changed - `server/src/services/sandbox-disk-full-fault.ts` (new, test-only): `sandboxDiskFullFaultTarget` wraps the target of a final save only when the harness armed that save. The capture commands then exit 1 with the exact message the in-sandbox script writes on ENOSPC, and the provider download fails with `No space left on device`. The gate follows `@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL` is read once at import and ignored under `NODE_ENV=production`. Disarmed, the hook is one comparison and returns the save's own target. - `server/src/services/remote-workspaces.ts`: the capture call in `saveRemoteWorkspace` passes its target through the hook. No other product change. `workspace-recovery.ts` is not touched. - `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the control-file seam (atomic writes), the rest evidence (`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`), the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry that must finish the task). - Harness wiring: `--faults DF1` (refused without `--env daytona`, with a storm, or with PV faults), no scheduled restart, arm at launch, clear at the first rest just before the Retry click, receipts in `<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: { faultedSaves, restedDiskFull, retryOffered, retryWorked }`. - Docs: a "Sandbox disk full (DF1)" section in `tests/restart-chaos/README.md`, and the census plan marks DF1 as built. ## Verification - `cd server && pnpm exec vitest run src/services/sandbox-disk-full-fault.test.ts`: the gate is off by default and under `NODE_ENV=production`, the save's own target comes back when disarmed, only a listed final save is faulted, the product capture turns it into the named disk-full failure, and clearing the file disarms the next save. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/disk-full-faults.test.ts tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts tests/restart-chaos/provider-faults.test.ts`: planning, the control file, and grading. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the server `tsc --noEmit`: clean. - Live wave: a Daytona DF1 wave is running. Its result is added here before merge. ## Risks - Low risk for production. The hook reads its variable once at startup and is ignored under `NODE_ENV=production`. Without the variable it does one comparison and returns the same target object. - The hook fails every sandbox command of an armed capture, including the read-only fallback's inventory. A real full disk can let that inventory succeed and fail later at the download. The outcome is the same (`sandbox_disk_full`), but the fallback's inventory step is not exercised under DF1. - DF1 runs only with `--env daytona`. A local run has no sandbox workspace save, so the harness refuses DF1 there. ## Model Used - Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
81c9143fe4 |
fix: upgrade Better Auth to 1.7.16 with account migration (#18906)
## Thinking Path > - Paperclip manages AI agents and their work. > - The board uses Better Auth for credential accounts and sessions. > - The server pins Better Auth 1.7.2. > - The November security release fixes 12 issues in Better Auth. > - The new version stops writing the old required account issuer field. > - This change updates the packages and the database contract together. > - New users can sign up, and existing users can still sign in. ## Linked Issues or Issue Description **What happened?** The server uses Better Auth 1.7.2. The [security release](https://better-auth.com/blog/security-release-2026-11-03) fixes 12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7, which is below this security release. **Expected behavior** Use the patched stable release. Keep new and existing credential accounts usable. **Steps to reproduce** Read the server dependency pin. Upgrade it without a schema change. Better Auth then rejects the required account issuer column. See the [upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide). ## What Changed - Pin Better Auth to 1.7.16. The repository refresh bot owns the committed Core lockfile. - Make the old issuer column nullable. Keep existing issuer values. - Replace the issuer index with a unique provider/account index. - Generate the migration and retain the five newest schema snapshots. - Test new signup, new-account sign-in, legacy issuer sign-in, duplicate rejection, and separate provider identities with real PostgreSQL. - Update the database schema test. Test the historical issuer backfill separately. Verify upgrades over existing rows and rollback on duplicate provider/account keys. - Document the duplicate check and migration order. - Add the approved two-line dependency resolution step before the backup job’s frozen install. Leave the frozen check and backup tests intact. ## Verification - Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent `0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`. - Nine changed files. The only workflow edit is the approved two-line backup dependency resolution step; no managed lockfile edits. - Migration `0465_better_auth_account_identity.sql` is three SQL statements; snapshot changes only `public.account`, five newest snapshots retained. - DB typecheck and 23 focused tests passed on the parent. Both real PostgreSQL migration/signup suites were rerun November 11: all 4 tests passed. Application source and tests are unchanged in this workflow-only commit. - The parent’s sandbox-provider lane passed all 205 file-sync tests but reported an unhandled EPIPE during the symlink rejection case; its aggregate verify therefore failed. No unrelated source change was made. The new head reruns the full applicable CI. - Reproduced the old backup install failure locally. Resolving manifests followed by a frozen install passed locally; restored the bot-owned committed lock afterward. Hosted backup qualification remains required. - Current-head CI and review are running; no final-head green claim or merge yet. ## Risks Apply the migration before the new server starts. Check for duplicate provider/account pairs first. Duplicate rows stop index creation. Keep separate users separate when resolving duplicates. The migration keeps issuer data. Do not restore its NOT NULL constraint while new rows contain null. No deployment is included. ## Model Used OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
84fc67317e |
feat(runner): export authorized read responses as artifacts (#20606)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use authorized API reads to get task evidence. > - Small responses return inline JSON, so an agent must copy each document into a local helper. > - Copying changes bytes and uses model output for data transport. > - This pull request adds an explicit artifact export for non-sensitive GET reads. > - Helpers can download the exact response bytes and check their identity. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native `call_api` response capture path. **Current behavior** Small JSON responses have no artifact reference. Local helpers need the model to copy the JSON data. **Proposed behavior** `saveResponse: true` saves a complete authorized GET response below or above the inline threshold. Zero-byte success returns an explicit empty receipt without storage. The receipt returns an artifact ID, byte size and SHA-256 for `download_artifact`. Other callers keep the current behavior. Related work: #17393 covers saved app-tool result size. #18105 covers mutation response paging. This change adds an explicit read export. ## What Changed - Add GET-only `saveResponse` validation and capture through the current artifact store. - Refuse sensitive operations, denied reads, partial responses and combined text paging. - Keep route authorization, actor visibility, storage budgets and download integrity checks. - Update the native tool contract, generated catalogs and API guide. - Add boundary and real-route byte comparison tests. - Preserve saved run state. This is an optional input field; old calls and receipts keep their current shape. No schema migration or identity reset is needed. The seeded catalog fixture is generated. ## Verification - Final published head `d47b52e1bf412418e24768f865cc139097a6d068`: [typecheck, build, all tests, Linux Rust and aggregate verify passed](https://github.com/paperclipai/paperclip/actions/runs/42918883477); [Windows Rust passed](https://github.com/paperclipai/paperclip/actions/runs/42918883348); [quality gate passed](https://github.com/paperclipai/paperclip/actions/runs/42918883016). Greptile completed successfully at this exact head, 5/5; all five review threads resolved. Automated Paperclip review remains queued due to a failed reviewer run; no final verdict is claimed for that service. - Current head: `d47b52e1bf412418e24768f865cc139097a6d068`. Merged current master to refresh GitHub's stale base snapshot; the PR now contains exactly nine intended artifact transport files, with no lockfile or UI changes. Fresh final-head CI/review is required. - Previous head `986ad87490fff93cfec313ba6fd56d3dc0bb665b` passed typecheck, build, Rust verification, artifact/native suites and completed test shards; remaining shards were still running when the base snapshot was refreshed. Greptile was 5/5, with one unrelated master UI finding resolved as outside the corrected PR diff. - Review fixes verified at `986ad87490fff93cfec313ba6fd56d3dc0bb665b`: 2,135 checks passed across the five focused suites; Runner TypeScript build and regenerated contract/catalog checks passed. This head replays the original implementation on current master and retains the original PR commit in its history. - Review red/green at replayed baseline `da3a1d4de9`: empty HTTP 200/204 tests failed with real storage's `File is empty`, then passed with zero new assets or quota reservations. Error-body tests failed without size/truncation fields, then passed. - Empty successes now return `empty: true`, `byteSize: 0`, `data: null` without storage. Nonempty JSON results such as `[]` export normally. Error receipts report byte size and truncation. Docs and tool guidance state the existing 10 MiB download limit and paged text alternative. - Previous-head CI typecheck/build/tests passed except the Rust process-lineage cleanup test (`unattributed_orphan`); final-head CI and fresh review remain required. - Red at `7368a0b15c`: the small-document export test failed with `Invalid call_api arguments` before production edits. The same check passed after the change. - Final post-rebase run at `6e7a35ed6a59bb74f4218312d9f10557099433c3`: all 2,124 checks passed across the API, capture, artifact boundary, real-route export and artifact download suites. - Real-route export test passed: HTTP bytes, stored bytes and downloaded bytes were equal. Foreign-company access and forged download metadata were rejected. - Existing API integration and artifact suites passed. Nearby API integration checks also passed (59 passed, 2 skipped). - Disposable native helper scan passed: 12 existing documents, 15 GET requests, zero mutations, 73,919 bytes, 17.985 seconds. All files passed size/hash checks. The bridge passed file references only. Production token savings were not measured. - Runner TypeScript build and generated contract checks passed. - `pnpm verify` returned exit 2 and required broader CI coverage. Local preflight stopped at missing Cargo after dependency and SDK setup. The local server typecheck stopped during the cycle layer without a diagnostic; the container recorded an OOM kill. This check has no pass verdict. Required CI must cover typecheck, build and test at the final head. ## Risks - Each explicit export uses the current capture budget and storage admission. Frequent small exports can reach that budget. - An artifact holds one HTTP response. Callers must still follow list pagination. - No database or saved-state migration is required. The option does not change default response behavior. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in Paperclip Runner. Used reasoning, shell execution and native tools. The active agent configuration supplied this exact model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
4d141cdf64 | Carry plugin task refusals and batch project reads (#20762) | ||
|
DottaandClaude
|
ea25af759e |
fix(native): name a resume that finds no saved Runner state, so Retry starts a fresh session (#20757)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can continue the provider session of the previous run on the same task. In a remote sandbox, the Runner keeps the session state inside the sandbox, and the server saves a harness backup after each run. > - On staging, five runs in 48 hours failed with `native_session_recovery_failed: Error: PRP provider resume state is unavailable` (class P in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`). > - In each case we read, the previous run had an accepted result, but its `harness_state.backup.persist` failed. Its per-turn sandbox was then destroyed. The next run started in a new sandbox with no Runner state for the session. > - The Runner transport threw an untyped error. The runtime could not tell it apart from a transport drop. A governed-wait or same-session checkpoint ended as a generic `native_session_interrupted` with an "ambiguous state" hold that said not to open a replacement session. > - This pull request types that refusal and lets the driver classify it as `provider_state_incompatible` when nothing is in flight. The existing recovery policy then decides. > - The benefit is that the run ends with the permanent named code `native_provider_state_incompatible`. Retry, or the next message, continues in a fresh provider session in the same workspace. No provider turn or tool call can be repeated, because the follow-up never started a turn. ## Linked Issues or Issue Description Refs #20712, #20723 (the same classification for a moved remote workspace root). Refs #20691 (fixes the `provider_identity_incomplete` part of class P). **What happened?** 1. Run A on a chat task finished. Its result was accepted. Its harness backup failed to save (`harness_state.backup.persist` failed, `native.post_result_failure`). 2. Run B started for the next chat message in a new sandbox. It rebound run A's checkpoint with `providerRecoveryPolicy: allow_replacement_after_governed_wait`. 3. The Runner found no control-plane state and no archived authority epoch. It threw `PRP provider resume state is unavailable` as a plain `Error`. 4. The runtime refused a replacement for the governed-wait policy and failed run B with the generic `native_session_recovery_failed`. The task stayed blocked with an "ambiguous state" hold. **Expected behavior** The run ends with a permanent named code and a working Retry, as it does for other deterministic refusals. **Steps to reproduce** 1. Create a Runner transport that resumes a saved provider session from an empty state directory. 2. Request `thread/read`. On `master` it rejects with a plain `Error`. 3. Recover a saved Codex, OpenCode or ACPX session through the driver when the transport fails that way. On `master` the result has no `failureClass`. **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `packages/paperclip-runner/src/contracts/harness-driver.ts`: add `NativeRunnerResumeStateUnavailableError` (same message as before) and `isRunnerResumeStateUnavailable`. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: throw the typed error at the three branches where the control-plane state is absent. The branch with no PRP identity and the quarantine branch are unchanged. - `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`: classify the typed error as `provider_state_incompatible` when the saved session has no turn or runtime request in flight. Extract the in-flight test that the moved-root check already used, without changing it. - `doc/execution-semantics.md`: describe the new classification (additive). - Tests: a transport test (local and external Runner state), driver tests for Codex, OpenCode and ACPX Claude, in-flight cases that stay unclassified, and a server case that shows the blocked task offers Retry and that one click starts exactly one fresh-session run. ## Verification - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts -t "no saved state"`: 5 pass. With the driver change reverted, the 3 "reports … as provider state incompatible" cases fail (no `failureClass`). - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-cold-restore.test.ts`: 5 pass. On `master` the error is a plain `Error`. - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts src/live/runnerd-codex-transport-cold-restore.test.ts src/native-session-runtime.test.ts`: 410 pass. - `cd server && npx vitest run src/services/workspace-recovery-retry-copyback.test.ts -t "does not continue automatically"`: 5 pass, including the new case. It checks `canRetry: true`, an accepted Retry, exactly one queued successor, and `forceFreshSession: true`. - `cd packages/paperclip-runner && npx tsc --noEmit -p .`: clean. - `pnpm verify`: exit 2 (broader verification required) because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change only adds a class to a failure that already failed. A checkpoint with `allow_replacement_after_resume_failure` already replaced the session for this failure, and still does. - For `same_session_only` and `allow_replacement_after_governed_wait`, the run now ends with `native_provider_state_incompatible` instead of a retryable `native_session_interrupted`. Same-run retries stop at once. They could not succeed, because the state is absent. - A session with a turn or runtime request in flight keeps the old unclassified behavior. This PR does not change it. - The server code path is unchanged. The Rust Runner is unchanged, so no new runner image is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
9574065a28 |
fix(workspaces): hand on a proven-unchanged preparation epoch instead of abandoning it (#20736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox first restores its workspace's saved version, publishes that root as its dirty epoch, and only then starts work > - A restart that kills the run in that window leaves the epoch dirty. The ownership settle (`settlePreparationOnlyRemoteWorkspaceEpoch`) always abandons it: the root is forgotten and the retry restores the saved version into a new directory > - The saved provider session still names the old directory, which is what #20712 and #20723 now have to handle > - In the common case the killed preparation only restored the saved version and changed nothing. A durable proof for exactly that already exists (`proveRemoteWorkspaceUnchanged`, used when the next admission finds an ended writer) > - This pull request makes the settle use that proof first, and keeps abandoning whenever the proof fails > - The benefit is a stable workspace root across a restart during preparation, with no change for anything unproven ## Linked Issues or Issue Description Refs #20712 **What happened?** In restart-chaos waves, a server kill during a follow-up's preparation (right after the workspace restore published) made the ownership settle abandon the epoch. The retry restored into `<workspace>-<epoch>-<lease>` instead of the published root, and the inherited session checkpoint named a root that no longer existed. **Expected behavior** When durable records prove the killed preparation only restored the saved version, the workspace is handed on at that saved version and keeps its root. Anything unproven is abandoned exactly as before. **Steps to reproduce** 1. Run a remote native task to a saved version, then start a follow-up or continuation run. 2. Kill the server after the follow-up's restore publishes (crash point `remote_workspace.prepare.after_publish_before_provision`). 3. Let ownership reconciliation settle the dirty epoch, then let the retry prepare. ## What Changed - `recovery/remote-workspace-unsaved-files-settlement.ts`: before it abandons a preparation-only epoch, the settle calls the existing `releaseUnchangedRemoteWorkspaceWriterById` (lock the workspace, check the writer, then `releaseUnchangedRemoteWorkspaceWriter`). That releases the writer at its saved version (dirty false, writer cleared, location and root kept, baseline marked released, run reference set to `saved`) only when `proveRemoteWorkspaceUnchanged` holds. The proof requires all of these durable records: - the unchanged-baseline marker for this exact epoch, run and saved version. It is written in the same transaction that publishes the restored root, so a kill mid-restore leaves no marker; - no recorded process, process group, process start or provider session; - no execution-start run event, no native result and no native runner checkpoint (the controller writes one before every runner command); - no workspace operation other than the file-neutral restore, finalize and recovery receipts (provisioning and saves record theirs before they touch files); - every workspace repository already in the saved version. - If the proof fails for any reason, the settle abandons the epoch exactly as before. The existing settle preconditions (run terminal, still `preparing`, no finalization, no work-started event, no save in flight) are unchanged and run first. - `packages/shared/src/crash-point-registry.ts` and `remote-workspaces.ts`: add the crash point `remote_workspace.prepare.after_publish_before_provision`, so the chaos harness can kill exactly at this boundary. It counts only a preparation that published an unchanged baseline (a restore of a saved version), and it is a no-op unless a test arms it. `tests/runner-e2e/source-sdk-public-inputs.json` is refreshed because the registry file is a pinned SDK input. - Test `ownership-reconciliation-report.test.ts`: a proven-clean epoch is handed on with its root kept, with no abandonment receipt, and the writer hold is free for the retry. These cases are still abandoned exactly as before: no baseline (an epoch saved by the previous release), a baseline for another epoch, an already-released baseline, a provisioning operation. A run with a recorded process is still neither handed on nor abandoned by this settle. Previous-release story: an epoch saved by a release that did not write the baseline marker has no proof, so it is abandoned as before. No persisted shape changes; the marker already exists on master. ## Verification - Red: with the release branch disabled, the new test fails on the clean case (the epoch is abandoned). Green with the change. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/remote-workspace-snapshot-reuse.test.ts src/__tests__/crash-point-call-sites.test.ts` passes. `packages/shared` `crash-points.test.ts` passes. `tsc --noEmit -p server` passes. - Live Daytona waves (claude, workload S1, the new crash point at the continuation's preparation (`:2` before the point was gated to saved restores; `:1` now), image built from master `e039be9202`, one task each): | | before (release disabled) | after | | --- | --- | --- | | Killed epoch 2 | abandoned (`workspace.remote_epoch_abandoned`) | handed on (`workspace.snapshot_skipped_unchanged`, epoch 2) | | Retry root | new root `…-4-ecf1c81e…` | same published root `…-1-33e93b90…` | | Retry `task.prepare` | 4598 ms | 4731 ms | | Retry restore / environment acquire | 1141 / 1381 ms | 1106 / 1478 ms | | Task result | done, no Retry | done, no Retry | | Stopped or orphan sandbox at rest | none | none | Preparation time does not change in this setup. The Chaos Lab environment uses ephemeral leases, so every run gets a fresh sandbox and a full restore with or without this change, and the killed run's sandbox is destroyed either way. The gain is the stable root, which keeps the saved session's working directory valid. ## Risks - Low. The release path is the same one admission already uses for an ended writer, under the same proof. The settle now tries it first. - A proof false positive would keep a root whose files changed. The proof refuses on every record that precedes a write, and on any missing or older-release evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
f25fee3532 |
fix(native): hash runner artifacts and run archive tar off the event loop (#20722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process prepares native runs, renews every controller lease on timers, and serves every request > - On a run path, the server hashed the 31 MB runner binary with `readFileSync` (more than once per run), hashed provider packs and harness backups synchronously, and ran `tar` with `execFileSync` (timeouts up to 60 s, one with no timeout) > - On a slow disk, each of these calls freezes the whole process. A 103 s stall on a hosted instance lapsed every lease. The last event before that stall was immediately before a synchronous read and hash of the runner binary (see #20694 and #20709) > - This pull request moves those reads, hashes and archive commands off the event loop. It computes the runner binary digest once per boot > - The benefit is that a slow disk delays only the run that waits for it. Turns also start faster, because the runner binary is no longer re-read for every run ## Linked Issues or Issue Description Refs #20694, #20709 **What happened?** On a hosted instance, the server process stopped for 103 s without a restart. Every controller lease lapsed, and two tasks needed a manual Retry. The last run event before the gap was immediately before `matchesControllerRunnerArtifact`, which does `createHash("sha256").update(readFileSync(controllerRunnerBinary))`. Four workspace restores were running at the same time, and file I/O was very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is not proven. But each synchronous call below blocks the event loop for as long as the disk is slow. **Expected behavior** File hashing and archive commands on a run path must not block the event loop. A slow disk must delay only the run that waits for it. **Steps to reproduce** 1. Make a remote runner binary slow to read (the test uses a FIFO whose writer waits), or make `tar` slow (the test puts a `tar` wrapper that sleeps first on `PATH`). 2. Call `resolveRemotePaperclipRunnerBinary`, `stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a 10 ms timer runs. 3. Before this change, the timer stops for the whole slow step (814 ms, 1,764 ms and 3,379 ms). After this change, it keeps firing. ## What Changed - `server/src/services/native-runtime/off-loop-file-work.ts` (new): - `sha256FileStreamed` - `cachedFileSha256`: a streamed digest, reused while the file's device, inode, size, mtime and ctime do not change - `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file only when the header cannot answer, so the result equals `runnerBinaryTarget(readFileSync(path))` - `sha256DirectoryTreeStreamed` - `execFileBuffered`: `execFile` with stdin input, `maxBuffer`, `timeout` and `env`. It rejects on a non-zero exit, a timeout or a `maxBuffer` overflow, like `execFileSync`. - `native-session-executor.ts`: - `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot reads the binary once, not once or twice for every run. - The remote Claude daemon target is read from the binary header. - The bundled remote runner binary is checked by `verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a cached digest. - Provider-pack upload verification uses the new `verifyRemoteProviderPackManifest`. It runs the same checks, in the same order, with the same errors. One generator drives both the sync and the streamed readers. - Harness backup build and verify on run paths use streamed digests (`buildNativeHarnessBackupManifestStreamed`, `verifyNativeHarnessBackupStreamed`). The digest format does not change. - `stageRemoteRunnerDirectory` (3 calls), `assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract use `execFileBuffered`, with the same arguments, timeouts and buffer limits. `stageRemoteRunnerFile` uses `readFile`. - `packages/paperclip-runner/src/live/runner-binary.ts`: the default daemon lookup (`resolvePackagedRunnerBinary`, used by the local and remote resolvers) classifies the daemon with `runnerBinaryFileTarget`. It reads the 64 KiB header, not the whole file. - `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`: the release-manifest checks now live in one helper, `bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and the server's streamed check both use it. - `native-codex-runner.ts`: - `resolveRemotePaperclipRunnerBinary` reads only the header. - The local `runnerDigest` uses `cachedFileSha256`. - New `verifiedBundledRemoteRunnerBinary`. **Ordering and atomicity.** I checked each converted call for code that relied on the sync call being atomic with respect to other runs: - **Checkpoint copy-back.** It extracts into a private `mkdtemp` directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`) stays synchronous, so no other run can see a half-installed checkpoint. This function already awaited a remote command before the extract, so this change adds no new interleaving with the target. - **Harness backup verify then restore.** The verify was sync, but the restore after it was already async. The window between them existed before. One owner serializes the turns of a session, so no other turn publishes a backup for that session while a turn verifies it. - **Staging and provider-pack hashing.** These read build-owned, immutable trees, or the private staging copies of a run. ## Verification - Red/green: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/off-loop-file-work.test.ts` - With `native-session-executor.ts` and `native-codex-runner.ts` from `origin/master`: 3 failed. The longest timer gaps were 814 ms (runner binary target read), 1,764 ms (directory staging tar) and 3,379 ms (checkpoint list, size check and extract). - With this change: 6 passed. The gaps stay under 400 ms. The same file also checks the digest cache, the header fallback, and the `execFileSync`-like errors (exit code, timeout kill, `ERR_CHILD_PROCESS_STDIO_MAXBUFFER`). - `native-session-executor.test.ts`: - The streamed provider-pack verifier returns the same manifest as the sync reader, and rejects a changed asset tree with the same error. - The streamed harness backup manifest equals the sync one. - The streamed verify agrees with the sync verify, for a valid backup and for a corrupt one. - Run together: `native-session-executor`, `off-loop-file-work`, `native-codex-runner`, `setup-readiness`, `antigravity-connection-probe` (1,653 passed). `run-completion.test.ts` passed all 153 tests. Its `afterAll` embedded-Postgres cleanup timed out locally, which is not related to this change; CI runs it. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - **Turn start.** This change adds no wait before `turn.start`; it removes work. The first run of a boot streams the runner binary once. Later runs reuse the digest after one `stat`. - **Cache staleness.** The cache keys on device, inode, size, mtime and ctime. An in-place rewrite changes mtime and ctime, and a replaced file changes its inode, so a changed binary is hashed again. - **stderr.** `execFileSync` wrote a child's stderr to the server's stderr. `execFileBuffered` captures it. Tar warnings no longer appear in server logs; errors and exit codes are unchanged. - **Persisted state.** No persisted shape changes. Old saved state and new code (and the reverse) read the same manifests and digests. - **Not converted in this PR:** small metadata syscalls (`existsSync`, `lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the native state machines. Some of these rely on running synchronously. Also not converted: the sync harness backup verify inside the sync restart-authority checks, and the sync stamp verify in `native-harness-backup-storage.ts`. Both are listed for a follow-up. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments carry the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
94c7c83ce8 |
fix(runner): refuse a moved-root same-session checkpoint by name (#20723)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox resume a task's provider session from a saved checkpoint > - #20712 made a checkpoint that names an abandoned remote workspace root decline recovery, so a replacement session opens in the new root > - A checkpoint that requires the same session (an unfinished goal, `same_session_only`) cannot be replaced. It then failed with the generic `native_session_recovery_failed`, which recovery treats as a retryable interruption > - A retry cannot move a saved session to another root, so the failure repeats. The upgrade-safe rule requires a permanent, named code for this case > - Also, the working-directory errors said "Codex" on Claude and other ACPX runs, because one app-server driver serves all of them > - This pull request reports the moved root as provider state that is incompatible, and makes the errors name the agent generically > - The benefit is a named, permanent refusal with a working board Retry, and error text that is correct for every agent ## Linked Issues or Issue Description Refs #20712 **What happened?** After #20712, a `same_session_only` checkpoint with a moved remote root ended with `native_session_recovery_failed: persisted session working directory is not the assigned remote workspace`. That is not a named permanent code, so recovery could retry a run that cannot succeed. On ACPX Claude runs, path errors read "Remote Codex working directory ...". **Expected behavior** The run ends with the permanent code `native_provider_state_incompatible`. Automatic retries stop. A board Retry continues the task in a fresh provider session in the same workspace. Errors do not name Codex on other agents. **Steps to reproduce** 1. Save a native checkpoint with `providerRecoveryPolicy: same_session_only` under one remote workspace root. 2. Assign the run a different sibling root (the epoch was abandoned). 3. Resume the run. ## What Changed - `codex-app-server-driver-impl.ts`: the moved-root decline from #20712 now carries `failureClass: provider_state_incompatible`. The runtime already maps that class: - checkpoint allows replacement (`allow_replacement_after_resume_failure`): a replacement session opens in the assigned root, as before; - checkpoint requires the same session: `NativeProviderStateIncompatibleError`, code `native_provider_state_incompatible`. - The server already treats `native_provider_state_incompatible` as a permanent failure (no automatic retry, board action text "Retry the task to continue in a fresh provider session with the same workspace"). `rebindNativeSessionCheckpoint` returns no checkpoint after that code, so a board Retry starts a fresh session in the same workspace. No server change is needed. - A moved-root checkpoint with an active turn or a pending runtime request now reports `provider_side_effect_pending`. The runtime never replaces it under any policy, and the run ends with the permanent code `native_provider_side_effect_pending` for an operator to reconcile. Before, it threw the working-directory refusal, which recovery recorded as the retryable `native_session_interrupted`. - `harness-driver.ts`: the `PROVIDER_STATE_INCOMPATIBLE` doc now names the moved-workspace case. - `codex-boundaries.ts`: working-directory errors say "Agent working directory", "Remote agent working directory" and "Provider thread response" instead of "Codex". Tests updated to the new text. - Tests: - `native-session-runtime.test.ts`: the real app-server backend with `remote_runner` authority and a moved root, for each recovery policy. Same-session policies end with `native_provider_state_incompatible`, the replacement policy opens a replacement in the assigned root, an in-flight turn ends with `native_provider_side_effect_pending` under every policy, and no transport starts. - `native-backend-factory.test.ts`: the decline carries the failure class, and an ACPX Claude refusal reads "Remote agent working directory does not match the assigned workspace". Previous-release story: no persisted shape changes. Saved checkpoints are classified at recovery time. A saved same-session checkpoint under an old root now ends with the named permanent code instead of a generic one. ## Verification - Red: without the failure classes, the same-session and in-flight runtime cases fail (generic `native_session_recovery_failed`). Before the rename, the ACPX Claude message test fails. - Green: `cd packages/paperclip-runner && npx vitest run src/backends src/drivers/codex src/live/runner-prepared-launch-workspace.test.ts src/native-session-runtime.test.ts` passes (784 tests). - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Existing server coverage for the code: `native-session-resume.test.ts` (no rebind after `native_provider_state_incompatible`) and `native-session-executor.test.ts` (source-code mapping). ## Risks - Low. Error message text changes. A search found no code that matches on the old text outside the updated tests. - Same-session checkpoints with a moved root now stop for a board Retry instead of a generic interruption. That is the intended behavior: a retry cannot succeed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
fcf4e39b41 |
fix(runner): do not count stopped time against a turn start or an ACPX sidecar request (#20720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run starts each provider turn with `turn.start`. The control plane waits for the Runner, and an ACPX Runner waits for its sidecar, which starts the provider and sends the prompt. > - Both waits use a 30 s bound. When the host or container stops the whole tree (SIGSTOP, a host sleep), the clocks keep moving while no process can act. > - On thaw both bounds have passed. The control plane reports `PRP command turn.start timed out`, and the Runner fails the sidecar request and closes the ACPX session. > - #20704 added a stopped-time clock for startup and recovery. This pull request puts `turn.start` and its follow-up waits on the same clock, and adds the same rule to the Runner's sidecar response wait. > - The benefit is that a freeze during a turn start no longer fails the turn or closes the session. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A follow-up run sent `turn.start` at 01:22:21. The harness then stopped the server, Runner, sidecar and Claude twice: 28 s, then 33 s after a 1 s thaw. At 01:23:25 the control plane reported `provider_transport_failed: PRP command turn.start timed out`. Three same-run resumes then failed with `ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and the task needed a manual Retry. The Retry run succeeded. **Expected behavior** Time that the processes spend stopped does not count against the turn-start bound or the sidecar request bound. A real timeout still fires while the processes run. **Steps to reproduce** 1. `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-stall.test.ts`. On master it fails with `provider_transport_failed: PRP command turn.start timed out`. 2. `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_sidecar_stopped_time`. On master it fails with `ACPX sidecar request timed out at initialize`. It stops the process early in the wait and again in its last half second. Note on the repeated `ACPX session is not open`: the wave ran `eafd5853d9`, which does not contain #20675. On master, #20675 settles a cut-off start from the sidecar's dispatch journal. If the prompt never left, the Runner reopens the session and sends it once. If the journal shows the prompt left, or cannot prove it, the session stays closed and the run ends with a working Retry, as the class rule requires. This pull request stops the freeze from cutting off the start in the first place. Refs #20704, #20675 ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `#startTurn` runs under `#withAdmissionClock`. The turn-start deadline, the held-tool-call wait, the start-event wait, the adoption check time and the `session.snapshot` wait all read `#waitNow()`, so they share one stopped-time credit. - `packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs`: `receive_response` blocks in 1 s slices. `credit_stopped_time` treats a slice overshoot beyond 2 s as a stop. It moves the deadline by the overshoot, and to at least 5 s after the thaw, so a stop in the last slice still leaves the sidecar time to answer. Total credit per request is capped at 15 min. - The control-plane clock (`RunnerStallLedger`) now uses the same rule: a stop is credited in full plus a 5 s grace, capped at 15 min per admission. - `packages/paperclip-runner/src/live/runnerd-codex-transport-stall.test.ts` (new): drives the real transport and Runner. It SIGSTOPs the Runner tree, jumps `Date.now` by 40 s mid-wait, then resumes the Runner. It fails if the turn start leaves the clock, or if the command wait stops reading it (checked by removing `clock: this.#admissionClock`). This is the transport-level test requested in the #20704 review. - `packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_stopped_time.rs` (new, its own test binary): a sidecar stops the test process for 6 s against a 3 s bound and answers after the thaw. - Unit test for `credit_stopped_time`. ## Verification - Red on master, green here: both commands in "Steps to reproduce". - `npx vitest run src/live/runnerd-codex-transport.test.ts src/live/runnerd-codex-transport-stall.test.ts src/live/runnerd-acpx-source-reconciliation.test.ts src/live/runnerd-codex-transport-adoption.test.ts`: 342/342 pass. - `cargo test -p paperclip-runner-core --lib acpx_sidecar_transport` (12 pass), `--test acpx_sidecar_transport --test acpx_park_preparation --test acpx_provider_turns` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: clean. - Not run: a new chaos wave. ## Risks - A real stall longer than 2 s in the Runner (or 5 s in the control plane) during a turn start or a sidecar request now extends that wait by the stall plus 5 s. The waited-on process could not be observed during the stall either. The 15 min cap bounds a process that keeps stalling. - The Rust wait now wakes once per second while it waits for a sidecar reply. This is an active request with no I/O per wake. - The 1 s control-plane ticker from #20704 now also runs during a turn start, and stops when the start returns. - No persisted state, launch identity or saved-session shape changes. Old saved runs resume through the same code, and an older Runner keeps its old wait. Nothing new runs before `turn.start`, and no wait is added on the normal path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
1fa7e92fa8 |
fix(native): restart in place a Runner that only settled an earlier attempt's suspend (#20724)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run talks to a Runner process. When the control plane loses
that Runner, the run resumes itself with a limited number of same-run
attempts.
> - A server freeze while the controller suspends a finished turn can
stop the Runner inside `runner.suspend`. The controller journal then
keeps that suspend as pending.
> - The next attempt starts a new Runner. It settles the old suspend
from the welcome, records `suspended`, and exits with code 0 before it
accepts any command of the new attempt. The server reported "runnerd
exited unexpectedly with code 0", so one freeze spent one same-run
attempt. If a second fault then occurs, no attempt is left in reserve.
> - #20714 made both sides agree on that suspend's acknowledgement. This
pull request handles the clean exit that follows it.
> - During startup, the server restarts that Runner in place once,
through the existing `#recoverRunnerProcess` path, and only when the
durable Runner journal proves the exit.
> - The benefit is that one freeze during a suspend costs zero same-run
attempts.
## Linked Issues or Issue Description
Refs #20714.
**What happened?**
A restart-chaos task (local Runner, Claude, a freeze of the server
during a live turn) needed a manual Retry. #20714 fixed the identity
mismatch. The second same-run attempt still ended with
`runner_local_connect_failed: runnerd exited unexpectedly with code 0`,
because that Runner only settled the suspend left by the first attempt
and then exited as suspended.
**Expected behavior**
The attempt continues. The server restarts the Runner once. The new
Runner starts from a clean `suspended` state and serves the attempt's
commands. No provider work is repeated.
**Steps to reproduce**
1. Start a native run. Let the controller send `runner.suspend` when it
closes the session.
2. Stop the Runner after it journaled the suspend and before the
controller records the result (a server freeze does this).
3. Resume the same run. The new Runner settles the old suspend and exits
with code 0, and startup fails.
The new live test reproduces this with the real runnerd and the fake
Codex provider.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
e039be9202 |
fix(runner): replace a follow-up session whose remote workspace root moved (#20712)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - `pnpm verify` (see the PR comments for the selected scope). - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
e719d42557 |
fix(runner): accept the acknowledgement of an indeterminate suspend replay (#20714)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run talks to a Runner process (runnerd). The controller in the server sends commands. The Runner journals each command before it acts and replies with a result. > - `runner.suspend` and `runner.shutdown` are terminal commands. The Runner waits for a `command_result_ack` before it exits. It compares that acknowledgement with the receipt that it journaled. > - A Runner that is killed inside `runner.suspend` journals the command but not its result. The next Runner replays the command as `indeterminate`. It sends it on the wire as `failed` with `execution_indeterminate`, and it still treats it as terminal. > - The controller acknowledged the `failed` status that it received. The Runner compared that with its journal status `indeterminate` and exited with "terminal command result acknowledgement changed its durable identity". The controller also lifted its terminal fence for a failed suspend and sent the next command to a Runner that was about to exit. > - This pull request makes both sides agree on the status of that one receipt, and keeps the fence. > - The benefit is that a freeze or restart during a suspend no longer burns the same-run retries and leaves the task at a manual Retry. ## Linked Issues or Issue Description **What happened?** A restart-chaos task (local Runner, Claude, a 28-second freeze of the server during a live turn) needed a manual Retry. Retry worked. 1. The turn completed. The server then sent `runner.suspend`. The server froze before the Runner reported the result. 2. The close found the Runner not durably suspended (`commandStatus: pending`, `runnerLifecycle: ready`). The run failed with `native_runner_checkpoint_owner_lost`, and same-run recovery stopped the old runnerd. The suspend was journaled but not finished. 3. Attempt 2 started a new runnerd on the same state. The welcome delivered the stale suspend. The Runner replayed it as `indeterminate` and sent `failed`. The controller acknowledged `failed`. The Runner exited with code 1: "terminal command result acknowledgement changed its durable identity". The controller had recorded the suspend as `failed`. 4. Attempt 3 found the fence for a suspend that the controller no longer listed as pending. The Runner reconciled it and exited with code 0. The server reported "runnerd exited unexpectedly with code 0", and `native_session_retry_exhausted` stopped automatic recovery. **Expected behavior** The controller acknowledges exactly the receipt that the Runner sent. The Runner accepts that acknowledgement, records `suspended`, and the next attempt starts from a clean suspended Runner. **Steps to reproduce** 1. Start a native run with a local Runner. Let the turn complete so that the server sends `runner.suspend`. 2. Freeze the server (SIGSTOP) for about 30 seconds before the suspend result arrives, so that recovery stops the Runner inside the suspend. 3. Let same-run recovery start a new Runner. It exits with "terminal command result acknowledgement changed its durable identity". The unit tests in this PR reproduce the same sequence without a live run. **Paperclip version or commit** `master` at |
||
|
DottaandClaude
|
6d1eae73c0 |
fix(runner): do not count time the control plane was stopped against a Runner command wait (#20704)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime resumes a run after a control-plane restart. It launches a Runner and waits for recovery barriers (`run.attach`, a `runner.drain` probe, the provider identity). > - These waits poll with a wall-clock deadline. When the control plane and its Runner are stopped together (SIGSTOP, host sleep), the wall clock passes the deadline before either process can act. > - On thaw, the wait reports a timeout although nothing was slow. Recovery then spends an attempt and a retry delay. > - This pull request gives each startup or recovery admission a clock that does not count stopped time. > - The benefit is that a freeze or host sleep during an admission no longer turns into a false `PRP command ... timed out` failure. ## Linked Issues or Issue Description **What happened?** A restart-chaos wave (local, codex, fault TM2) stopped the server and its Runner tree for 44 s, 1 s after a resumed Runner started. The resume barrier `runner.drain` had a 30 s deadline. On thaw, the server reported `runner_local_connect_failed: PRP command runner.drain timed out` 45 s after the wait began (`provider.session.load` span 45130 ms). Recovery spent attempt 1 and waited 30 s more before attempt 2. **Expected behavior** Time that the waiting process spends stopped does not count as waiting time. A real timeout still fires when the Runner does not answer while both processes run. **Steps to reproduce** 1. Start a Runner command wait with a 30 s deadline. 2. After 1 s, stop the process for 44 s (the test moves the system clock without running timers). 3. Let the Runner complete the command 2 s after thaw. 4. Before this change, the wait rejects with `runner.drain timed out`. After this change, it resolves. Note: in that chaos task, this false timeout was not the reason that the task needed a manual Retry. An earlier SIGKILL of the whole server tree stopped the Codex turn while a foreground shell command (`./bench.sh`) was running. On restore, Codex had no active turn (`provider_turn_lost_on_restore`). Recovery correctly blocked automatic replay (`uncertain_provider_action`), because no durable record proves that the interrupted command had no side effect. The manual Retry then completed the task. This PR does not change that policy. Refs #20675 ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: add `RunnerStallLedger`, the clock of one startup or recovery admission. `now()` is wall time minus the stopped time observed since the admission began. Its poll loops and a 1 s ticker share one last-observation time, so each stop is credited once, wherever the process was when it stopped (a poll loop, a checkpoint flush, a spawn). Only the part of a gap beyond 5 s is credited. - `#start` and `#resume` run under one ledger (`#withAdmissionClock`). A nested startup keeps the outer ledger. The ticker stops when the admission ends. - During an admission, `waitForRunnerCommand`, the provider-identity wait, the cold admission deadline, the recovery admission deadline, and the default 30 s command deadline all use this clock. - Waits after admission, including `turn.start` and its event and snapshot waits, keep wall-clock deadlines. Their behavior does not change. - Tests in `runnerd-codex-transport.test.ts`: the chaos sequence (red before, green after), the unchanged wall-clock timeout outside an admission, a barrier that still times out after a freeze, a 70 s freeze between two barriers on one cold deadline, and a slow-but-running process that still reaches its deadline. ## Verification - Red: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "stopped"` failed before the fix with `runner_local_connect_failed: PRP command runner.drain timed out`. - Green: the same command passes after the fix. - `npx vitest run src/live/runnerd-codex-transport.test.ts`: 301/301 pass, including the existing "shares Pi's absolute cold deadline across recovery barriers and rejects a late ACK" test. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: clean. - `pnpm verify` exits 2 (changes outside resolved test graphs). Required CI supplies the broad coverage. - Not run: a new chaos wave. The unit tests reproduce the exact wait and clock sequence from the artifacts. ## Risks - A real event-loop hang longer than 5 s during an admission now extends that admission by the hang minus 5 s. The Runner answer could not be observed during the hang either. Each credited gap still costs 5 s of budget, so a process that is slow on every observation still reaches its deadline. - The ticker runs only while a startup or recovery admission runs (an active operation), every 1 s, does no I/O, and is cleared in `finally`. It does not keep an idle process awake. - A `turn.start` that is cut by a freeze still reports a wall-clock timeout, as before. That case needs its own change, because its event and snapshot waits share one deadline. - No persisted state, launch identity, or saved-session shape changes. Old saved runs resume through the same code. A deploy mid-run only changes how long an in-process admission wait tolerates a stop. No new wait or file operation runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
3fdcc2a28c |
fix(runner): resend an ACPX turn whose start was cut off before the prompt left (#20675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) sends `turn.start` to its Runner. The Runner asks the ACPX sidecar to start the turn, and the sidecar sends the prompt to the provider > - A start can fail without a definite answer. A host freeze can outlast the sidecar request deadline, or the sidecar or Runner can die during the start > - The Runner then saved the session `closed`. Every same-run resume failed with `ACPX session is not open`, and the task needed Retry > - The Runner cannot tell from its own state whether the provider has the prompt. Resending could duplicate it. A continuation could lose it > - This pull request adds a durable record that answers that question: a sidecar prompt-dispatch journal, written before the prompt reaches the provider > - The benefit is that a start cut off before the prompt left resends the prompt exactly once, with no Retry. Every case the record cannot prove keeps today's behavior ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude, workload C1, fault TM2). A follow-up run reopened the parked Claude session and sent `turn.start`. The sidecar first starts Claude and loads the conversation; the prompt goes out only after that. After 24 s without a reply, the harness froze the server, Runner, sidecar and Claude (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar deadline had passed. It reaped the sidecar and saved the session `closed`. Three same-run resumes failed with `PRP command turn.start failed: … ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and a person had to click Retry. **Expected behavior** When a durable record proves the prompt never reached the provider, the run resends it once by itself. When the record cannot prove it, the run behaves as today. **Steps to reproduce** 1. On master, run `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_park_preparation cut_turn_start`. 2. `a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once` fails: the session is `closed` and the next start fails with `ACPX session is not open`. Refs #20507, #20645, #20674 ## What Changed - New file contract in `protocol/provider-schemas/acpx-sidecar.schema.json`: `turnDispatchJournalFile` (`paperclip-turn-dispatch.jsonl`) and `turnDispatchJournalRecord` (`received` / `dispatched`). The generator emits the file name and schema into TypeScript and Rust, so the pack contract digest covers them. - Sidecar (`turn-dispatch-journal.ts`, `codex-runtime-adapter.ts`, `runtime-host.ts`, `acpx-runtime-sidecar.ts`): when the runtime port starts a turn, before any provider work, it truncates the journal and writes `received`. ACPX calls `onAcpMessage("outbound")` synchronously before it writes a message to the provider. On the turn's first `session/prompt` request the port appends `dispatched` and fsyncs it there, so the record exists before the bytes leave. A failed dispatch write aborts the prompt write. A failed `received` write removes the file and runs the turn unjournaled. - Runner (`acpx_turn_dispatch.rs`, `acpx_provider_backend.rs`): the journal is read only after the provider lifetime fence proves that the sidecar and provider are gone. A lone complete `received` record for the turn proves the prompt never left. - A start that failed in flight saves the session `prepared` with no active turn and returns `ACPX turn start failed before its prompt reached the provider`. The run's next `turn.start` reopens the conversation and sends the prompt once. Until the proof settles, the saved state stays `turn_starting`, so a Runner killed during the proof goes through the restore check below. - A Runner killed while the session was `turn_starting` restores the session idle (`session.resumed`, no `turn.failed`). The same-run resume then starts the turn once instead of sending a continuation that would lose the prompt. - A `dispatched` record, another turn's record, a torn or malformed file, or no file keeps today's behavior (closed session, or lost-turn recovery). - Fake sidecar: writes the same journal and adds `hang-before-dispatch` / `hang-after-dispatch` faults. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` ("ACPX turn starts cut off before the prompt left"). The design note is `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md` (in #20674). ## Verification - Red on master, green here: `cargo test -p paperclip-runner-core --test acpx_park_preparation cut_turn_start` (4 tests: cut before dispatch reopens and sends once; cut after dispatch stays closed and never resends; Runner killed before dispatch restores idle; Runner killed after dispatch or with no journal reports the lost turn). The two "before" cases fail on master. - `cargo test -p paperclip-runner-core`: all pass. `cargo fmt --check` clean. - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx/turn-dispatch-journal.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts`: 121 passed. One test uses a real ACPX runtime and an ACP peer that refuses a prompt arriving before its `dispatched` record; it fails if the hook is removed. - `node scripts/generate-acpx-sidecar-contract.mjs --check`, `node --test scripts/contracts/contract-fingerprints.test.mjs`, and `tsc -p tsconfig.test.json` pass. - Time to first token, measured with the same open/write/fsync sequence on this Mac (APFS, load average ~11, 300 samples): `received` p50 0.24 ms, p95 0.59 ms; `dispatched` (with fsync) p50 4.7 ms, p95 11.5 ms. Against a Claude TTFT budget of 9.5–16.5 s p50 this is under 0.1%. No wait, round trip or file transfer is added. Recovery work (fence, journal read) runs only after a failed start. - Live restart-chaos wave and runner image: see the comments on this pull request. ## Risks - A false "not dispatched" proof would duplicate a prompt. It needs a lone, complete `received` record for this exact turn ID, read after the fence proves the sidecar is gone. `dispatched` is fsynced before the prompt write, and a failed write aborts the send. A failed `received` write removes the file, so it cannot leave a lone record. - Upgrade safety: saved provider state gains no field (its decoder denies unknown fields), and only lifecycles an older Runner already handles are written. An older Runner ignores the journal file. State saved by an older Runner has no journal, which reads as unknown and keeps today's behavior. A new Runner with an older sidecar never sees a `received` record, so no proof forms. - The changed sidecar schema changes the pack contract digest, so Daytona needs a new runner image. Older image packs are not reused. - A failed start now waits for the provider lifetime fence (at most 5 s, usually immediate) before it settles. This is only on a failure path. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
e33ede9a8a |
fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nicky Leach
|
0d4b1d8c89 | Gate agent plugin tabs by execution environment (#20657) |