mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
d634944df16462a8979b6f722ff9cf45457d55ea
4061
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d634944df1 |
Guarantee agent-error candidates in retention fixtures
Random UUID ordering could put every error agent into kept or archived retention rows, while the pagination test required an eligible candidate of every kind. Put the retained errors in deterministic positions and seed a separate unretained error after assigning retention. The controlled ordering reproduces the missing-kind failure before the eligible row is added. All ten retention-candidate tests pass afterward. Pagination, filtering, and the independent query oracle stay unchanged. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
d377c1dae4 |
Preserve SSH account defaults, command paths, and setup retries
Default new API-created SSH environments to the selected agent account. Keep absent authentication fields unchanged on reads and updates so older SSH environments retain their remote login. Launch SSH ACP with a non-login shell to retain the prepared remote PATH and private HOME. On a failed Codex host runtime install, remove only newly created links that still point to this install's staged skills. Preserve manual skills and native authentication so a corrected retry can start. Four regression cases failed first; all seven targeted cases now pass. The broader eight-suite run passes 529 cases. Server, adapter-utils, and Codex typechecks, SDK input checks, and secret/PII scans pass. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
4c06df094c |
Merge current CI entry points from master
The trusted typecheck workflow now invokes ci-typecheck-lane.mjs. The stacked branches lacked that script and could not start qualification. Refresh the SSH base from master so its CI source matches the workflow, then propagate the base to the Tailscale connector branch. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
89e6575418 |
test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs get their tools from a session tool authority, which holds the run's assigned app (MCP) tool catalog > - #21415 releases a finished session's authority, so the server heap no longer grows with each finished session > - Two behaviors of that fix had no test: a released session fails closed, and the release when a follow-up run retires its warm predecessor and then fails before it admits its own authority > - The memory saving was measured once, by hand, with no script in the repository > - This pull request adds the two tests and a benchmark script. It changes no product code > - The benefit is that a later change cannot silently bring back the leak or let a released session reach its tools ## Linked Issues or Issue Description Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate; a reviewer comment there named the follow-up failure path). **What is missing?** #21415 tests that finished sessions return the authority map to its starting size. No test checks that a released session refuses late tool calls. No test checks the follow-up path, where a follow-up run retires the previous warm session (so the release at forget time is skipped while the follow-up owns the scope) and then fails before it admits its own authority. **Expected behavior** A late tool call or completion feedback for a released session fails with `native_session_tool_authority_unavailable`. The next run of the same scope gets a new authority with only its own assignments. When a follow-up fails after it retires its predecessor, the authority map is back at its starting size after that run settles. **Paperclip version or commit** master `edb6945ff3` (after #21415, `85f4a91e70`). ## What Changed - Test "refuses a released session's late tool call and completion feedback and admits the next run with only its own assignments". - Test "releases a retired warm predecessor's tool authority when the follow-up fails before admitting its own". A follow-up of the same warm scope with changed configuration retires the predecessor. Then its assigned tool listing fails. The test asserts that the predecessor was closed, that the follow-up failed with that error, and that `sessionToolAuthorityCount()` is back at its starting value. - `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained bytes of one assigned tool catalog per session, and schema copies per `definitions()` call. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "released native session tool authority|retired warm predecessor"`: 2 passed. - Red check: with the release at settle removed from `native-session-executor.ts`, both tests fail (`expected 1 to be +0`, and the late tool call is not refused). - Retained bytes per finished per-turn session were measured through `executePaperclipNativeSession` with the real `createAssignedMcpTools` (80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On `85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794 bytes (test-harness noise), and the authority map stayed at its starting size. - `cd server && node --expose-gc --import tsx scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes 2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per assigned tool catalog. Each `definitions()` call makes 80 new schema copies (about 296 KB), and none are kept. - Full `native-session-executor.test.ts` locally, with a load average near 78 from other work on the machine: 1,483 passed and 5 failed. All 5 failures were timeouts in tests this change does not touch (the slowest took 163 s). CI gives the clean full-suite result. ## Risks - Low risk. Tests and one benchmark script only. No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df1b12ad8e |
fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop
## Linked Issues or Issue Description
**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.
**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.
**Paperclip version or commit**
`master` at
|
||
|
|
a38694571e |
Preserve native provider compatibility and clean partial SSH uploads
Apply preconfigured SSH homes only to supported native providers and permission modes. Preserve isolated mode for OpenCode, Copilot, and restricted Claude while retaining admitted recovery input. Stage host-auth Codex runtime files in private SSH directories so existing cleanup owns partial uploads. Read the selected remote profile before creating its tracing overlay. Eight new regressions first failed. Verified 21 focused cases and 229 tests across six broader adapter, runtime, configuration, and provider suites. Adapter-utils, Claude, Codex, and server typechecks pass. SDK source, whitespace, and secret scans pass. Fresh CI and Apex review remain required. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
84a4d345b3 |
refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by the `runner-artifacts` stages of the `Dockerfile` > - The application image does not use those stages. It has five stages of its own that compile the daemon, it builds the Runner again inside the server build, and it builds a second provider pack > - So one `Dockerfile` holds two recipes for the Runner, and the two results differ: another base image and another daemon binary > - This pull request removes the second recipe. The application image copies the Runner out of the `runner-artifacts` stage, which the same build runs once > - The benefit is one Runner recipe, and an application image whose Runner is the Runner the contract describes. A later pull request can replace the stage with the published image ## Linked Issues or Issue Description This is the second part of a chain. The first part is #21331, which removes the cache check that cannot run on this `Dockerfile`. Refs #21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode of the vendor step), #21307 (the pull-request check of the application image) and #21312. No issue exists for this change. **Problem or motivation** The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps` and `runner-build` compile the daemon on Debian trixie. The server build then runs the full build of the Runner package. The `runner-provider-pack` stage builds the provider pack. The `runner-artifacts` stages do all of that again for the Runner image, on Debian bookworm, with a stripped daemon. **Proposed solution** Read only the `runner-artifacts` stage from the application stages. Build it inline in every build. Do not pass a published image yet. **Alternatives considered** - Pass the published Runner image to the build in this pull request. That needs the resolver, labels and a rule for commits without an image. It is the next step, and this one must be safe alone. - Keep the old stages as a fallback. Two recipes is the problem. ## What Changed - `Dockerfile`: remove the five stages. `build` starts from `base`. It copies `/runner-package/dist` and the build stamp from `runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and runs the vendor check before the first application build. - `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN` with a bind mount of `runner-artifacts` copies the pack, the receipt and the archives, and moves the large files to their own tree. `build` and `runner-provider-pack` now come after the stages they read. - `Dockerfile`: the three `COPY` instructions of `production` that take the pack are byte-identical. A test pins all three. - `Dockerfile`: `build` declares the build argument `PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step runs the new `scripts/check-runner-image-expectation.mjs` on `/runner-image.json` of `runner-artifacts`. **This pull request declares the argument. No build sets it. The next pull request, which gives a published Runner image to the build, is its first user.** With the argument set, the build fails unless `runner-artifacts` is the published image of exactly those inputs. `docker-app-check.yml` also runs when the script changes. - Comments: the `Dockerfile` has 220 comment lines, down from 247. The comments of the two moved stages moved with them, unchanged, so the diff shows them as added lines. Eight comments that named a removed stage or a step that moved get the smallest correction. Two comments in `prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the `Dockerfile` relies on the standalone rebuild; that clause is removed. - Tests: replace the two cargo-chef tests of `docker-build-stamp.test.ts` with three tests of the new shape. Update the pack test, `docker-image-layer-reuse.test.ts`, `vite-build-commit.test.ts`, the stage rules of `runner-image-identity.test.ts`, and the include-layout test of `runner-release-crate-cache.test.mjs`, which now reads `runner-artifacts-runnerd`. - Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The application image" in `doc/runner-image-contract.md`. ## Verification All builds: `linux/arm64`, local, `--build-arg PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The builds ran before the rebase onto `master`, on the head of #21310 as it was before it merged. `master` changed neither the `Dockerfile` nor an input of the Runner in between. - `docker buildx build --target production`, cold, before and after: 506 s and 410 s. Compressed size: 2,971 MB and 2,900 MB. - The image config and the text of all 33 history rows are identical. The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ # buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly once. - A comparison of every file of the two images (path, type, size, mode, link target, and SHA-256 under `/app` and `/opt`). The differences are in "Risks". All other files are byte-identical, the root `node_modules` included. - From the image after: the four `/app/packages/paperclip-runner/...` paths and the contents of `/opt/paperclip-runner/` exist. `paperclip-runnerd --build-metadata` runs. `provider-pack.json` has `runnerSourceRevision` equal to the build commit and no `runnerInputsDigest`. The entry points `index.js`, `live/index.js` and `testing.js` of the Runner import. - The vendor step logged `reusing the current Runner build` in both of its runs. Every `Compiling` line of the log is in `runner-artifacts-runnerd`. - `--target build`, `--target runner-provider-pack` and `--target cloud` built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or `pkg-config`. - `--target runner-artifacts` and `runner-artifacts-slim`, before and after: `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon, `provider-pack.json` and the two stable layers have the same digests before and after. - Without a build commit: `build` and `runner-provider-pack` built, and the pack was skipped with the same message. This run was on the next branch of this chain, after the rebase. The last steps of `production` were not run for this case. - The expectation argument, on this `Dockerfile`, `--target build --output type=cacheonly`, after the rebase: - No argument, inline Runner: the build passes (233 s). The step passes. - Argument, inline Runner: the build fails at the step. The message says that the `runner-artifacts` of the build are not a published Runner image. - Argument `d4df566f…` and the published image of those inputs as `--build-context runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`: the step passes, and the build runs no Runner stage. The vendor step then fails with `inputs changed since the recorded build`, because this tree differs from the tree of that image by one comment line in `build-inputs.mjs`. With that line put back, the same build completes, and the vendor step reuses the published build. - Another digest with the same context: the build fails at the step. The message names the inputs the image has. - A tree whose lockfile is behind the manifests of the Runner fails at the frozen install of the Runner stages, after 169 s, with the message that stage already had. - `docker buildx build --check`: no warning, before and after. `scripts/check-docker-deps-stage.mjs` and `scripts/check-node-version-policy.mjs` pass. - Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the changed server test files with the other tests that read the `Dockerfile` or the image workflows (231 pass in 10 files on Linux), `vite-build-commit.test.ts` (10 pass), `runner-release-crate-cache.test.mjs` (11 pass), `docker-deps-postinstall.test.mjs`. The changed server test files typecheck. - Not run: the full test suite, any `linux/amd64` build, `podman`, the workflows and the release pipeline. Only CI can prove amd64, `docker.yml`, `docker-merge-group.yml`, `release.yml`, `docker-app-check.yml` and `runner-image.yml`. ## Risks - **The digest of the Runner image changes.** Its identity covers the whole `Dockerfile`. The next `master` push publishes a new `inputs-<digest>`. The contents do not change: see the comparison above. - **A build that is given a Runner of other inputs fails; it does not ship that Runner.** With the argument, it fails at the expectation step. Without the argument, it fails at the vendor step when the build inputs of the Runner differ. One comment line in `build-inputs.mjs` changes those build inputs in this pull request, so the image published before this pull request cannot be vendored into a tree after it. - **What a consumer of the application image sees** (measured on `linux/arm64`): - The daemon is the stripped binary compiled on Debian bookworm: 23.2 MB, was 36.4 MB, at `/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the vendored copy in `server/dist`. Its hash changes. It has no symbols for a crash backtrace. - `/app/packages/paperclip-eval-kernel/dist` is absent. Only `dist/eval/workflow-harness.js` of the Runner imports that package. Only `eval/index.js` exports the harness again, and no module imports that file. The server runtime bundle does not name the package. - The provider pack: `provider-pack.json` is byte-identical. Two provider executables and one licence file are hard links to their other copy in the pack, where they were separate files. One empty directory is absent. The pack is 195 MB smaller, and the layer of its large files gets a new digest once. - The transport archives have other bytes, because the Runner build gives large files a fixed timestamp. The receipt names the same files with the same digests. - Paths, modes, owners, the image config and the layer order do not change. - **What a contributor and a maintainer of the image build see:** - **The image build does not run the drift checks of the Runner.** Before, the server build ran the full `build` script of the Runner, so a failed drift check failed the image build on `master`. Now the Runner stages run only the output steps of that script. Repository CI runs the checks on every pull request. `.github/docker-context-checks.Dockerfile` and three exceptions in `.dockerignore` exist for those checks in the image build; a separate follow-up removes them, and this chain does not edit them. - **The dependency-caching layer of the daemon is gone.** Before, cargo-chef compiled the Rust dependencies in a layer of their own, and a change to the Rust source compiled only the Runner's own crates. Now such a change compiles the dependencies again (146 s here for the whole daemon). A change to the server, the UI or the Runner's TypeScript still reuses the compiled daemon. - **A lockfile behind the manifests of the Runner fails the build.** Before, `docker build` resolved it inside the image. The image workflows refresh the lockfile first. A local build prints the command to run. The next pull request makes this failure early and clear. - **The build needs more disk**: about 25 GB of builder storage here, was about 20 GB, with the image export. - Rollback: revert this pull request. Images that exist do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
1865127a38 |
Preserve SSH runtime delivery and native authentication choice
Use portable SSH decoding. Deliver Codex tools and skills separately from native remote credentials, and preserve subscription refreshes for selected accounts when controller API keys exist. Carry prepared home mode into native selection and probe the remote login without staging controller credentials. Share portable remote home discovery between execution and probes. Verified 449 adapter and shared runtime tests, nine focused native admission/probe/recovery regressions, and nineteen real Runner probe tests after building the local daemon. Adapter-utils, both provider adapters, and server types pass. Six broad selection tests hit a pre-existing bundled-skill path failure, reproduced on the unchanged selection source; final CI remains required. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
426e8a35dc |
perf(runs): read live runs in a fixed number of statements (#21417)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board shows the runs that are live now. The sidebar and the active agents panel read `GET /api/companies/:companyId/live-runs` > - That route checked read access and built the output-silence summary one run at a time, with 8 statements per run > - On a hosted instance this route ran about 6 times per minute with an average of 2.1 s of server time, and its statements and allocations add load to a busy main thread > - This pull request reads the access decisions and the silence state for all runs of the request in one batch > - The benefit is a fixed number of statements per request (15 instead of 8 + 8 per run) and about one quarter of the allocation ## Linked Issues or Issue Description No public issue exists. This is an enhancement. **What existing behavior does this improve?** The live-runs read on the board (`GET /api/companies/:companyId/live-runs`). **Subsystem affected** Server: agent routes, run read authorization, active-run watchdog. **Current behavior** For each run, the route calls `canActorReadHeartbeatRun` (task existence, instance-admin flag, membership, run policy, task privacy) and `buildRunOutputSilence` (two watchdog decision reads and one evaluation-task read). The instance-admin and membership reads repeat the same answer for every run. Measured locally with a statement and allocation harness: | Live runs | Statements per request | Allocated per request | | --- | --- | --- | | 10 | 88 | 10.0 MB | | 50 | 408 | 27.9 MB | | 200 (route returns 50) | 408 | 28.6 MB | **Proposed behavior** The route reads every access decision in one batch and every silence summary in one batch, so the statement count does not grow with the number of runs. **Reason and benefit** Fewer statements and less allocation on a route that the board calls often. **Breaking changes** None. The response is the same. ## What Changed - `server/src/routes/agents.ts`: the live-runs handler decides reads with the existing `getHeartbeatRunReadDecisions` (the run history list already uses it) and builds silence summaries with one `buildRunOutputSilences` call for the readable runs. Unreadable runs are still redacted with `redactHeartbeatRunListRow`, and no silence state is read for them. - Active-run watchdog Postgres adapter: new `findLatestDecisions` and `findOpenStaleRunEvaluations` read all runs of one company in one statement each (`DISTINCT ON`). The single-run `findLatestDecision` and `findOpenStaleRunEvaluation` now call the batch with one run, so both paths use the same query. - Watchdog application: new `createBuildRunOutputSilences` use case groups runs by company. `createBuildRunOutputSilence` now calls it with one run, so the single and batch summaries use one rule. - `recoveryService` and `heartbeatService` expose `buildRunOutputSilences` as pass-throughs. - Tests: new `live-runs-batched-reads.test.ts`; updated watchdog use-case tests and the live-run route unit test. ## Verification - `cd server && npx vitest run src/__tests__/live-runs-batched-reads.test.ts` (embedded Postgres): - The batched read decision equals one `canActorReadHeartbeatRun` call per run for 24 actor kinds (owner, admin, operator, viewer, member, board key, cloud tenant, instance admin, local implicit board, board without membership, agent keys, agent runs on behalf of users, low-trust agents, agent of another company, unauthenticated). The runs mix open tasks, private tasks readable by grant, creation or assignment, private tasks nobody may read, private-project tasks, a hidden task, company-scoped runs, a run whose task was deleted, runs of another company, and runs of another company asked for as this company's. - For every actor that passes the company gate, each live-runs row is either the full row or exactly `redactHeartbeatRunListRow` of it (field by field), as one check per run decides. - Output silence per run equals the single-run reader for: no output yet, active snooze plus a later continue, expired snooze, dismissed false positive, recent output, open evaluation task, done evaluation task, queued run, an evaluation task of another company, and a run that has no rows at all. - The statement count is the same with 19 and 49 live runs for 6 actor kinds. Before the fix this test fails (for example 347 against 107 statements). - `npx vitest run src/modules/active-run-watchdog src/__tests__/heartbeat-active-run-output-watchdog.test.ts src/__tests__/agent-live-run-routes.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`: pass. - `npx tsc --noEmit -p server/tsconfig.json` and a test-tree typecheck of the changed tests: pass. `pnpm check:module-boundaries`: pass. - After the fix, measured with the same harness (embedded Postgres, 400 tasks, 40 agents, 5 requests each): | Live runs | Statements per request | Allocated per request | | --- | --- | --- | | 10 | 15 | 6.4 MB | | 50 | 15 | 7.3 MB | | 200 (route returns 50) | 15 | 7.8 MB | ## Risks - This changes an authorization read path. The route now uses the batch helper that the run history list and the live events socket already use. The test compares it with the per-run check for every actor kind in the visibility fixture. - If several open evaluation tasks point at the same run, the batch picks one of them. The single-run read also picked an unordered one before. - No schema, migration, locking or transaction change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e8a0316483 |
Refresh the SSH account branch from master
Keep the SSH base current so the stacked Tailscale migrations can follow the latest upstream schema sequence. The merge preserves managed SSH account selection, host-auth compatibility, and the PATH-only ACP prerequisite check. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
85f4a91e70 |
fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.
## Linked Issues or Issue Description
**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).
**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.
**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.
**Paperclip version or commit**
`master` (and the hosted build that crashed).
## What Changed
- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
- after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.
## Verification
Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.
Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.
Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:
| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |
Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.
Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`
## Risks
- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
24302a09ad |
fix(environments): stop warm sandboxes once their warm window ends (#21410)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes (for example Daytona) through reusable environment leases > - A warm run ends with its lease `retained` and its sandbox still running, so the next turn on the task starts warm > - After the warm window, only the provider's own idle auto-stop stopped that sandbox. Paperclip never stopped it > - On staging, idle Daytona sandboxes get provider activity about every 10 minutes. With a 12 minute auto-stop, they never stop and keep billing > - This pull request makes Paperclip stop a warm sandbox itself when its warm window ends and no live run holds it > - The benefit is that idle sandboxes stop on time, while warm turns inside the window stay warm ## Linked Issues or Issue Description **What happened?** Daytona sandboxes for ended warm runs stay in state `started` long after their warm window. On 2026-11-13 the staging Daytona org had 111 started sandboxes. One group of 23 idle sandboxes (auto-stop 12 min) got new provider activity exactly 10 minutes after their last use, so Daytona never auto-stopped them. **Expected behavior** Paperclip stops a sandbox when its warm window ends and no live run holds it. Daytona auto-stop stays only a backstop. **Steps to reproduce** 1. Run a warm `paperclip_runner` task on a reusable sandbox environment. 2. Let the run succeed. The lease becomes `retained` and the sandbox keeps running. 3. Wait past the warm window. Nothing in Paperclip stops the sandbox; it runs until the provider auto-stop fires, or forever if activity resets the provider idle clock. **Deployment mode** Hosted (Paperclip Cloud staging) with the Daytona sandbox provider plugin. **Additional context** Plugin Runner task leases (`driver: "plugin"`) are out of scope: their release path does not use the stop-only contract, so this change leaves them to their own cleanup. ## What Changed - New `server/src/services/warm-sandbox-expiry.ts` owns warm sandbox expiry: - The warm window is the run's resolved runner idle timeout (`resolveWarmRunnerIdleTimeoutMsForRun`). The window starts at the lease release (`lastUsedAt`). - Plugin Runner task leases (`driver: "plugin"`) are excluded. Their cleanup releases through the plugin, not the verified stop-only contract, so expiry could free the task's files. - A lease saved before this change has no recorded window. It uses the longest window a run can configure (24h), so it is never stopped early. - A candidate is a `retained` `reuse_by_environment` lease whose run has ended. It is held (never stopped, never a deadline) when another lease holds the same sandbox, a run on the task is `running` (it may be resuming this sandbox), or an operation is in progress on the lease (stop intent, export resume, restore recovery, cleanup claim, successor/supersede markers). - Due sandboxes go through the existing stop-only contract (`prepareSupersededWarmSandboxStop`). The cleanup sweep stops the exact sandbox at the provider. The lease then ends `released` as stopped and kept, which the next run on the task resumes. - Acquisition (plugin and built-in sandbox paths) records the run's warm window on the lease (`warmSandboxIdleTimeoutMs`). - `nextEnvironmentCleanupAt` includes the earliest warm window end. The environment cleanup worker sleeps until that deadline. It also wakes on every lease release and runs at boot, so a sandbox left running across a restart or deploy is reconciled. There is no interval scan. - `recoveredOrphanFirstAttemptSql` treats a warm-window stop as due at once, so the first stop does not wait for the 5 minute retry backoff. - The pending-cleanup sweep hands due warm sandboxes to the stop-only cleanup before it reads its rows, so one pass stops them. - Only this instance's own lease rows are judged. Nothing lists provider resources, so a sandbox of another company or environment is never touched. ## Verification - `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.test.ts` (server): 9 passed. - Inside the warm window: kept, and the worker deadline is exactly the window end. - Past the window: stop intent recorded, first stop due now. - Through the cleanup sweep: the provider stop is dispatched once; the lease ends `released`, stopped and kept. - A live run's lease on the same sandbox: kept, and no deadline. - A run on the task is starting: kept. - The lease's own run is still running: kept. - A plugin Runner task lease past its window: left to its own cleanup, and no deadline. - Orphan after a restart: a lease retained 35h ago without a recorded window is stopped on the first (boot) pass; one retained 1h ago is kept with a deadline at 24h. - Red/green: with the wiring removed (sweep step, deadline, first-attempt marker), 4 of the 8 tests fail (window deadline, past-window stop, sweep stop, restart reconcile). With the change, all 8 pass. - CI on this PR covers the broad suites for the final revision. A local `pnpm verify` selected a very wide set (the change touches `heartbeat/recovery.ts`) and hit 30 s timeouts in unrelated suites under machine load, so I stopped it and rely on CI. - After deploy, I check the staging Daytona org: started counts for ended warm runs drop, and live or in-window sandboxes stay started. ## Risks - A run that starts on the task in the moment between the deadline check and the stop could resume a sandbox that is being stopped. A `running` run on the task holds the sandbox, and the stop intent is a compare-and-swap on the exact retained row, so this window is small. The same race already exists for the retained-orphan settle. - The first boot after deploy stops every retained warm sandbox that is older than 24h (or past its recorded window). This is the intended cleanup. It makes one stop call per lease, 25 leases per pass. - The stop-only contract disables Daytona auto-delete on the stopped sandbox (existing behavior of that contract). Daytona auto-archive still moves it to cold storage. The retained-orphan settle destroys it when the task no longer uses it. - The deadline query reads `retained` leases without a dedicated index, the same as the existing pending-cleanup deadline. It runs only when the cleanup worker wakes. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Daytona API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e7e28ba241 |
feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns` tasks at once. Other runs wait in the agent's queue with status `queued` > - On hosted staging, one agent had all 20 slots full and 295 queued runs. Some tasks waited 3 to 7 hours and never started > - The UI showed these tasks as live or working: a pulsing "Live" badge in the task header and in task rows, a spinning "Working … for 4 hours" in the task chat, and "N live" counts that included every queued run > - People could not tell that work was waiting, why it waited, or what to change > - This pull request makes a deep queue an honest, visible state: "Waiting for Codie · 20 slots max · 41 of 295 in line · Configure" > - The benefit is that operators see the real state and the one setting that changes it ## Linked Issues or Issue Description Refs #21391, #21349, #21242 **What happened?** A queued run that never started looked the same as a running run. The task header, the task rows in lists and the inbox, the task chat and the sidebar counts all showed queued runs as live or working. **Expected behavior** A run that waits for one of its agent's concurrency slots shows that it waits. It names the agent, the agent's limit and the run's place in line, and it links to where the limit is set. The place in line is the order in which the server starts the agent's queued runs. **Why does it matter?** With a deep queue, the UI said "working" for hours while nothing ran. Operators could not see the cause or the fix. ## What Changed - Server: a new module `server/src/services/heartbeat/queued-run-line.ts` now holds the facts that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The dispatch path in `queue.ts` and the new "place in line" read use this one builder, so the line the UI shows is the dispatch order. The logic moved without change, including the recovery-share fix from #21391. - Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit, running count, position, queued count) and the label "Waiting for <agent>" with cause `agent_at_capacity` to a queued, never-started run when its agent has no free slot. It reads one batch of agents and running counts. Then it builds one line per full agent (at most 8 agents per read), using only the three context keys that ordering needs. No new polling. - Server: the place in line uses `queuedRunDispatchLine`, an O(n log n) replay of the dispatcher. The plain replay rescans the queue after each step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000, 600 ms at 10,000, all on the event loop). Dispatch itself is unchanged: it stops after the free slots. Each agent's line is memoized for 5 s and shared by concurrent reads. In a preview worktree, the line applies the same creation-time cutoff as dispatch. - Server: the task list summary adds `runningDescendantCount` (descendants whose run started) next to `liveDescendantCount`. `liveDescendantCount` still decides whether a blocked parent is covered. The "n live below" chip uses the new count. - Shared: the new `ExecutionAgentQueue` type and the optional field `ExecutionProjection.agentQueue`. - Server: `/companies/:companyId/live-runs` lists running runs before queued runs, so a deep queue cannot push running work out of the 50-row limit. - UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20 slots max · 41 of 295 in line · Configure", with no spinner and no elapsed timer. Configure opens the agent's Advanced view, where the run policy (`maxConcurrentRuns`) is edited. The task chat (both renderers), the run ledger and the Storybook story use it. - UI: the task header and the task rows (task list, inbox, legacy views) show a quiet "Waiting" instead of the pulsing "Live" when every live run of the task has not started. The full sentence shows on hover. "n live below" counts only started sub-tasks. A waiting sub-task still covers its blocked parent. - UI: the sidebar "N live" counts (Dashboard, agents) count only started runs. - UI: the task chat pill now names other server holds on a queued run (for example "Waiting for maintenance") instead of "Working". - Docs: DESIGN.md records the waiting-for-slot hold. - Storybook: `runs-agent-queue-wait` stories (full line, task chat, task row, one slot, mobile), with play checks. ## Verification - Red/green server test: `server/src/services/execution-projection-agent-queue.test.ts` (embedded Postgres). Before this change, the projection has no `agentQueue`. Now it checks the agent name, limit, running count, positions 1 to 4 in dispatch order (a person's retry first, then priority, not creation order), no wait while a slot is free, and isolation from other companies' runs. It also checks that the positions equal the line that the dispatch builder produces. - `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`, `recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and `agent-live-run-routes` in `server/`. - Runtime cost (rule `runtime-dependency-boundaries`): the assumption is that the line stays cheap for a deep queue. `queued-run-line.test.ts` checks that `queuedRunDispatchLine` equals the dispatcher's claim-every-run order over 400 random queues (ages around the aging boundary, recovery and person runs, blocked runs, deferred share runs, all share counters). It also orders 10,000 runs in under 250 ms (the quadratic replay takes about 600 ms). The DB test projects a 300-run queue end to end with correct positions 1 to 300. Remaining limit: the line is up to 5 s stale. - Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts `runningDescendantCount: 0` for a parent whose only live descendant is queued and never started. - UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep "Live", no "running below" count for waiting sub-tasks) and `TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the Configure link, other holds named). The suites for the run ledger, sidebar, sidebar agents, task detail, task columns and task chat also pass. - The Storybook stories render in headless Chromium and their play checks pass. - After deploy: open a queued task on staging and confirm the waiting state. ## Risks - Cost on read: a projection read that includes a queued run of a full agent now reads that agent's queued runs (only small columns) and the dispatch facts. This is at most one line per full agent every 5 s, for at most 8 agents per read. Reads with no queued, never-started runs do no more work. - The place in line is the server's prediction of dispatch order. A claim that is refused at start time (a gate, a blocker) can let the next run start first. The position updates on the next refresh. - The process-local "declined aged claim" hint that the dispatcher uses is not visible to other servers, so the projection does not use it. This can change only the aged-oldest share step. - Configure opens the agent's Advanced view (the YAML run policy). The agent's Runtime tab hides the run-policy fields today. - Low risk otherwise: no schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, headless browser) and subagents for the UI survey. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b0e99dfcfa |
fix(connections): show the setup card unless the person asked to set up the connection (#21400)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents find and request app connections with `connections_search` and `connection_request` > - For one unconnected match, the search told the agent "The user asked to connect" and said to configure the connection with `call_api` > - That text was given for every search, also when the person only asked a question or asked for app data > - The agent then created a draft connection itself and sent an absolute `http://127.0.0.1:<port>` link, and the person got no setup card to accept or decline > - This pull request gives the configure instruction only when the person's own message asks to set the connection up, and makes the agent's sign-in handoff link a relative app route > - The benefit is that a question or a data request shows the setup card, and setup links open on the host the person uses ## Linked Issues or Issue Description **What happened?** In chat eval cohort 12 (OpenCode with DeepSeek V4.1 Flash), the person wrote "Read my Notion and tell me the latest on the launch". `connections_search("Notion")` returned one result with the instruction "The user asked to connect. If method.management.allowed, use search_api/call_api ... to configure it". The agent called `POST tools/apps/connect` twice and replied with `http://127.0.0.1:<port>/COM/apps/connect?resume=...`. No setup card was shown, so the person could not decline. The same happened for "What is the easiest way to connect Notion here?". **Expected behavior** When the person asks about an app or needs its data, the agent shows the setup card (`connection_request`) and answers in plain words. The agent configures a connection itself only when the person asks for that setup. Links the agent gives are relative app routes. **Steps to reproduce** 1. Have no Notion connection. 2. In chat, write "Read my Notion and tell me the latest on the launch". 3. See that the agent creates a draft connection and links an absolute URL, with no setup card. ## What Changed - `server/src/services/connection-intents.ts`: for a single unconnected tool match, read the latest human message on the task. Give the configure instruction only when its first clause opens with a setup verb and names the matched app ("Connect Notion, then ...", "Can you set up Notion?"). "Connect Slack, then read my Notion" does not count for Notion. Otherwise tell the agent not to create a connection or draft. For a setup question, the agent answers in plain words and links the catalog setup page `/<prefix>/apps/connect?source=<app>`, because `connection_request` ends the turn before any answer text is shown. For a data request, the agent calls `connection_request` to show the setup card. A saved custom connection always uses the card. The check is conservative: a miss never configures anything. - `server/src/services/tool-access/oauth/authorization-start.ts`: the agent handoff link (`authorizationUrl`, also `requiredUserAction.setupPath`) is now a relative app route, not built from the server's redirect origin. - Tests: a red/green service test for the three messages, and the handoff test now requires a relative link. ## Verification - `cd server && npx vitest run src/__tests__/connection-intents-service.test.ts -t "configure a single match"`: passes for a data request, a setup question, a mixed-app request and an explicit setup request. With the old `connection-intents.ts` it fails (`expected 'The user asked to connect. ...' to contain 'connection_request with service notion'`). - `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts -t "generic app authentication"`: passes. - A narrow paid eval run of `connection-setup.notion-declined` and `connection-setup.choose-provider` follows on this branch. ## Risks - A person who asks for setup with other words ("I'd like Notion hooked in") now gets the setup card instead of agent-side configuration. The card still completes the setup. - One small indexed read of the latest human message per single-result search. There is no change to agent boot or time to first token. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e40a16e5c5 |
fix(heartbeat): let Stop take a task out of the agent's queue (#21407)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs a limited number of tasks at once. Other tasks wait
in line as queued runs.
> - An agent can have hundreds of waiting tasks. People need one click
to take a task out of line.
> - Stop already cancelled a queued run, but a task that was in progress
stayed in progress. It still looked busy.
> - The classic task thread also hid Stop until the run started.
> - This pull request makes Stop on a waiting task free the task and
show it as not started, in both thread views.
> - The benefit is that people can clear a long queue one task at a
time, and the agent moves on to its next task.
## Linked Issues or Issue Description
**What happened?**
An agent had many queued runs. A person pressed Stop on a task that
waited in line. The run was cancelled, but the task stayed "in
progress". In the classic thread, Stop did not show until the run
started.
**Expected behavior**
One Stop click takes the task out of line. The queued run and its wake
are cancelled. The task lock is released. The task does not look in
progress. No new run starts for that task. The agent's next waiting task
still starts. An activity entry records the change.
**Steps to reproduce**
1. Give an agent one run slot and keep it busy.
2. Assign two more in-progress tasks to the agent, so their runs wait in
line.
3. Press Stop on one waiting task.
4. See that the task still shows "in progress".
**Paperclip version or commit**
`master` at
|
||
|
|
b7774a01cf |
fix: name activity actors and explain stopped tasks in plain words (#21406)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People must never see internal jargon or raw ids in chat replies or
task comments
> - Chat eval cohort 12 found three sources: activity records show the
board user only as the raw id `local-board`, agents turn internal field
names such as `checkoutRunId` into words like "no checkout", and a
recovery notice says "the original assignee is not invokable"
> - The recovery notice appeared on 170 fixture tasks. It is not a loop:
each task got it once, because recovery blocked every in-progress task
of an agent that a person had paused
> - This pull request names activity actors, lists these internal terms
in the chat guidance, rewords the notice, and leaves in-progress work of
a manually paused agent alone
> - The benefit is plain replies and no false "blocked" tasks after a
person pauses an agent
## Linked Issues or Issue Description
**What happened?**
- `activity.audit-change`: the agent read `GET
/api/issues/{id}/activity` and wrote that the change was made by "the
board user identity (`local-board`)". Activity rows carry only
`actorType` and `actorId`.
- `activity.company-overview`: the agent wrote "no checkout" to explain
why no one is working on a task.
- `tasks.organize-open-work`: 170 of 501 fixture tasks got one system
comment each: "Paperclip cannot safely continue automatic recovery
because the original assignee is not invokable...". The owners were
paused by a person, and the stranded-work sweep blocked each in-progress
task. The existing rule already keeps `todo` work of a manually paused
agent, because blocking it fights the pause and strands the task after
resume.
**Expected behavior**
Replies name actors by name ("Board", a member's name, an agent's name)
and describe states in plain words. A person's pause does not block the
agent's in-progress work. A stopped-task notice says what happened and
what to do, in plain words.
**Steps to reproduce**
1. Change a task's priority as the local board user, then ask the chat
agent who changed it.
2. Pause an agent that owns in-progress tasks with no active run, and
let the stranded-work sweep run.
## What Changed
- `server/src/services/activity.ts`, `server/src/routes/activity.ts`,
`packages/shared/src/types/activity.ts`: company and task activity reads
add `actorName` (agent name, member name, "Board" for `local-board`,
"Paperclip" for the system). Raw ids stay in their existing fields.
- `server/src/services/agent-conversations.ts`: the existing "no raw IDs
or internal codes" rule now names checkout, native run and local-board
as internal terms, and tells the agent to name an actor by `actorName`.
- `server/src/services/recovery/service.ts`: the notice is now "<agent>
can't run right now, so Paperclip stopped this task instead of retrying
it. It is still assigned to them. To continue, resume or fix the agent,
or assign the task to someone else." The manual-pause skip now covers
`in_progress` as well as `todo` work with no run.
## Verification
- `cd server && npx vitest run
src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 4 pass. With
the old recovery service the two new tests fail (`expected 'blocked' to
be 'in_progress'`, and the old notice text).
- `cd server && npx vitest run src/__tests__/activity-service.test.ts -t
"names each activity actor"`: passes (the method did not exist before).
- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: passes.
- `src/__tests__/activity-routes.test.ts` has three tests that time out
locally on unchanged master code too; CI runs them.
- Paid eval run of `activity.audit-change` and
`budgets-costs.spend-breakdown` on a branch with this change.
## Risks
- The chat prompt grows by about 40 words. This is a small part of the
existing directive, so time to first token does not change in a
measurable way.
- A budget- or error-paused agent still gets the notice, now in plain
words. Only a manual pause keeps in-progress work as is; the sweep picks
it up after resume.
## Model Used
- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1529bca5c1 |
fix(chat): do not post a completion report that an earlier reply already gave (#21401)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In agent chat, a delegated task that finishes wakes the chat agent with `chat_task_completed`, so it tells the person in the same conversation > - A turn that runs for another reason, such as the person's form answer, can read the task when it is already done and report it > - The completion turn then starts after it and reports the same news again > - The person gets two completion messages for one task > - This pull request records when the chat agent reads its handed-off task in the done state, and skips the completion turn when an earlier reply already reported that completion > - The benefit is one completion report per task, and no extra model call for it ## Linked Issues or Issue Description **What happened?** In chat eval cohort 12, case `questions-and-approvals.duplicate-answer`: the delegated draft task finished at 13:48:23 while the chat agent's turn (woken by the form answer at 13:48:16) was running. That turn read the task, saw it done and replied "Acorn is confirmed ... the draft is finished" with a task link (13:48:50). The `chat_task_completed` turn started at 13:48:53 and replied "The Acorn draft is done" (13:49:22). **Expected behavior** The person gets one completion report per finished task in the conversation. **Steps to reproduce** 1. In chat, delegate a short task. 2. While it runs, send a message that wakes the chat agent. 3. The task finishes during that turn, and the turn reads and reports it. 4. See a second completion report from the `chat_task_completed` turn. ## What Changed - `server/src/services/chat-completion-delivery.ts`: `recordCompletedHandoffRead` appends a `chat_handoff.completed_read` run event when the conversation agent reads its own handed-off task in the done state (with the task's status version). - `prepareChatCompletionTurn`: a delivery is marked delivered (with the earlier reply as its response) when an earlier reply of the same agent in the conversation was posted after completion, links the task, and comes from a run that read the task at that exact done status. Both conditions are structural (run event and task reference), not reply wording. A reply that links the task without such a read (for example "Morgan is working on it", from a read before completion) does not count. - `server/src/services/heartbeat.ts`: when every delivery of the completion turn was already reported, the run ends as succeeded before the conversation turn starts, with no model call. - `server/src/routes/issues.ts`: task reads by an agent run call the recorder (one primary-key read of the handoff, and an event only for the chat agent's own done handoffs). ## Verification - `cd server && npx vitest run src/__tests__/chat-completion-delivery.test.ts`: 53 pass, including two new tests: the already-reported completion is delivered without a turn, and a stale linking reply still gets the completion turn. - Red: with the earlier-reply check disabled, "does not report a completion again ..." fails (`expected false to be true`). - A narrow paid eval run of `questions-and-approvals.duplicate-answer` follows on this branch. ## Risks - If the earlier reply linked the task but did not say it was finished, the person does not get a second message. The read-at-done-status condition makes this unlikely. - The skipped completion run is a short succeeded run with no model call. Boot and time to first token are unchanged for all other turns. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5c5a2fcebe |
fix(dispatch): start the assigned-work sweep's continuations and count only system recovery for the recovery share (#21391)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. #21234, #21277
and #21349 order them: person work first, a recovery share, aging, and a
share for the longest-waiting run
> - On hosted staging, one agent had about 250 queued runs. For 90
minutes after #21349 deployed, every run that started was a retry or
continuation with `retryOfRunId`. The oldest queued runs, from 15 hours
earlier, never started
> - The dispatch order was not bypassed. The oldest-run share offered
those runs, but the claim refused them every time, without a log at info
level or a visible state
> - The refusal came from the chat-control recovery gate. It needs
`retryOfRunId` to trace an automatic continuation. The assigned-work
invariant sweep queues continuations that name their source only in
context, so the gate returned "unresolved", and an unresolved claim
stays queued forever
> - This pull request lets the gate check such a continuation against
the task's previous run, and makes the recovery share count only system
recovery starts
> - The benefit is that the sweep's continuations start (it already told
people "you don't need to press Retry"), and system recovery gets its
share even when a person's recovery restarts keep starting
## Linked Issues or Issue Description
Refs #21234, #21277, #21324, #21349, #21242
**What happened?**
On hosted staging (build
|
||
|
|
30c718e04b |
test(decisions): wait for both provider calls instead of a 1-second poll in the concurrent admission test (#21387)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Company decision models admit model calls against a budget, and the server test suite checks this admission > - The test "admits concurrent work from the same budget observation and records both obligations" fails sometimes on loaded CI runners > - Cloud readiness runs this test, and any Cloud readiness failure stops the Core image from being published > - On 2026-11-13 this test failed in Cloud readiness for two master commits and stopped image publication > - This pull request makes the test wait for the real condition instead of a 1-second poll > - The benefit is that a slow runner no longer fails the test, so the release gate does not stop on a timing flake ## Linked Issues or Issue Description No public issue exists. This is a test flake. **What happened?** The test failed in Cloud readiness "verify / General tests (server (4/10))" on master. The failure was `AssertionError: expected "vi.fn()" to be called 2 times, but got 1 times` at the second `vi.waitFor`. It happened for master commit |
||
|
|
944b1c64d7 |
feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use connected apps through Paperclip's MCP client in the tool gateway > - Some provider actions run for minutes: CI runs, deploys, batch jobs > - Today such a call is lost if it outlives the tool timeout or if Paperclip restarts or deploys mid-call, and a retry can start the work a second time > - Recovery must just work, and the target rate of runs that need a manual retry is 0% > - This pull request implements the MCP Tasks extension (`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client > - The benefit is that a long provider call becomes a durable task: it survives timeouts, restarts, and deploys, and it is never started twice ## Linked Issues or Issue Description **Problem** A long provider tool call holds an HTTP request open. When it passes the tool timeout, or when Paperclip restarts or deploys during the call, the result is lost. The agent then often repeats the call, and the provider does the work again. **Proposed solution** Implement the client side of the MCP Tasks extension. Declare it on stateless tool calls. Store each returned task durably, wait on it during the call, keep polling it in the background after the call ends, and let the same call attach to the stored task instead of sending `tools/call` again. **Why it matters** CI, deploy, and batch actions are where agents lose the most work today. A durable task handle makes these actions safe across timeouts, restarts, and deploys, without a person. Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP client, and MCP Skills work this builds on). ## What Changed - **Off by default:** MCP Tasks are behind the new instance experimental setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default off. When it is off, Paperclip never declares the extension, nothing attaches to or polls tasks, pending rows do not block idle sleep, and behavior matches master. A test covers the off state. - `mcp-http.ts`: the tasks extension is declared per request, only on HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`. - New `mcp-tasks.ts`: parses task results, bounds the server's timing hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals task IDs at rest (AES-256-GCM, bound to company and connection; a keyed hash gives uniqueness). - New company-scoped table `tool_mcp_tasks` (migration `0475`, online-compatible). It links the task to the call, run, task, connection, and grant. It stores no provider result. - `tool-gateway.ts`: - A returned task is recorded, then polled up to the normal tool timeout. Its result goes through the same validation, redaction, and recording as a direct result. - Past the timeout, or when the caller disconnects, the agent gets `provider_task_pending` ("call again with the same arguments"). The task stays durable. - A call with the same endpoint, tool, full arguments, grant, agent, task, and actor attaches to the undelivered task. A retry with the same idempotency key also attaches, and it can never start different work under that key. - Input: URL requests show the provider link (Paperclip never answers them). Form requests become a question card; only the person's answer is sent with `tasks/update`. Other requests stop with a clear "needs input" error. - Stopping the run sends `tasks/cancel` and marks the task cancelled. A caller that only disconnects keeps the task. - `failed` becomes a tool error with the provider's message. An unknown task or a passed TTL becomes `provider_task_expired`. Every task stops being tracked at its TTL, or after 7 days at most. - Audit entries: `tool_gateway.task_started`, `task_input_sent`, `task_cancelled`. - New `mcp-task-work.ts`, registered at startup with the shared deadline scheduler: it resumes polling pending tasks after a restart, cancels tasks whose task closed or run was stopped, and wakes the assignee once when a task settles after its run ended. - Docs: `doc/connections/MCP-TASKS.md`, linked from `doc/connections/README.md`. Background work (per `rules/sleep-aware-background-work.mdc`): trigger is a task row write or release (`mcp-tasks` queue) and one startup pass. The deadline is the earliest `next_poll_at`. Cadence is the server's `pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting for a person. It is quiet when no row has a next poll. This is the narrow external-source exception: providers give no wake Paperclip can receive, and polling stops with the task. ## Verification - `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake 2026-07-28 server that returns tasks only to a client that declares the extension and checks `Mcp-Name`. They cover: - polling to completion, with the result identical to a synchronous call; - timeout, then an attaching call with no second `tools/call`; - a keyed retry, and a reused key with different arguments (refused); - restart mid-task: a fresh gateway with no in-memory state polls the row, wakes the agent once, and the recovered run gets the result once with no second `tools/call`; - no wake while the run is still going; - URL input (link, never answered), and form input (question card, then `tasks/update` with the answer); - cancellation by the run, by closing the task, and no cancellation when only the caller leaves; - failure, an unknown task, and TTL expiry (ends in under 3 s, no hang); - another agent never attaches; - synchronous calls unchanged and older servers never see the extension. - Red/green: with attach disabled, the timeout, keyed-retry, and restart tests fail (3). With `tasks/cancel` disabled, both cancellation tests fail (2). Before `tools/call` declared the extension, 13 end-to-end tests failed. All pass with the change. - Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`, `mcp-skills`, `server-startup-feedback-export`, `tool-gateway`, `tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests). - `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db check:migrations` passes (the migration is classified online-compatible). Snapshots pruned to five. - Agent boot and time-to-first-token: no change in the boot or first-turn path. The only new per-call cost is one indexed lookup on stateless tool calls. - Not run: a live provider that returns tasks. No public MCP server that returns tasks was available without new credentials. ## Risks - Shipped off by default. Review findings that only affect the setting-on path after this merge are tracked as follow-ups. - Migration: a new table with indexes and foreign keys to `companies`, `tool_invocations`, and `tool_connections`. Online-compatible. - Behaviour change only for servers that return tasks, which they may do only to clients that declare the extension. Older servers and stdio servers are unchanged. - A task that settles while nobody is waiting stays attachable for 1 hour, so a deliberate repeat of the same call within that hour collects the earlier result instead of starting new work. - The wake after a background task settles uses the generic task wake; the agent must call the tool again to collect the result. - Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling and roots input requests; showing a long action's progress in the task chat. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5026093d0f |
feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is interrupted, Paperclip records a fresh continuation and delivers it later as a new run. Since #21353 the task reads as continuing (`working`, no Retry) while it waits > - #21373 added "Move it forward now" (`retryAction: "move_forward"`). It is offered only after the delivered continuation run waited 15 minutes in the agent's queue > - Before delivery (`continuationDelivery: "pending"`) the task showed no button. On hosted staging such continuations waited 30 to 110 minutes behind a deep queue, and the person had no control > - #21378 and #21379 made the click safe in that state: a `move_forward` request with a recorded, undelivered continuation is accepted and delivers that continuation as the one run, with its recorded isolation > - This pull request also offers "Move it forward now" while the continuation waits for delivery, once it has waited 15 minutes since it was recorded > - The benefit is one action, with the same label and copy, for both waiting states. The click never starts a second run ## Linked Issues or Issue Description Refs #21353, #21373, #21378, #21379 **What happened?** A task has a fresh continuation that is recorded but not delivered (`continuationDelivery: "pending"`). It waits behind a deep agent queue for 15 minutes or more. The task shows "continuing" with no action. **Expected behavior** After 15 minutes the task still reads as continuing (`working`, responsibility `system`, never `failed`). It also offers "Move it forward now". The click delivers the recorded continuation as the one run. **Steps to reproduce** 1. Record a fresh continuation for a task while its agent is busy. Do not run the delivery pass. 2. Wait 15 minutes. 3. Read the task presentation. It shows `canRetry: false` and no `retryAction`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: the fresh-continuation read also loads the record status, schema, `recordedAt` and reconciled run. A `continuing` state can now carry `pendingSince` when the primary record is `resolved`, `continuationDelivery` is `pending`, the schema is the fresh-continuation schema, and the reconciled run is the observed run. These are the same facts that `undeliveredContinuation` checks at click time. - The projection offers `canRetry: true, retryAction: "move_forward"` (state `working`) when `pendingSince` is 15 minutes or older, no person Retry for that run already waits, and the run's agent is the current assignee. Under 15 minutes, nothing changes. - No button when the system gave up delivery (`invalidated`; the normal Retry rules apply), when the delivery is `not_requested`, when a person already asked, or for a held restore (the existing execution-blocker post-filter removes the offer). - New integration test `server/src/services/fresh-continuation-move-forward-offer.test.ts`. It uses the real click path for the three recorded kinds (`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`): presentation, `requestTaskRetry` with the shown `observedRunId` and `retryAction`, `resumeTaskRetry`, then the delivery pass. - Updated guards in `task-interruption-automatic-path.test.ts` and `recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a pending continuation shows no Retry at 30, 45, 110 and 240 minutes now expect the move-forward offer. The `working` / `system` / never-failed assertions stay. New cases cover 14 minutes, `not_requested`, a waiting person Retry and reassignment. - `doc/execution-semantics.md`: describes the new offer and its races. - No shared type, validator, UI or API client change. The action and its UI copy ("Waiting for the agent to finish other work. Paperclip continues this task by itself." / "Move it forward now") already exist and do not depend on the state. No runner-e2e SDK public-input pin covers the changed files. ## Verification - Red: with the presentation change reverted, 16 tests fail (`expected { intentId: null, …(7) } to match object { responsibility: 'system', … retryAction: 'move_forward' }`). - Green: `npx vitest run src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14 pass. For each kind: - The click gives exactly one run. It is the recorded continuation (`recoveryActionId`, `previousRunId`, `retryOfRunId`, `forceFreshSession`). The fresh isolated workspace is kept (`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The record reads `delivered` with that run. Two later delivery passes and a repeated resume create no second run, and the offer then ends. - The click and two concurrent delivery passes give one run. - Delivered between the offer and the click: the click is refused (`accepted: false`, no wake) and the only run is the delivered continuation. - The system gave up the delivery between the offer and the click: `alreadyMovedOn: true`, no run. - The task moved to another run between the offer and the click: refused, the only run is that other run. - Under 15 minutes: no button. - Guards: `task-interruption-automatic-path.test.ts` (37/37), `recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`, `fresh-continuation-retry-delivery.test.ts`, `fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass. Also the saved-work, workspace-recovery, native restart and closed-session retry suites that read the presentation. - `npx tsc --noEmit -p .` in `server/`: no errors. ## Risks Low risk. The change is presentation only. It adds four selected JSON fields to a read that already exists and one conditional module import. The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is unchanged and already holds the task row lock. A record without `recordedAt` or with an older schema gets no button, as before. People now see a button on tasks that showed none after 15 minutes. The button can only deliver the recorded continuation or do nothing. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f7ddab8e94 |
fix(runner): give agent task and workspace reads the workspace page href (#21384)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - People ask a chat agent where the files of a task are.
> - The answer must link the page of the workspace that holds the task's
files, when that page exists.
> - The agent reads the task with `get_task`. The task's
`currentExecutionWorkspace` had no `href`.
> - The chat directive tells the agent that a workspace without an
`href` has no page. So the agent said the page did not exist, although
isolated workspaces were on.
> - This pull request adds the workspace page `href` to the agent's task
and workspace reads, as `get_workspace` already does.
> - The benefit is that the agent links the real workspace page and does
not make a false claim.
## Linked Issues or Issue Description
**What happened?**
A person asked the CEO chat agent: "Where are the files for the website
task actually being worked on?" The agent named the task workspace and
linked the file. It did not link the workspace page. It also said "the
workspace has no separate page on this instance". The instance had
isolated workspaces on, so the page existed at
`/<prefix>/execution-workspaces/<id>`. The chat eval
`chat-features.runner-opencode.local.workspaces.locate-files` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) failed its
`bound-workspace-page-link` check in four cohorts and in one more paid
run for this reason.
**Expected behavior**
The agent reads the task, gets the workspace page link from the tool
result, and links that page in its reply.
**Steps to reproduce**
1. Turn on isolated workspaces (`enableIsolatedWorkspaces`).
2. Let an agent complete a projectless task, so the task has a bound
execution workspace.
3. In a different chat, ask the CEO where the files of that task are.
4. The agent calls `get_task` for the task.
`data.currentExecutionWorkspace` has no `href`. The agent reply names
the workspace and says it has no page.
**Additional context**
`get_workspace` and `list_workspaces` add the page `href` only for the
current task. The agent reads another task with `get_task` (`GET
/api/issues/{id}`) or with `call_api`. Those results did not get the
`href`. The run transcript shows the agent's reasoning: "The task API
includes currentExecutionWorkspace with no href", followed by the
directive rule "a workspace without an href has no page on this
instance".
## What Changed
- `server/src/services/task-workspace-tools.ts`: add
`withWorkspacePageHref` and `apiResultMayHaveWorkspace`. They add the
page `href` to the workspace in successful agent reads of `GET
/api/issues/{id}` (`currentExecutionWorkspace`), `GET
/api/issues/{id}/workspace` (`workspace`) and `GET
/api/execution-workspaces/{id}`. The function uses the same `href`
builder as `get_workspace`. It adds no `href` when the page is off, when
the read failed, or when the workspace is in a different company.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
apply the function to the results of the task bridge tools (`get_task`)
and `call_api`. The tool authority reads the page setting only for these
three operations.
- Tests: unit tests for the function, and an integration test through
the real tool authority and API. The integration test checks `get_task`
and `call_api` with workspace pages on (the `href` is present) and off
(no `href`).
## Verification
- Red/green: with the base tool authority, the new integration test
fails (`undefined` instead of `/<prefix>/execution-workspaces/<id>` for
both reads). With the fix, it passes. The unit tests fail on the base (4
failures) and pass with the fix.
- `cd server && npx vitest run src/__tests__/chat-project-tools.test.ts
src/__tests__/task-workspace-tools.test.ts
src/__tests__/agent-conversations.test.ts`: 110 passed.
- `cd server && npx tsc --noEmit -p .`: passes.
- Latency: the change runs only when the agent calls one of the three
reads. It does not touch run boot or time to first token. Local
`get_task` timing through the tool authority, 40 calls per sample, pages
on: base p50 51 ms and 74 ms; fix p50 49 ms and 54 ms. The difference is
in the noise.
- Paid eval: one narrow run of
`chat-features.runner-opencode.local.workspaces.locate-files` on this
branch:
https://github.com/paperclipai/paperclip/actions/runs/43436040405
(result in a comment below).
## Risks
- Low risk. The change only adds an `href` field to three agent read
results. It does not change REST responses for the board UI.
- The `href` appears only while isolated workspaces are on, because the
page route exists only then. This is the same rule `get_workspace` uses.
- An agent can now link a workspace page that it read through
`get_task`. The page itself still applies normal board access checks.
## Model Used
- Claude, Anthropic, model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6917d0568d |
fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server starts queued agent runs. A legacy-runtime run gets a 60 s controller lease when it is claimed. Only the run's own execution renews that lease. > - The queued-run dispatch claimed all the runs of one agent in a loop. It started their executions only after the last claim finished. > - When the database is slow (for example, while a flood of recovery runs fills the queue), one claim can take tens of seconds. The earlier claims' leases then expire before their executions start. The run fails with `controller_lease_lost` and `renewalCount: 0`. > - Also, the lease watch waited for its first 10 s tick before it renewed, even when the claim-time lease was almost spent. > - This pull request starts each run as soon as it is claimed. It also makes the lease watch renew immediately when the lease is older than one renewal interval. > - The benefit is that a slow claim batch no longer makes runs fail before they do any work. ## Linked Issues or Issue Description Refs #21127, #21042, #21222 **What happened?** On staging, legacy-runtime runs failed with `controller_lease_lost`. Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`, and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease watch started on a lease that had already expired, and no renewal was ever sent. In one run, the claim was at 23:20:02 and the loss was at 23:23:11. Losses came in groups on the same agent at the same moment. For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all lost at 23:23:11.6. **Why did it happen?** `startNextQueuedRunForAgent` claimed every run of a batch before it called `executeRun` for any of them. Each claim commits a 60 s lease, but nothing renews it until the run's execution starts its lease watch. A queue flood (fixed separately in #21222) made each claim slow, so a batch took minutes. This bug predates today's changes. The queue flood exposed it. The run-start hand-over (#21127) did not cause it: no renewal was queued on the controller-lease connection. **Expected behavior** A claimed run starts its execution, and renews its lease, without waiting for other claims. A lease watch that starts late renews at once. A lease that lapsed and was revoked, reaped or taken over is still lost and is never revived. ## What Changed - `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent` starts each run's execution right after its own claim, not after the whole batch. The concurrency cap, the agent start lock and the run-admission check before each claim are unchanged. If a later claim throws, the runs claimed before it are already executing, not stranded. After merging master, the loop walks master's queued-run dispatcher (#21234: person-initiated runs first, paced recovery delivery): each successful claim is reported to the dispatcher (`dispatcher.claimed`) and then started at once, so master's order and pacing are unchanged. Master's long-queued-run bookkeeping (#21349) is kept too: a claim that returns nothing is still recorded as declined (`noteDeclinedAgedClaim`), and a successful claim still clears that record before the run starts. - `server/src/services/legacy-controller-lease.ts`: `watchLegacyControllerLease` renews immediately when the claim-time lease has less than 50 s left (older than one renewal interval). It uses the same conditional renewal statement. That statement extends only a lease that this boot still holds on a running row. The deadline is unchanged. - Tests: red/green tests for both changes. A test also proves that a late watch on a revoked, reaped or lapsed lease still loses it and does not change the row. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass. - Red on master (source reverted, new tests kept): "starts each claimed run before claiming the next" fails (`executeRun` not called while the second claim is in progress). "renews at once when the watch starts on a lease older than one renewal interval" fails (aborted at the claim-time deadline). The three "starts late ... still loses it" cases fail on master because master loses them by deadline and does not check ownership. - After deploy: `controller_lease_lost` with `renewalCount: 0` should drop to about zero on staging, also with a full queue. A late watch now shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a silent expired start. ## Risks - Low. The executions of a batch now start while later claims of the same batch are still in progress. Execution does not claim runs and is not awaited by the loop, so the cap and the lock are not affected. - A late lease watch sends one extra renewal at start. A normal claim starts its watch within milliseconds with about 60 s left, so it sends no extra statement and adds no wait to a turn start. - No schema change. Mixed versions are safe: the renewal statement is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8af889f612 |
feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - MCP Apps (#21119) show interactive widgets from MCP servers, such as Circleback meeting cards, with the agent's reply in the task chat. > - The first version deferred three parts of the MCP Apps spec. A widget could not propose a chat message (`ui/message`). It could not give the agent context (`ui/update-model-context`). It could not go full screen. > - Paperclip also ignored the `icons` that MCP servers declare on tools (SEP-973). > - The security review of #21119 showed what these features need from day one: explicit consent, untrusted-data labels and strict icon sanitizing. > - This pull request adds the three host features and provider tool icons, with those protections built in. > - The benefit is that widgets can now work with the person and the agent, not only display data. ## Linked Issues or Issue Description Refs #21119 **Subsystem affected** MCP Apps host (task chat widgets), tool catalog, run wake prompt. **Problem or motivation** The MCP Apps host refused `ui/message` (-32601), dropped `ui/update-model-context` and kept every view inline. MCP tool `icons` were not parsed or shown. So widgets could not ask the agent a follow-up, the agent could not see what the person selected in a widget, large widgets were cramped, and tool rows used only generic icons. **Proposed solution** Add consent-gated view messages, per-view app context delivered to the task's next run as untrusted data, a full-screen dialog, and sanitized provider icons. Details are under "What Changed". **Alternatives considered** - Send view messages with a new server route that creates the comment itself. Rejected: comment creation has many rules (agent chat ownership, paused conversations, queued delivery, wakeups, receipts). The widget's Send button uses the same send path as the composer. A small route then records which view the comment came from. - Keep the iframe in place for full screen with CSS. Rejected: chat containers can create containing blocks that break `position: fixed`. A moved iframe always reloads, so the dialog hosts a new frame instead. **Roadmap alignment** Continues the MCP feature work after #21119, #21140 and #21219. It does not touch the MCP Tasks work (gateway `tools/call` polling and elicitation). ## What Changed - **`ui/message`.** The host shows "<App> wants to send: …" with **Send** and **Dismiss** under the widget (and inside the full-screen dialog). Nothing is sent before the click. Send posts the text as the viewer's own comment through the chat's normal send path (`handleThreadAdd`). `POST /api/issues/:issueId/mcp-app-views/:viewId/messages` then logs `issue.mcp_app_message_sent`. It accepts only the caller's own comment on the same task, matched by `clientRequestId`, from the last 10 minutes. One proposal per view can wait at a time. The view gets `{}` after the send, or -32000 "Message sending denied" after Dismiss. Only `user` text up to 8,000 characters is accepted. - **`ui/update-model-context`.** `PUT /api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the latest context per view in `tool_mcp_app_views.model_context` (text blocks and `structuredContent`, 16 KB at most). Updates are applied in order and deduped in the browser and on the server. A change logs `issue.mcp_app_context_updated`, and an empty update clears the context. The task's next run gets the three most recently updated contexts as an `## App context (untrusted)` wake-prompt section. The section quotes the data as JSON and says it is third-party data, not instructions. The read runs in parallel with the other run preparation and never fails the run. CLI adapters and the native runner render it through the same `renderPaperclipWakePrompt`. - **Who may do this.** Messages and context need a signed-in person with `issue:comment` on the task. Agents and API keys get 403. Other tasks and companies get 404. `message` and `updateModelContext` are advertised in `hostCapabilities` only where the viewer can send messages. - **Full screen.** The host advertises `availableDisplayModes: ["inline","fullscreen"]` and handles `ui/request-display-mode`. It honors the view's declared modes and answers with the resulting mode. It sends `host-context-changed` with `displayMode` and the container (fixed `height`/`width` in full screen), and again when that container resizes. A new **Open full screen** control sits next to "Widget from …". Full screen is a Paperclip dialog that closes with Exit, Escape or the view's own request. - **Provider tool icons.** `icons` on MCP tool descriptors are sanitized into the new `tool_catalog_entries.icons` column, outside the version hash, so a new icon never quarantines a reviewed tool. Only `https:` URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A `data:` icon is at most 32 KB, a URL at most 2,048 characters, with at most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists them keyed by gateway tool name, in one cached request per chat. Task chat tool rows and the app's actions list show them through `ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a fixed size and never inlines markup. It checks the source again before rendering and falls back to the taxonomy icon. - **Schema.** Migration `0477` adds `tool_catalog_entries.icons`, `tool_mcp_app_views.model_context` and `model_context_updated_at`, all nullable. - **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new behavior, and the finished items are removed from "Limits and follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a message", "Full screen" and "Tool rows with provider icons". The fixture view now uses `ui/message`, `ui/request-display-mode` and `ui/update-model-context`. ## Verification - `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts server/src/__tests__/tool-catalog-publication.test.ts server/src/__tests__/openapi-routes.test.ts packages/shared/src/mcp-icons.test.ts packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These tests cover: - Context: stored, deduped, logged once, 413 when too large, cleared, scoped to the task and company, refused for agents (403), and rendered into the wake prompt as untrusted data. - Messages: attributed only to the viewer's own recent comment on the same task. A teammate's comment, an unknown request id or an old comment gets 404. An agent gets 403. - Icons: stored sanitized, and the version hash and status do not change. Another company sees nothing. - `cd ui && pnpm exec vitest run src/components/mcp-app/ src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69 pass. These tests cover: - The host protocol for each new method: capabilities, consent wait, one proposal at a time, invalid input, ordered and deduped context, the size limit, and the display-mode rules. - The consent card (nothing sent without Send). - The full-screen dialog (one frame at a time). - Icon rendering (no-referrer `img`, no inline SVG, fallback). - `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for the server: clean. - Red/green: the new route and host tests fail on `master`, where the routes do not exist and the host answers -32601 or drops updates. The icon descriptor test failed when the publication write was removed. - Not run locally: the full suite, and `packages/adapter-utils/src/server-utils.test.ts › runChildProcess › does not crash when the child closes stdin before a delayed write`. That test times out on this Mac with or without this change, so it is a local platform issue. CI covers both. ## Risks - Migration risk is low: three nullable columns, no backfill. - Every run on a task now does one extra indexed read, in parallel with the other preparation. Errors are logged and the run continues without app context. - If the attribution request fails after the comment is sent, the comment stays without the `mcp_app_message_sent` activity row. This is documented. - Switching between inline and full screen restarts the view, so state inside the widget resets. This is documented as a limit. - Icon names shortened with a hash by the gateway (very long names) do not match, and those rows keep the taxonomy icon. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. One subagent (the same model) built the provider-icon part. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
39ef43a73e |
fix(retry): move a recorded continuation forward before its delivery (#21379)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run cannot resume, the server records a continuation for the task and delivers it later as a new run > - #21378 made a person's Retry deliver a recorded, undelivered continuation as the one run, with its workspace isolation > - #21373 added the `move_forward` Retry action, which does nothing when the observed run is not queued or running > - Together, a `move_forward` request before delivery returned "already moved on" and started no work > - That state can last a long time when the agent has a deep queue, and it is when a person wants to move the task forward > - This pull request accepts the request when the task has a recorded, undelivered continuation > - The benefit is one action that moves waiting work forward in each state, with one run and the recorded isolation ## Linked Issues or Issue Description **What happened?** A task has a continuation that is recorded but not delivered. The latest run is the failed source run. A Retry request with `retryAction: "move_forward"` returns `accepted: false, alreadyMovedOn: true`. No run starts. **Expected behavior** The request is accepted. Wake admission delivers the recorded continuation as the one run for the task. The run keeps the recorded isolation. A later delivery pass creates no second run. **Steps to reproduce** 1. Record a fresh continuation for a task and do not run the delivery pass. 2. Call `requestTaskRetry` with `observedRunId` set to the source run and `retryAction: "move_forward"`. 3. The receipt says `alreadyMovedOn: true`. Refs #21373, #21378, #21348. ## What Changed - `admitTaskRetry` in `server/src/services/task-retry.ts` does not return `alreadyMovedOn` when `undeliveredContinuation` finds a recorded, undelivered continuation for the task. The check runs under the task row lock. - A new test in `fresh-continuation-retry-delivery.test.ts` covers the three record kinds: `fresh_isolated_workspace`, `writer_ended` and native. - The presentation code is unchanged. The interface does not offer the action in this state yet. ## Verification - The new test failed before the change for the three kinds (`expected { requestId: null, … } to match object { accepted: true }`). - `npx vitest run src/services/fresh-continuation-retry-delivery.test.ts`: 15 of 15 passed. - `npx vitest run src/services/task-retry.test.ts`: 117 of 117 passed. - `npx vitest run src/__tests__/recovery-wake-herd.test.ts -t move`: 2 of 2 passed. The case where the run ended and no continuation is pending still returns `alreadyMovedOn`. - `npx tsc --noEmit -p .` in `server/`: no errors. ## Risks Low risk. The change adds one read under a lock that the function already holds. The read runs only for a `move_forward` request when the observed run is not queued or running. A request without a pending continuation behaves as before. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ed16a5b1a8 |
fix(recovery): do not continue from a latest run read while it was still finishing (#21365)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent's run ends without a live next step, a background sweep finds the stranded task and queues a continuation > - The sweep reads the task's latest run first, and checks for a live execution path several queries later > - A run that finishes between those two reads looks neither live nor failed. The sweep fell through to its generic continuation (`issue.continuation_recovery`) with the stale "running" snapshot > - In a chat, that woke the agent about a second after its turn had answered the person, and the agent posted a redundant "nothing new is pending" reply > - This pull request makes the sweep act only on a latest run it observed as finished. A run it read while still active is left to its own completion and to the next sweep > - The benefit is that a chat turn that already answered is never woken again by recovery, with no change to real recovery ## Linked Issues or Issue Description **What happened?** In a chat, the person asked a follow-up and the agent answered. The answering run's success was written at 10:09:28.840. A continuation run was queued at 10:09:28.841 with `wakeReason: issue_continuation_needed`, `source: issue.continuation_recovery`, and `retryOfRunId` set to the run that had just succeeded. The agent ran again and posted an extra reply. The generic continuation source is only reachable when the sweep's latest-run snapshot is neither succeeded nor live, so the snapshot was read while the run was still `running`, and the run had finished by the time of the live-path check. **Steps to reproduce** 1. Have an assigned `in_progress` task whose latest run is `running`. 2. Start the stranded-work sweep. It reads the latest run as `running`. 3. Before the sweep's live-path check, let the run finish with `succeeded` and release the task lock. 4. The sweep finds no live path and queues `issue.continuation_recovery` for the finished run. **Expected behavior** The sweep queues nothing for a run it read while still active. Its own completion settles it, and the next sweep decides from the committed outcome. ## What Changed - `server/src/services/recovery/service.ts`: after the live-path check, the stranded sweep skips a non-review task whose latest-run snapshot was not terminal. Review tasks keep their participant-run check, which already does this. - `server/src/services/recovery/service.ts`: an optional test hook, `afterStrandedLatestRunRead`, runs after the sweep reads the latest run, alongside the existing `beforeStaleLockClear` and `beforeOrphanedRunTerminalWrite` hooks. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: a regression test finishes the run between the two reads and expects no continuation. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "still finishing"` fails without the fix (one `issue.continuation_recovery` wake with `retryOfRunId` set to the finished run) and passes with it. - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-issue-liveness-escalation.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/durable-chat-wakeup.test.ts src/__tests__/live-recovery-work.test.ts`: 790 passed. One unrelated test timed out under parallel load and passes on its own (its latest run is already terminal, so the new check does not apply). - `cd server && npx tsc --noEmit` passes. ## Risks - Behavior: for any task whose latest run is `queued`, `running` or `scheduled_retry`, the live-path check already skipped it, so the only changed outcome is the race above. A run stuck in `running` with no owner is still settled by the existing orphaned-run reaper; the sweep then sees its terminal status. - Latency: the check is in memory on an already-read row. It adds no query and nothing before a turn starts; recovery after a real stall is at most one sweep later only inside the race window. - Saved state: no persisted shape changes, so old saved runs and mid-run deploys behave as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3f030971dc |
fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When Paperclip cannot resume a cut-off run exactly, the fresh-continuation fallback records a continuation for the task. A delivery pass later starts it as a run. Staging shows this pending state can last 30 to 110 minutes for backlog items. > - The fallback also decides how the next run is isolated from the old writer. For `fresh_isolated_workspace`, the next run must prepare a new workspace. > - A person's Retry pressed while the continuation was recorded but not yet delivered started its own run at once. That run did not carry the recorded isolation, so it could use the shared directory next to a writer that is not proven stopped. It also did not carry the continuation's recovery context. > - This pull request makes wake admission deliver the recorded continuation as the person's run, under the task lock. It also folds a saved Retry into a delivered continuation for every record kind. > - The benefit is one run per interruption, whatever arrives first, and the recorded isolation decision is never lost. ## Linked Issues or Issue Description Refs #21348 **What happened?** A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while a fresh continuation was recorded but not delivered, was admitted at once as an ordinary `task_retry` run. That run had no `freshContinuationIsolation` and no `recoveryActionId`. On a projectless task the run used the shared directory that the fallback had decided to isolate from. The later delivery pass marked the continuation as superseded by that run. The reproduction showed exactly one run, so the reported "two runs" did not happen in that order. The isolation was lost for all three record kinds tested. For `writer_ended` and native records, a Retry saved before delivery was not folded into the delivered continuation, because the fold only covered `fresh_isolated_workspace`. **Expected behavior** With a recorded, undelivered continuation, the next run admitted for the task is that continuation. It has the recorded isolation and the person's Retry is recorded as answered by it. **Steps to reproduce** 1. Record a fresh continuation (`ensureFreshContinuation`, delivery `recorded`) for a projectless task whose legacy writer is on another host. 2. Call `requestTaskRetry` and then `resumeTaskRetry`. 3. Before this change: one `task_retry` run without `freshContinuationIsolation` or `recoveryActionId`, while the record is still `pending`. ## What Changed - `fresh-continuation-retry-fold.ts`: - `undeliveredContinuation` reads the task's pending fresh-continuation record under the task lock (`for update`). It returns nothing when a run of the task was created after the record. - `undeliveredContinuationContext` gives the run the continuation's `recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for `fresh_isolated_workspace`, `freshContinuationIsolation`. - `deliverContinuationAsRun` marks the record `delivered` to the new run in the same transaction, settles its coalesced holds, and folds other waiting Retries. - `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry into any delivered fresh continuation, not only an isolated one. - `heartbeat/queue.ts` (wake admission, issue transaction): a non-system, non-delivery wake with no execution blocker takes the pending continuation. The run is inserted as before, then delivered as the continuation. The fold after a delivery wake now applies to every fresh-continuation kind. No change to ranking or to the claim loop. - `execution-recovery-resolution.ts`: the delivery pass folds a saved Retry for every fresh-continuation kind. The superseded-run fold keys on `recoveryActionId`. - The person's run keeps its `task_retry` wake reason and the user's wake receipt, so the existing #21349 rules rank it as person work. - Limits: the person-triggered delivery does not spend an automatic attempt. It adds no `issue.continued_automatically` entry and no scheduled transient retry, and it is not gated by the hourly cap or by delivery-pass pacing. The record itself was already counted when it was recorded. The agent's concurrency limit still applies at claim time. ## Verification New file `server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12 tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and native (`remote_epoch`) records, through the real `requestTaskRetry` + `resumeTaskRetry` path: - A Retry during pending delivery gives exactly one run. That run is the continuation, carries the recorded isolation (the workspace readiness reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as person work. Later delivery passes add nothing. - Delivery first, then a saved Retry: one run, and the Retry is coalesced. A Retry on the delivered run joins it through the busy path. - A Retry and two delivery passes race (concurrent transactions): one run. - After the continuation's own failure, a Retry gets its own run. - The delivery pass is held by pacing (backlog source, agent queue full), and the hourly cap and the shared attempt ledger are exhausted. The person's Retry still starts the run. Red before the fix: 8 of 12 failed. Green after: 12 of 12. Local runs, one file at a time: - `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs a person's Retry pressed before delivery…", a deadlock in `afterEach` truncate. It passes alone, and it fails the same way on unmodified master under this machine's load. - `__tests__/recovery-wake-herd.test.ts`: 29 passed - `__tests__/execution-control-work.test.ts`: 27 passed - `__tests__/issue-recovery-actions.test.ts`: 112 passed - `workspace-recovery-fallback.test.ts`: 19 passed - `heartbeat/queue.test.ts`: 101 passed - `task-retry.test.ts`: 117 passed; `legacy-preparation-retry-hold.test.ts`: 5 passed - `server` `tsc --noEmit`: clean ## Risks - A person's message, chat input or manual wake that arrives while a continuation is pending also becomes that continuation, with a fresh session and the recorded isolation. Before, such a wake ran without isolation and superseded the record. This is the same rule for every admission path, and it closes the shared-directory gap there too. - A wake that takes the context and is then deferred by a later gate can keep the isolation marker in its saved context. If the record is delivered meanwhile, that later run prepares a new isolated workspace. That is the safe direction. - No schema change and no migration. Nothing is added before turn start for healthy runs: the new read runs only at wake admission. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent with tool use (shell, file edit, test runs). Model ID `claude-opus-5-5`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5ba26600ef |
fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs agent turns through a Runner, often in a remote sandbox, and must recover runs after a restart without help from a person > - When a Runner process is lost, a new Runner restores the conversation. If the provider turn is gone, the Runner sends `turn.failed` with `provider_turn_lost_on_restore` and `recoverable: true`. The Runner sends this only after it proves that the old process ended > - The runtime can continue such a turn in the same conversation, but only when the server sets `resumeInterruptedTurn`. The server set it only for a local `resume_dead_runner` claim or a scheduled resume, and a scheduled remote resume is refused. So on a remote Runner reattachment the runtime did not continue the turn, and the run ended > - This pull request sets `resumeInterruptedTurn` for a `reattach_remote_runner` claim too > - The benefit is that a remote run continues the lost turn in the same conversation immediately, instead of failing and then restarting 3 to 6 minutes later ## Linked Issues or Issue Description **What happened?** On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3 hours ended with `native_provider_terminal_failed` and the message "The runner restored the conversation after process loss, but the previous turn is no longer active." The Runner marked each of these events `recoverable: true`. Example: run 646e2efb. A new Runner process started at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at 09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In each case, a separate recovery path continued the task only 3 to 6 minutes later. **Expected behavior** When the Runner marks a lost turn as recoverable after a restore on a remote reattachment, the run continues the turn in the same conversation. **Steps to reproduce** 1. Start a native run on a remote sandbox Runner. 2. Lose the Runner process while a turn is active, so that a new Runner reattaches through a `reattach_remote_runner` claim. 3. The new Runner restores the conversation and reports `provider_turn_lost_on_restore` with `recoverable: true`. 4. See that the run ends with `native_provider_terminal_failed` and does not continue the turn. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with Codex on a Daytona remote sandbox. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: `resumeInterruptedTurn` is now also true for a `reattach_remote_runner` restart-recovery claim. The runtime still continues the turn only when the Runner itself sends `turn.failed provider_turn_lost_on_restore` with `recoverable: true` for the checkpointed active turn. - `doc/execution-semantics.md`: one paragraph describes this behaviour, and says that a lost turn that still ends the run keeps `native_provider_terminal_failed`. - Tests: - `native-scheduled-resume-continuation.integration.test.ts`: "continues a lost turn on a same-run remote Runner reattachment" (2 cases: a plain reattachment and a `replaceUnansweredRunner` reattachment). - New `native-recoverable-lost-turn.integration.test.ts`: a guard through `executePaperclipNativeSession`. The runtime saves a failed-terminal checkpoint and then raises the lost turn (recoverable and not recoverable). The run ends as `native_provider_terminal_failed`, and the hold cause stays one that the replacement lane continues. - An earlier revision of this PR also mapped a recoverable lost turn to `native_session_interrupted`. Review showed that this does not make the run retryable, and the hold it writes is not picked up by any automatic continuation. That hunk is removed. ## Verification - Red without the product change: the two integration cases fail with `passed` = `[false]` (expected `[true]`). - Green with the change: - `cd server && npx vitest run src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts` — 14 passed. - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` — passed (with the scheduled-resume file: 224 passed, 14 skipped, pre-existing skips). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts` — 1469 passed, 7 skipped. - `cd server && npx tsc --noEmit` — exit 0. - Staging evidence (hosted staging, build `d1d71a7755`, read-only trace): 10 runs in 3 hours ended with `native_provider_terminal_failed` although the Runner marked the lost turn `recoverable: true`. Run 646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49, `turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed` 09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes later through a separate recovery path. ## Risks - This change has not run live on staging or on a Daytona sandbox yet. The evidence is the staging trace and the local tests. - This PR does not introduce or widen the gap where a recoverable lost turn still ends the run. On master, that run already ends as `native_provider_terminal_failed`, and the replacement lane continues it after its decision (3 to 6 minutes on staging). This PR removes the run-ending case whenever the Runner reports the turn recoverable for the checkpointed active turn on a remote reattachment. Where the run still ends (for example, when the continuation itself is lost), the behaviour is unchanged, and the guard test `native-recoverable-lost-turn.integration.test.ts` proves that it stays `native_provider_terminal_failed`. - The remaining case is tracked with the owner of the recovery path: recovery evidence will accept `provider_turn_lost_on_restore` / `recoverable` directly, after their ownership-deadline PR #21364. An earlier hunk here tried to shorten that case. An integration test through the real path showed that it would create a board-owned ambiguous hold that no automatic pass continues, so the hunk is removed. - This change does not fix why the turn is lost. In staging, a retry replaced a live Runner. A separate fix for that is in progress. - The runtime continues a turn only when the Runner sends the recoverable lost-turn signal. The Runner sends it only after it proves that the old process ended. So no provider turn runs twice. - No Runner image change. No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d9b42b50fd |
feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is interrupted, Paperclip records a fresh continuation and continues the task by itself. Since #21353 the task reads as continuing (`working`, no Retry) for as long as that continuation waits > - That is truthful, but a person has no control while it waits. On hosted staging one agent had 21 running and 209 queued runs, and continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a waiting continuation into the agent's queue within minutes, so most of that wait is in the queue, behind other work > - #21353 removed Retry from this state because a Retry started its own run, which raced the continuation and could give the task two runs > - A Retry that observes the queued continuation run cannot start a run: admission sees a queued run and joins the Retry to it (`coalesced`, `runId` = that run). Under #21234's ranking, a queued run that a person joined starts ahead of system recovery and ordinary work > - This pull request keeps the continuing state and, once the queued continuation has waited 15 minutes, also offers Retry labelled "Move it forward now" > - The benefit is that a person can move a long-waiting task to the front of its agent's queue, and the click can never create a second run ## Linked Issues or Issue Description Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200 **What happened?** On hosted staging, tasks with a recorded continuation waited 30 to 110 minutes behind a deep agent queue. The task showed "continuing" with no action. The person could only wait. **Expected behavior** The task still never shows "failed". After a long wait the person can move the waiting continuation forward. The click must not start a second run. ## Which states a Retry joins (established before building) I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its tests, and drove the real click path (`requestTaskRetry` followed by `resumeTaskRetry`, as the route does) in a probe against each record state: | Continuation state | What a Retry does today | Offer Retry? | |---|---|---| | Recorded, waiting for delivery (`continuationDelivery: "pending"`, no run yet) | The Retry is admitted at once as its own `task_retry` run, for every isolation (`fresh_isolated_workspace`, `writer_ended`, native). The delivery pass then marks the record delivered (superseded). #21348's fold never runs: it folds only a Retry that is still `deferred_issue_execution` when the continuation run is created, and only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred, so the click races the delivery pass. | **No** (unchanged) | | Delivered, continuation run `queued` | `requestTaskRetry` sees the latest run is busy and records the Retry as `coalesced` into that run in the same transaction. No run is created; `resumeTaskRetry` acts only on deferred Retries. The run is now ranked as a person's (`joinedByUser` in `queue.ts`). | **Yes, after 15 minutes** | | Continuation run `running` | Same join, but the run already started, so there is nothing to move forward. | No | | Continuation ended | A Retry gets its own run (the existing failed-run path). | Existing behaviour | So the Retry offered here is joined by the existing busy-run admission in `requestTaskRetry`, not by #21348's fold. Its source run is the queued continuation itself (the run the presentation names as `observedRunId`), not the run the continuation was created for. ## What Changed - `server/src/services/task-interruption-presentation.ts`: for a task whose latest run is `queued`, created 15 minutes ago or more, and named as `continuationRunId` by a delivered fresh-continuation record, the projection returns `state: "working"`, `canRetry: true`, `retryAction: "move_forward"`, responsibility `system`, `observedRunId` = that queued run. It does not offer this when a person's wake already joined the run, when the run's agent is not the assignee, or when a person-owned wait applies (approval, reconnect, budget, pause and the others are checked first). Two small reads run only for queued runs older than 15 minutes. The existing restore-hold check that withdraws `canRetry` also withdraws `retryAction`. - `packages/shared/src/types/execution-projection.ts`: new optional `retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It only changes the label; `canRetry` still decides whether Retry is offered. - `ui/src/components/ExecutionBlockerNotice.tsx`: a `working` presentation with `retryAction: "move_forward"` shows "Waiting for the agent to finish other work. Paperclip continues this task by itself." and a "Move it forward now" button that calls the same task Retry. Token-only classes; `pnpm check:token-gates` is clean. - One join-only action, `move_forward`, carried from the offer to admission (contracts kept in sync): - `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts `retryAction: "move_forward"`. The route already passes the body to `requestTaskRetry`. - `server/src/services/task-retry.ts` (`admitTaskRetry`): right after the stale check, still under the task row lock and before any other Retry path, a `move_forward` Retry whose observed run is no longer `queued` or `running` returns `{ accepted: false, alreadyMovedOn: true, requestId: null }` and writes nothing. This block was cleared with the owner of Retry admission. The owner will later extend it so that "continuation recorded, not yet delivered" also counts as actionable, after their pending-delivery change lands (about two lines on their side). Until then this presentation never offers the button in that state, so the early return cannot misfire from our UI. - `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click sends `retryAction: "move_forward"` when the offer has it. `alreadyMovedOn` shows "The task already moved on, so nothing new was started. It now shows its current state." - What a click does at each timing: continuation `queued` → joined to that run (moves it forward); `running` → joined (nothing more to do); ended → "already moved on", nothing started. - A joined click cannot become a run later. The busy path writes it as `coalesced` with `runId` set to the continuation and `finishedAt` set (`task-retry.ts:456-461`). Promotion after a run ends (`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`; `modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`) and `resumeExecutionWaitComments` (`:876`) pick up only `deferred_issue_execution` wakes. - `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry for this state. That row sends the failed run as the observed run, which is no longer the latest run, so its Retry would be refused as stale. - `doc/execution-semantics.md`: one paragraph on the offer, when it ends, and why there is none before delivery. ## Verification - Red (before the product change, new tests only), in `server/`: `pnpm exec vitest run src/__tests__/task-interruption-automatic-path.test.ts src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both failures are the missing offer (`canRetry: false`, no `retryAction`) for a continuation queued 15+ minutes. - Green after merging master |
||
|
|
eb54831d28 |
fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs often use a remote Runner in a sandbox. After a restart, a deploy or a same-run retry, the server reattaches to that Runner or starts a new Runner over its saved state > - A Runner that no controller reaches before its connection lease expires stops its provider, saves its state as `recoverable_failure` with `lease_expired_requires_bootstrap`, and exits. The Runner protocol says that only a new process with a fresh bootstrap ticket can reconnect, and that the new process continues from the same cursor > - Reattachment started a new Runner over saved state only for a settled `suspended` exit (or an exit before `session.open`). So it refused this exit, and the run ended with `runner_remote_process_identity_unavailable` (`native_continuation_requires_reconciliation`) > - This pull request lets reattachment launch a Runner over the exact saved state of a Runner that exited to ask for a fresh bootstrap > - The benefit is that the run continues by itself after a long controller gap, with no Retry and no manual sandbox repair ## Linked Issues or Issue Description Refs #21329, #21362, #20853 **What happened?** On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on remote Daytona Runners ended with `native_continuation_requires_reconciliation` and the error `runner_remote_process_identity_unavailable`, about 5 per hour, after #21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df, dd595934, 50c6e6e6) ended the same way. The last `native.remote_runner_recovery_failed` event recorded `process_verification` / `process_not_running`, `runnerLifecycle: recoverable_failure`, `runnerRecoverableFailure: lease_expired_requires_bootstrap`, and the Runner diagnostic "connection lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the 15 runs that ended with this error had this exact exit. Timeline of run c6fabe9a: - 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded sandbox. - 10:54:02: the controller sent `turn.start`. At 10:54:47 the command timed out: "PRP command turn.start timed out (runner connected; pending ahead 0; last runner event seq 2 91s ago)". The run resumed in the same run and the turn then ran (tools completed from 11:14). - 11:39: a graceful server shutdown suspended the controller. At 11:43 the next controller started the reattachment, but it did not reconnect to the Runner until the server restarted again at 11:58. In that gap the Runner's lease expired and the Runner exited. - 11:58:53: the reattachment proved the process gone and refused the `recoverable_failure` state. The run failed at 11:59. Why `turn.start` timed out while the Runner was connected: the Runner answers `turn.start` only after Codex answers its `turn/start` JSON-RPC request. While it waits, the Runner buffers Codex notifications and commits no events, so the controller sees a quiet connected Runner with nothing pending ahead. The controller's bound for the `turn.start` command result is a fixed 30 s. #21329 changed only the later wait for `turn/started`. In a sandbox that needed 8 minutes to start, Codex answered later than 30 s. That timeout is recoverable, and the run did recover from it. This pull request does not change it (see Risks). The failure that ended the runs is the refused reattachment. **Expected behavior** When a remote Runner exits because its lease expired (or its reconnect grace or deadline ended) and its saved state is intact, reattachment starts a new Runner from that state and the run continues. **Steps to reproduce** 1. Start a native Codex run on a remote sandbox Runner. 2. Keep every controller from reaching the Runner for longer than its connection lease (on staging: a deploy that stalled the reattachment for about 15 minutes). 3. The Runner saves `recoverable_failure` / `lease_expired_requires_bootstrap` and exits. 4. The next reattachment fails with `runner_remote_process_identity_unavailable`, and the run ends. **Agent adapter(s) involved** `paperclip_runner` with Codex on a Daytona remote sandbox. The rule is not provider-specific. ## What Changed - `remote-runner-recovery-diagnostics.ts`: new `exitedRunnerRequestedFreshBootstrap` (diagnostic) and `runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle `recoverable_failure` with `lease_expired_requires_bootstrap`, `transport_reconnect_grace_exceeded` or `transport_reconnect_deadline_exceeded`. These are the three codes the Rust Runner records after `shutdown_preserving_cleanup`. The state check also requires no pending terminal delivery and no warm transition. - `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when the exact Runner process is proven gone and its diagnostic names one of these exits, the server reads the saved state again. If the state has the same six identity fields and still names that exit, the reattachment launches a Runner from it instead of failing. The outbox does not need to be empty and a turn may be active: the new Runner replays from the same cursor. The attempt bound is the one the reattachment already admits (3, or 4 for an unanswered-Runner replacement claim). Other exits are unchanged. - `native-session-executor.ts` (host preparation in `createRunnerdBackend`): a resumed sandbox for a reattachment that did not adopt a process re-reads the Runner state. When it still has the same six identity fields and names the fresh-bootstrap exit, preparation keeps it in place, as for an adopted live Runner. It no longer fails with `runner_harness_state_mismatch: resumed_not_suspended` (found by Greptile). - `doc/execution-semantics.md`: one paragraph describes the rule and the staging evidence. ## Verification - Red on `origin/master`: the 5 new "launches a new Runner over the saved state after ..." cases failed (the reattachment threw `runner_remote_process_identity_unavailable`). The 7 new fail-closed cases passed. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote controller restart adoption"` passes 118 tests. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm owner before refreshing run authority", warm-session supervision, not touched here) passes when run alone. It is a timing failure under full-file load. - Launch path: "keeps the saved mid-turn state of a remote Runner that exited for lease_expired_requires_bootstrap when the reattachment prepares its sandbox" runs `createRunnerdBackend` and `prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red before the host-preparation change (`resumed_not_suspended`), green after. An unknown recoverable failure stays fail-closed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts` on the final revision: 1494 passed, 7 skipped. - `cd server && node scripts/typecheck-layers.mjs` (includes test programs): passes. - Fail-closed cases covered: an unknown recoverable failure, a warm transition, a pending terminal delivery, a state that changed after the exit, a different Runner instance, attempt 4 on an ordinary claim, and attempt 5 on a replacement claim. - Staging read (read-only): the 5 sample runs and 69 runs with this error in the last 9 hours were classified from their `native.remote_runner_recovery_failed` events. ## Risks - Medium. Reattachment now starts a new Runner over a saved state that can be mid-turn. The Runner itself stops its provider before it saves this state. The new Runner restores the session. It continues the turn, or it reports the lost turn as recoverable only after it proves that the old provider process ended. A recoverable lost turn on a remote reattachment ends the run as `native_provider_terminal_failed` until #21362 merges; the replacement lane then continues the task. That is still better than the current reconciliation hold. - The attempt bound does not grow. A Runner that exits this way again on a later reattachment is bounded by the same provider attempt budget. - This does not change the 30 s `turn.start` command bound or the deploy gaps that let the lease expire. Those are separate follow-ups. 57 older runs with this error had a different exit (`ready`, not settled). This pull request keeps that case fail-closed; #20853 covers the settled-`ready` part. - No schema, API, UI or Runner image change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution, read-only browser reads of staging run data). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e3faf579a1 |
test(recovery): drain background runs before the fresh-continuation test cleanup (#21375)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server continues an interrupted task in a new run from saved
work
> - `fresh-continuation-fallback.test.ts` tests that behavior against a
real PostgreSQL database
> - Some tests start runs that continue in the background after the test
body returns
> - The cleanup truncated its tables while those runs still held locks,
and PostgreSQL sometimes reported a deadlock
> - The failure appeared on unrelated pull requests in server shard 9/12
> - This pull request makes the cleanup wait for in-flight runs and
retry the truncate on a deadlock
> - The benefit is that this file no longer fails CI at random
## Linked Issues or Issue Description
**What happened?**
The `afterEach` in
`server/src/services/fresh-continuation-fallback.test.ts` sometimes
fails with `PostgresError: deadlock detected` (code 40P01). The truncate
waits for an `AccessExclusiveLock`. A background run that a test started
holds an `AccessShareLock` and waits for another table. No assertion
fails. The test that receives the failure changes from run to run.
**Expected behavior**
The cleanup completes after each test. The file passes on each run.
**Steps to reproduce**
1. Check out master at
|
||
|
|
7b27fc453f |
fix(presentation): a provider failure the system is still recovering shows as working, not as needing Retry (#21369)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a provider ends a run's session with a terminal failure
(`native_provider_terminal_failed`), Paperclip recovers by itself: it
checks whether a safe replacement is possible, saves the remote files,
and then continues the task in a fresh run from saved work
> - These steps take 15 to 60 seconds. Until the continuation is
recorded, the task view did not know the system was still deciding
> - In a restart-chaos wave, two tasks showed a failed attempt with a
Retry button for 13 to 27 seconds, and one of them changed state in a
loop: failed, then working, then failed with Retry, then working
> - Product rule: a task that the system will continue by itself never
shows as failed with a Retry button, and the state does not change back
and forth during one recovery
> - This pull request reads the record that says the system is still
deciding (the native coordinator's terminal-failure row) and shows the
task as working during that time, within the existing 2-minute bound
> - The benefit is one steady "working" state from the failure to the
new run, and Retry comes back at once when the system hands the task to
a person
## Linked Issues or Issue Description
Refs #21264, #21353, #21200, #21320
**What happened?**
Restart-chaos wave `w-20261113T092657-s7302` on master
|
||
|
|
57b787d3d2 |
fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run finishes but copying its files out of the sandbox fails, Paperclip holds the task and tries to recover the saved files. When recovery gives up, the task offers Retry. > - Some saved files can never be recovered: the sandbox that held them was destroyed, or Paperclip cannot prove which runs used a shared sandbox. Each Retry ran the same check again, failed again, and parked the Retry in a wait that never ends. > - The task told the person to press Retry again or to wait for a Paperclip update. That breaks the rule that recovery must work and that a Retry that is shown must work. > - This pull request makes Retry continue the task in a fresh session and a new workspace from the work that is recorded, when the saved files are provably unrecoverable or cannot be proven safe. The task says clearly that the unsaved files were not recovered. > - The benefit is that an offered Retry starts a run, and nobody waits for a fix that another click cannot bring. ## Linked Issues or Issue Description Refs #21315, #21295, #21267 **What happened?** On hosted staging, two blocked tasks offered Retry (`canRetry: true`). Each Retry was accepted, and within about 3 seconds it was parked in a wait. No run started. Every later Retry did the same. - Task A: the hold cause was `native_workspace_sync_out_retry_exhausted`. Recovery ended with `source_unavailable` (`use_alternative_strategy`). The sandbox that held the files had a signed absence proof and a `destroyed` termination receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the agent's files failed. Retry to try again". - Task B: the same hold cause. Recovery ended with `allocation_ownership_ambiguous` (`service_required`), because the sandbox was a reusable sandbox that other runs also used. Each Retry opened a new recovery generation, ran the same check, and was parked with `saved_work_recovery_refused`: "Retry runs the same check again and succeeds only after Paperclip is updated." - In both tasks, the agent's turn had finished and its result was accepted. Its work was recorded outside the sandbox: a plan document and screenshot attachments for task A, and a pushed branch with an open pull request and an implementation document for task B. Root cause: admission had a branch for an accepted result whose files were not yet saved. When the recovery budget was spent, the branch always returned a `saved_work_recovery_*` wait. It did not check whether the files could ever be recovered, so the fresh-workspace path that admission already has was never reached. The automatic retained-source fallback did not help either, because it handles only the remote workspace v1 reference, and these runs use the native workspace sync v3 reference. Also, Retry opened a new recovery generation for an ownership refusal, which only repeated the same check. **Expected behavior** When the saved files are provably unrecoverable or cannot be proven safe, Retry continues the task in a fresh session and a new workspace from the recorded work. The old files are never applied and the source sandbox is not changed. No provider turn is replayed. The task says that the unsaved files could not be recovered. The Retry offer and admission agree. **Steps to reproduce** 1. Let a native run finish with an accepted result, and let its workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`). 2. Let saved-work recovery end with `source_unavailable` (sandbox destroyed) or with `allocation_ownership_ambiguous` (shared reusable sandbox). 3. Press Retry. 4. Observe that the Retry receipt waits with `saved_work_recovery_failed` or `saved_work_recovery_refused`, and no run starts. Press Retry again and observe the same result. **Paperclip version or commit** Staging on `origin/master` after #21315 (`bbafa78fe2`). **Deployment mode** Hosted cloud (staging), remote Daytona sandbox runner. **Agent adapter(s) involved** `paperclip_runner` (native runtime runs). ## What Changed - `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure` (`system-recovery-state.ts`) are the one rule for saved files that no recovery can apply. The rule is true for a spent recovery whose failure is a permanent source refusal: `source_unavailable`, `allocation_ownership_ambiguous` or `historical_source_missing`. It is false for `source_binding_invalid`, because a new recovery generation can still verify that source. It is false for `archive_unsafe` and `snapshot_verification_failed`, because a fresh attempt still imports earlier saved snapshots and would fail before the agent starts. It is also false after a fresh attempt is recorded. - Admission (`explicit-native-continuation.ts`) no longer parks a Retry of an accepted result in `saved_work_recovery_*` when every live recovery of the run is unrecoverable. The Retry takes the existing fresh-workspace path (`fresh_isolated_workspace`). That path keeps all stop proofs, leaves the source sandbox and workspace untouched, and starts a fresh session. - When such a Retry starts, the task gets one system comment. The comment says that Paperclip could not recover the unsaved files, why, and that the task continues from its recorded work (comments, documents, attachments and pushed commits). The settled hold records the same text. - Retry (`task-retry.ts`) binds to an unrecoverable recovery generation. It does not open a new generation that repeats the same check. - The blocker text for unrecoverable files says that Retry continues the task in a new workspace from its recorded work. It no longer says that Retry succeeds only after an update. - `canAdoptNativeSourceRetention` and the permanent source-refusal lists moved into `system-recovery-state.ts`, so that admission, Retry and messages share them. The old modules re-export them. ## Verification - Red/green: `cd server && npx vitest run src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on `origin/master` sources (or on the earlier head for the snapshot case) and pass with this change. They are built from the two staging shapes (destroyed sandbox with `source_unavailable`, and a shared reusable sandbox with `allocation_ownership_ambiguous`): - offers a Retry that continues the task (blocker and task view agree, and the text names the fresh workspace) - starts a fresh run from the recorded work: the Retry receipt has no wait and a run, the strategy is `fresh_isolated_workspace`, the hold is settled with the fresh attempt, one system comment explains the lost files with no raw IDs, the source lease is unchanged, and no new recovery generation opens - keeps a Retry out of a fresh attempt that would import a succeeded snapshot whose archive is unsafe - `workspace-recovery-warm-handoff.test.ts` › "wakes physical preservation from HTTP Retry before passive history without replaying the provider" failed on an earlier head of this PR, which also treated `source_binding_invalid` as unrecoverable. It passes unchanged now. - Updated tests that pinned the old repeat-the-check behavior: `workspace-recovery-preparation-gates`, `task-retry`, `system-recovery-state`, `workspace-recovery-retry-copyback`. - Related suites pass: `system-recovery-state`, `workspace-recovery-preparation-gates`, `workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`, `task-retry`, `saved-work-retry-eligibility`, `workspace-recovery-fallback`, `explicit-native-continuation`, `workspace-recovery-admission`, `task-interruption-query`, `native-workspace-sync-retention`. - `cd server && node scripts/typecheck-layers.mjs` passes (source and test layers). ## Risks - Medium-low. A Retry of an unrecoverable hold now starts a new agent turn in a new workspace instead of waiting. The turn starts only through the existing fresh-workspace admission, which keeps every stop and cleanup proof. - Unsaved files of the old run are not applied. That was already true; the change is that the task continues and says so. - `source_binding_invalid`, `snapshot_verification_failed` and `archive_unsafe` keep today's path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5504151eae |
fix(recovery): restart a remote bootstrap again after it waited for a busy agent directory (#21370)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each interrupted native run to restart
recovery. Recovery decides if the run can continue safely.
> - For a remote sandbox run that reached no provider work, recovery
restarts the bootstrap on the same run (`bootstrap_incomplete`).
> - The restarted executor can meet a busy agent directory while it
prepares. It then defers the run for a few seconds and asks restart
recovery to look again.
> - That deferral saves the coordinator as `retryable_failure` while the
run stays `running`. No remote proof accepted this shape, so recovery
blocked the run with `ambiguous_remote_recovery_evidence`.
> - This pull request lets restart recovery restart the bootstrap again
when the previous claim was a bootstrap restart and nothing reached
execution since that claim.
> - The benefit is that a deploy plus agent-directory contention no
longer ends the task as failed. No provider turn is replayed.
## Linked Issues or Issue Description
**What happened?**
On staging, native runs of a Codex agent on Daytona failed about 3 times
an hour with `native_restart_recovery_blocked` and error
`ambiguous_remote_recovery_evidence`. The same failure was seen on the
two days before. I read 12 failed runs. 11 of them have the same event
sequence:
1. The first executor prepares the remote workspace. A deploy restarts
the server.
2. Restart recovery waits for the old controller lease, then records
"Restarting an incomplete native runner bootstrap on the same heartbeat
run" (`bootstrap_incomplete`, controller generation 2).
3. The new executor starts the run again and records "Another run of
this agent is still using its files" (`agent_directory_operation_busy`).
4. About 15 seconds later, restart recovery records "Native restart
recovery blocked ambiguous or conflicting ownership"
(`ambiguous_remote_recovery_evidence`). The run fails.
The busy-directory deferral writes `phase: retryable_failure`,
`failureCode: agent_directory_operation_busy`, `recoveryMode:
bootstrap_retry`, no lease owner, and keeps the run `running`. Every
remote proof in restart recovery needs either `phase: observed` with a
lease owner, or a `failed` run with a specific saved failure code. So
none matched. With no checkpoint and no provider evidence, the
classifier returned `ambiguous_remote_recovery_evidence`. The other
result fields on the failed run (workspace restore failure,
fresh-continuation supersede) were written after this block. They are
consequences, not the cause.
**Expected behavior**
The deferral happens while the executor prepares, before it requests a
runner. The previous recovery claim already proved that the run reached
no provider work. If nothing reached execution after that claim,
recovery must restart the bootstrap again without manual Retry. If a
runner start, runner identity, provider event or checkpoint appears
after the claim, recovery must keep the run held and not replay provider
work.
**Steps to reproduce**
1. Start a native remote-sandbox run and stop the server before the
runner receives a command.
2. Let restart recovery claim the run as `bootstrap_incomplete`.
3. Make the agent directory lock busy when the new executor prepares
instructions, so it defers with `agent_directory_operation_busy`.
4. Let the scheduled restart recovery run at the retry time. The run is
blocked with `ambiguous_remote_recovery_evidence`.
The new integration test reproduces these steps with the real recovery
and deferral functions.
**Paperclip version or commit**
Staging Core `70be453c38`. Fix based on `e23549cd9c`.
**Deployment mode**
Hosted staging (cloud workspace), remote Daytona sandbox, frequent
auto-deploys (every 20 to 45 minutes).
**Agent adapter(s) involved**
`paperclip_runner` with the `codex_app_server` backend on a Daytona
sandbox.
## What Changed
- Restart recovery accepts a new remote proof,
`remote_bootstrap_restart_deferred_before_launch`. It applies when all
of these are true:
- The most recent recovery claim was `bootstrap_incomplete`, with a
recorded event watermark.
- The coordinator is `retryable_failure` with `failureCode:
agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no
lease owner, and no successor run. The run is still `running`.
- No `native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`, tool start or
runner-sourced event exists after the claim watermark. No process
identity was ever recorded for the run.
- No provider event exists. A checkpoint, if present, is only the
continued session with no turn, result or request.
- The workspace is a prepared remote workspace with no durable
controller checkpoint and no staged attachment.
- The run then goes through the same resumption admission as the other
settled remote retries (agent invokable, task not paused, dependencies
ready, no competing run).
- Added red/green integration tests built from the staging shape: the
run restarts again (generation 3), and a runner identity, a provider
event, or a checkpoint saved during the wait keeps it held.
## Verification
- Red: on the unchanged source, the new test returns `{ kind: "blocked",
reason: "ambiguous_remote_recovery_evidence" }`, the same error as
staging.
- Green: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-controller-recovery.integration.test.ts`
passes (251 passed, 14 skipped).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low risk. The change only adds a narrow proof. It requires that
nothing reached execution since a claim that already proved no provider
work. Any runner start, runner identity, provider event or new turn
keeps the run held, so a provider turn is never replayed.
- The restart counts against the existing provider attempt budget and
the no-progress deadline, so a loop of deferrals stays bounded.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser read-only API fetches) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Refs #20829
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
18fb6ee571 |
fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs that end in a terminal provider failure go through
automatic recovery: an exact replacement first, then a fresh
continuation from saved work
> - The exact replacement waits for proof that the previous provider
stopped. Without that proof it writes neither a successor nor a denial
> - That wait has no end. If the old sandbox or Runner can never be
proven stopped (for example after a server swap), the recovery resolver
keeps waiting for the replacement, and the task sits with no run and no
reason a person can see
> - This pull request bounds the wait. After 5 minutes on the database
clock from the run's terminal time, the replacement pass records a
denial, and the existing fallback continues the task in a fresh run in
an isolated workspace
> - The benefit is that these tasks continue by themselves instead of
stalling forever, and exact replacement still wins when proof arrives in
time
## Linked Issues or Issue Description
**What happened?**
A native run that failed with a terminal provider failure stayed stuck
when its previous provider could not be proven stopped. The replacement
pass skipped the run on every pass. It recorded no successor and no
denial, and it set no deadline. The recovery resolver waits for one of
those two records, so it never continued the task. The replacement queue
also did not wake again on its own, because a completed scan reports
nothing pending.
**Expected behavior**
Exact replacement gets a bounded first chance. After the bound, the task
continues in a fresh run that cannot use the old writer's workspace.
**Steps to reproduce**
1. Create a native run in `terminal_failure` with code
`native_provider_terminal_failed` and a finished time more than 5
minutes ago.
2. Record a process on the run that this server cannot prove stopped
(for example a PID recorded by another server).
3. Run the replacement pass and then the recovery resolver.
4. Before this change: nothing happens, on every pass. After this
change: the pass records a denial and the resolver records one fresh
continuation with isolation `fresh_isolated_workspace`.
**Paperclip version or commit**
`master` at
|
||
|
|
a090a6a932 |
fix(runner): stop a model from resending the same rejected tool call (#21360)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools such as `call_api`, and the tool
layer refuses an invalid call before it sends anything
> - Some model hosts drop optional tool arguments. A `PATCH` call then
arrives without `body` every time, even when the model says it added it
> - The refusal already names the missing argument, but nothing stops
the model from resending the same call. In one recorded run the agent
sent the same rejected call 12 times in 80 seconds and never finished
the task
> - This pull request makes the tool layer recognise a repeated
identical validation failure, correct it once with the exact arguments
that arrived, and then stop running that exact call for the rest of the
run
> - The benefit is that a dropped argument ends in a clear correction or
a reported blocker, not a long loop that burns time and tokens
## Linked Issues or Issue Description
**What happened?**
An agent created a task and then tried to update it with `call_api`
`PATCH /api/issues/{id}`. Every call arrived with `operationId`,
`pathParams`, `query` and `responseText`, but no `body`. The tool layer
refused each call with "requires body" and showed the shape to resend.
The model's reasoning said that the body was missing, but each new call
arrived without `body` again. The agent sent the same call 12 times and
was still running when the test ended.
**Expected behavior**
After the same call fails twice in the same way, the agent should get
one clear correction that says what arrived and what is missing.
Paperclip should then refuse to run that exact call again, so the agent
changes the call or reports the blocker.
**Paperclip version or commit**
`master` at
|
||
|
|
5ede64f247 |
fix(recovery): settle an older run's saved-work hold, fold a waiting Retry into the isolated continuation, and plain recovery wording (#21348)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, a saved-work hold protects the files it
left. The fresh-continuation fallback then continues the task in a new
run, or asks a person to press Retry.
> - Staging found that an older run's saved-work hold made the dispatch
gate cancel every new run of the task. The hold was never settled, so
the task could not run again, and no person could act on the reason.
> - Recovery notices also showed internal codes and words such as
"board", "inspect", "repair" and "no live execution path". People cannot
act on these.
> - A person's Retry pressed while an isolated continuation was waiting
for delivery was saved, and then ran a second time after the
continuation.
> - This pull request settles an older run's hold through the fallback's
existing machinery, uses plain wording in every remaining recovery
notice (server and UI together), and folds a waiting Retry into the
isolated continuation that answers it.
> - The benefit is that a task held by an older run continues by itself,
or on one Retry, in a fresh isolated workspace. People read only what
happened and the one thing they can do, and a Retry never starts a
redundant second run.
## Linked Issues or Issue Description
**What happened?**
1. An earlier run's saved-work hold (for example "Automatic recovery of
saved work failed … Retry to try again") made the dispatch gate cancel
every new run of the task with `execution_reconciliation_required`. The
fresh-continuation fallback only took holds of the task's latest run.
Each refused run looked like a newer run, so the hold was never settled.
2. Recovery notices and recovery-task comments showed internal codes,
raw IDs, "board", "inspect", "repair" and "no live execution path". The
hold for remote files that could not be restored asked a person to
"verify its exact stop receipt" and to "record workspaceRepairEvidence".
3. On a projectless task whose old legacy writer might still be alive, a
person's Retry pressed after the isolated continuation was recorded, but
before it was delivered, stayed deferred. When the continuation run
ended, task release promoted that Retry into a second, sequential run.
Concurrent writers were already impossible on this path, because the
Retry waited behind the continuation. The second run was redundant: the
continuation was the answer to the person's Retry.
**Expected behavior**
An older run's hold does not block new runs indefinitely. The task
continues in an isolated workspace, by itself while the interruption is
recent, or on a person's Retry. Notices say what happened and the one
thing to do. A person's Retry answered by the continuation never starts
another run.
**Steps to reproduce**
1. Leave a hold from an older run whose saved-work recovery gave up,
then start a new run of the task. Before this change, the run is
cancelled at dispatch every time and the hold stays.
2. Open a task whose run was cut off and its retries exhausted. Before
this change, the notice reads "No live execution path … visible for
intervention".
3. On a projectless task with an isolated continuation recorded, press
Retry before delivery, then let the continuation end. Before this
change, a second run follows it.
## What Changed
- Older-run holds (`fresh-continuation-fallback.ts`,
`modules/run-dispatch/adapters/postgres.ts`):
- A run that was cancelled at dispatch (status `cancelled`, never
started, `execution_reconciliation_required`) never ran and follows no
run. The held-run listing and the fallback's successor check now ignore
it.
- The older run's hold goes through the fallback's existing machinery. A
saved-work recovery that is still in progress keeps its bounded wait.
Holds that a person owns stay. Past the bound, the hold is settled and
the task continues in a new isolated workspace. Held files are kept,
recorded as `kept_not_applied`, and never applied.
- One recency rule, `interruptionEligibility` (the #21222 rule), judges
recency by the newest run that the hold refused at dispatch. It no
longer uses the end time of the hold's own run. If the last attempt is
older than 24 hours, no automatic continuation starts. A person's Retry
always continues the task.
- After the dispatch gate refuses a run for another run's hold, it hands
that hold to the fallback in the same transaction. The normal dispatch
path is not changed.
- A person's Retry and the isolated continuation (`heartbeat/queue.ts`,
`execution-recovery-resolution.ts`, new
`fresh-continuation-retry-fold.ts`):
- The wake-admission transaction that inserts an isolated continuation
run (context `freshContinuationIsolation`) also coalesces the person's
waiting `task_retry` wake for that task, under the task row lock. This
applies only to a Retry with no message attached. The wake gets `status:
coalesced` and `runId` set to the continuation. A server stop after the
commit cannot leave the Retry unfolded.
- The delivery pass is idempotent about this. An action whose own
continuation is already queued is no longer hidden from the pass. When
the pass finds that continuation, it marks the action delivered and
folds any Retry that is still waiting.
- Before this change, the fold covered only a Retry that was already
waiting when the continuation was recorded. A Retry that carries a
person's message is not folded. Only a Retry whose source run is the
continuation's own source run is folded, and only while the continuation
is queued or running. A Retry for the continuation's own failure gets
its own run.
- Plain wording (server and UI together):
- "No live execution path" is now "Task could not continue". One seed
(`buildNoRunningExecutionNoticeSeed`) builds it for every producer.
- Exhausted saved-work recovery: "Paperclip could not recover the
agent's saved files … Press Retry to try again".
- The non-retryable continuation notice no longer shows the error code.
Recovery-task comments no longer show `stranded_issue_recovery`, raw IDs
or retry reason codes.
- `legacy-execution-recovery.ts` uses plain text for remote files that
could not be restored.
- `IssueRecoveryActionCard`: "Board decision required" is now "Waiting
for you".
- The UI still recognizes notices saved with the old text
(`adjacent-errors.ts`, `system-notice-humanizer.ts`).
- The precise codes stay in the evidence and in the notice details.
- Merge with current master: `recovery/service.ts` keeps master's
three-way choice (retry exhausted / continuation refused /
non-retryable). The non-retryable branch now uses the plain notice
without the code.
- Code comments that the original commits added are removed
(CONTRIBUTING).
## Verification
Red-to-green tests:
- `server/src/services/fresh-continuation-fallback.test.ts`:
- "continues a task held by an older run's saved-work hold in a new
isolated workspace…"
- "keeps the wait of a saved-work recovery that is still advancing…"
- "keeps a hold a person owns blocking…"
- "leaves an old hold for a person's Retry, and that Retry runs
isolated" (refused run and waiting Retry)
- "continues a recent restart-stopped run whose prepared remote
workspace could not be restored…" (red on the wording)
- "leaves an older restart-stopped remote restore failure for a person's
Retry, which then runs isolated"
- Same file, "consumes a person's Retry exactly once when the server
stops right after admitting the isolated continuation": the test runs
the admission and nothing after it. The Retry is already coalesced. The
test then resets it to unfolded, as an older release would leave it, and
runs the next delivery pass. The result is exactly one run and the Retry
consumed. Red without the in-transaction fold, and red without the
delivery-pass fold.
- Same file, "never folds a Retry for the admitted continuation's own
failure into that ended continuation": the continuation ends (real
`cancelRun`), then the person presses Retry on it. The Retry gets its
own run. Red without the source-run and liveness guard.
- Same file, "runs a person's Retry pressed before delivery as the
recorded isolated continuation, never in the shared directory": a
projectless task, an old legacy writer that may still be alive, and an
isolated continuation pending. Delivery starts exactly one run, which
carries `freshContinuationIsolation` in a `fresh_isolated_workspace`.
Extended here: after the continuation run ends (real `cancelRun` release
path, then `resumeQueuedRuns` and delivery), no run comes from the
person's Retry. The Retry wake stays `coalesced` into the continuation.
Red without the delivery fold (the Retry became a second run), green
with it.
Local runs on this branch (based on master
|
||
|
|
0ec155ac31 |
fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, for example when a server swap reaches a sandbox late, the fresh-continuation fallback records one continuation of the task from its saved work > - The execution-continuation worker turns that record into a queued run. It took one record per agent per pass, and it took the oldest held record first > - After a large swap, the backlog of old records is hours deep. A record for a run that ended a few minutes ago waited behind the whole backlog. The record also stayed outside the queue while the agent's queue was deeper than its concurrency limit > - Meanwhile the task showed "todo" with nothing running, and the assigned-work sweep's wakes for it were folded into the pending record without any trace > - This pull request queues a fresh interruption's continuation on the next pass, keeps a share of every pass for the old backlog, and records a skipped wake row when a system wake is folded into a pending continuation > - The benefit is that an interrupted task gets its next run promptly, it waits in the agent's queue where the dispatcher ranks it, and nothing is lost or invisible ## Linked Issues or Issue Description Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that this builds on). **What happened?** On a hosted staging instance, five tasks had runs that ended after a server swap ("The runner restored the conversation after process loss, but the previous turn is no longer active."). The fallback recorded a continuation for each task 3 to 6 minutes after the run ended. Each record stayed `continuationDelivery: pending` for 30 to 110 minutes with no delivery attempt. During that time the worker delivered continuations whose source runs ended 4 to 7 hours earlier, one per agent per pass. The tasks read as failed and needing Retry. The sweep tried 3 to 4 times per task, and each wake returned `null` without a wake row. **Expected behavior** A recorded continuation for a recent interruption becomes a queued run within about one delivery pass. Old backlog continues to drain and is not starved. A wake that is not started records why. **Steps to reproduce** 1. Give an agent a queue that is deeper than its concurrency limit. 2. Record pending continuations for older interruptions that are held past the hold bound. 3. Record a continuation for a run that ended a few minutes ago. 4. Run `deliverReconciledExecutions`. Before this change the fresh record is not delivered. The oldest held record is delivered first, one per agent per pass. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `deliverReconciledExecutions` now sorts pending continuations into two classes by when the source run ended. "Fresh" means that the source run ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same value the dispatcher uses to tell fresh recovery from backlog. - A fresh continuation is enqueued without the queue-depth room check. It is limited only by the per-agent limit (one fresh continuation per agent per pass) and the per-company cap per pass. Among fresh records, the newest interruption goes first. A record whose wake was refused moves behind records that were not attempted yet. - Backlog continuations keep their earlier rules. These are the queue-depth room check, the bypass after the 5-minute hold bound, oldest held first, and one per agent per pass. - Each pass runs three phases: fresh up to the company cap minus `RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to the cap, then fresh again up to the cap. When both classes wait, the backlog gets at least 1 of the 5 slots per company per pass. The per-company cap (5) does not change, so a swap that interrupts many runs does not enqueue them all at once. All admissions still go through the one `wake(...)` call. - In `enqueueWakeup`, a system wake that is folded into the task's pending fresh continuation now writes a `skipped` wake row with reason `issue_continuation_pending`. Before this change it returned `null` with no trace. - Tests in `recovery-wake-herd.test.ts`: - New: a fresh continuation is queued on the first pass while the agent's queue is deeper than its limit. - New: a fresh continuation is delivered ahead of an hours-old backlog for the same agent, and the backlog keeps draining. - New: many fresh continuations for one agent are enqueued one per pass, newest first, and none is left behind. - New: the backlog keeps a share of each pass while fresh continuations fill the company cap. - New: a refused fresh wake stays pending and is enqueued on the next pass. - New: a folded system wake leaves a skipped row. - Updated: existing tests that used a recent source run to mean "held because the queue is full" now use backlog-age sources, and assertions that depended on exactly one admission per agent per pass now check the record under test. ## Verification - Red before the fix: the 6 new tests failed. The 18 earlier tests still passed. - Green after the fix: `cd server && npx vitest run src/__tests__/recovery-wake-herd.test.ts` gives 24 passed. - `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27 passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed. `src/services/fresh-continuation-fallback.test.ts` gives 62 passed. `src/services/workspace-recovery-fallback.test.ts` gives 19 passed. `src/services/native-runtime/native-safe-replacement.test.ts` gives 62 passed. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. ## Risks - Behavior shift: an agent with a full queue now gets fresh recovery continuations queued at once. They wait in its queue, where the dispatcher ranking (person-initiated first, recovery share) decides when they start. They are no longer held outside the queue. The per-company cap per pass still limits the burst. - Old backlog drains at a similar pace as before. It always gets at least one slot per company per pass when it waits. - The skipped wake row adds one small row each time the sweep retries a task that has a pending continuation. The sweep spaces its retries, so the volume is low. - No migration. No API or UI contract change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, read-only staging API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6368cd80c9 |
Refresh SSH account support against current master
Merge master
|
||
|
|
e23549cd9c |
fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run dies (for example during a server swap), the
fresh-continuation fallback records that Paperclip will continue the
task in a fresh run from saved work
> - The record says `continuationDelivery: "pending"` until a delivery
pass starts the new run. Delivery is paced, so a record can wait a long
time behind an older backlog
> - The task view showed such a task as "working" only for 15 minutes
after the record was made. After that it showed a failed attempt with a
Retry button, although the record still said Paperclip would continue
the task
> - Product rule: a task that the system has recorded it will continue
by itself never shows as failed with a Retry button
> - This pull request ties the "continuing" state to the record's own
delivery state instead of a timer
> - The benefit is that the task view stays truthful however long
delivery takes, and Retry comes back as soon as the system gives up
## Linked Issues or Issue Description
Refs #21264, #21324, #21297, #21320, #21200
**What happened?**
On hosted staging (build
|
||
|
|
f0679ee0c7 |
fix(recovery): bound automatic attempts per task, one deadline per interruption, and batch the invariant report's reads (#21347)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.
## Linked Issues or Issue Description
**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.
**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.
**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.
## What Changed
- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.
## Verification
Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".
Local runs on this branch (based on master
|
||
|
|
ff971149c2 |
fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue decides which queued run an agent starts when a slot frees up > - Recent changes start person-initiated work first and give recovery runs one start in four > - A system continuation of a person's interrupted Retry ranked as old-backlog recovery, and the recovery share always went to fresher recovery runs > - With a steady stream of new work, that continuation never started, so the person's task sat queued for hours > - This pull request adds aging and a longest-waiting share, keeps a person's interrupted request person-first, and passes the turn on when a long-queued run cannot be claimed > - The benefit is that every ready queued run starts within a bounded number of starts, and a person's Retry is not lost behind system work ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, one agent had 28 runs running and kept starting newer runs. The company had 67 queued runs, and 54 of them were older than one hour. A person pressed Retry on a task. The Retry run was interrupted by a server restart two minutes later. The system queued an automatic continuation (`issue_continuation_needed`) with `previousRunId` set to the interrupted Retry run. That continuation stayed queued for more than 2.5 hours. The task showed as "working". Other queued runs of the same person (some queued for more than 10 hours) also did not start. Root cause: - The dispatcher ranked the continuation as `recovery_backlog`, the last tier, because it was a system wake and its source run ended more than 30 minutes ago. The person-first check only looked at wake requests attached to the queued run itself. It did not look at the source run, which was the person's `task_retry`. - The 1-in-4 recovery share picked the first recovery run in sort order. Fresh recovery runs sort before backlog runs, and fresh recovery runs kept arriving, so the backlog run never got the share. - Nothing aged a queued run. A run in a lower tier or with a lower priority could wait forever. **Expected behavior** A system continuation of a person's Retry ranks as person-initiated. Every ready queued run starts within a bounded number of starts, even when fresh person and system work keeps arriving. A run that keeps failing to claim is reported and does not block the next-oldest run. **Steps to reproduce** 1. Give an agent one free slot and several queued runs. 2. Queue a person's `task_retry` run, start it, and interrupt it. 3. Queue a system continuation with `previousRunId` set to the interrupted run, and let its source age past 30 minutes. 4. Keep adding fresh system and person work. 5. See that the continuation never starts. **Paperclip version or commit** Staging Core |
||
|
|
566843820c |
fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.
## Linked Issues or Issue Description
Refs #21250, #21257 (merged), #21258, #21234
**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.
**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.
**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.
## What Changed
- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.
### Old → new wording
| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
<remedy>, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "<what
happened>, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |
Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.
### Deliberately changed test expectations
Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.
## Verification
Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.
Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.
Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.
Local runs, on master after #21257 (
|
||
|
|
43aabb7080 |
fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can configure a secret that Paperclip delivers to an agent's commands as an environment variable > - The value must stay out of what the agent runtime stores and what it sends to the model > - With the OpenCode runtime, a command that printed the variable put the value in OpenCode's tool output. OpenCode stored that output in its session database and sent it to the model. Paperclip's own run log already redacted it > - Paperclip did not tell the runner which configured values are secrets, so the runner could not protect them without also hiding ordinary settings > - This pull request sends secret names (never values) to the runner, filters exact secret values before OpenCode captures any tool output, redacts them from model requests, and tells the agent how to check a secret without printing it > - The benefit is that a configured secret stays out of model context and out of runtime storage, even when an agent prints it ## Linked Issues or Issue Description **What happened?** The `chat-features.runner-opencode.local.secrets.personal-secret` eval failed because the persisted-state scan found a configured secret value in OpenCode's session database. The person entered the value through the normal secret entry form. Nobody pasted it in chat. The agent ran a shell command that printed the environment variable it was given. Paperclip's run log redacted the value. OpenCode stored the raw output in its database and included it in the next model request. **Expected behavior** A configured secret must never reach model context or runtime persistence. Ordinary configured values must not be redacted. **Steps to reproduce** Configure a secret environment variable for an OpenCode agent. Have the agent run a command that prints the variable. Then search the OpenCode session data and the model requests for the value. ## What Changed - **Secret names cross to the runner.** `configuredEnvironmentProjection` takes the controller's secret key names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the names of configured entries that came from secrets. It travels with the configured projection across the TypeScript and Rust runner boundaries (`configured-environment.ts`, `configured_environment.rs`). The server passes `secretKeys` in `runtime-execution.ts`. - **Command output is filtered before OpenCode reads it.** For a launch that has configured secrets, the runner sets OpenCode's `shell` to a small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper runs the shell OpenCode would use anyway. It sends stdout and stderr through an `awk` filter that replaces exact secret values. OpenCode never receives the raw bytes, so its streamed previews, its database and its model requests stay clean. The wrapper keeps the exit status. It does not wait for background processes. The values come from the environment that OpenCode already holds and are never written to disk. - **Other tool results are rewritten before they are saved.** A local OpenCode plugin rewrites completed non-shell results (file reads, searches, MCP results) in `tool.execute.after`. OpenCode runs that hook before it stores or returns the result. The wrapper refuses to run any command until the plugin has started for the current launch (per-launch nonce). So no command can write a secret to a file that a later read returns without filtering. - **Model requests are redacted as a second layer.** The OpenCode provider proxy removes exact secret values from the string values of each parsed model request. Field names and other literals are never changed. A quick byte scan for each value, raw and JSON-escaped, sends the original bytes unchanged when no value is present. - **The agent gets guidance.** Native task prompts list the configured secret names. They say to check a secret with `[ -n "$VAR" ]` and never to print, echo, log or write a secret value. - **No added startup delay.** The wrapper is content-addressed under the runtime directory and reused across launches, so it is not rewritten on every launch. OpenCode's plugin-dependency install is marked as already done, so loading the plugin does not add wait time to the first tool call. - **Only exact values are redacted.** Command output is filtered line by line, so each line of a multi-line value that has at least 4 characters is redacted too. If a configured secret has no line of 4 or more characters, the launch is refused with `opencode_output_custody_unavailable` instead of running unprotected. ## Verification - `packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts` (10 tests): the wrapper filters stdout, stderr and a partial last line; keeps exit codes (7, and 143 for SIGTERM); filters each line of a multi-line value; refuses to run before plugin readiness or with an old nonce; returns at once when a background process keeps running; leaves ordinary values unchanged. The plugin rewrites nested results. Model-request redaction covers raw and JSON-escaped values and never changes field names or literals (a secret equal to `true` leaves `"stream": true` intact). Secrets the filter cannot match are refused. No value is written to disk. - `opencode-server-driver.test.ts`: one saved session launches three times. First with no secret marker (older controller), then with the marker, then with no marker again (rollback). The first launch is unchanged, the second applies custody, and the third launch still starts. This test fails without the driver change. - `configured-environment.test.ts`: the marker names only secret entries, crosses the runner spawn and OpenCode boundaries, reads as "no secrets" when absent, and is refused when malformed. Rust: `secret_name_marker_crosses_only_with_its_projection`. - `server/src/__tests__/configured-secret-guidance.test.ts`: guidance content, and no guidance when there are no secrets. - Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1 node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model asks for a command that prints the secret and writes it to a file, then a read of that file. Unprotected baseline: the value is in the session database and in model requests (red). With custody: it is in neither, and the exit status is kept (green). - **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs 697 ms baseline; first model request after the prompt 678 ms vs 698 ms (no regression); first command 139 ms vs 191 ms; each later command +9 ms (36 vs 27 ms). Proxy cost per request when no secret is present (the normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3 ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8 ms. The scan runs only for launches that have configured secrets, and it adds nothing to OpenCode's own startup. - A narrow paid run of `chat-features.runner-opencode.local.secrets.personal-secret` on this branch is linked in a comment. ## Risks - **Previous-release story.** The marker is new and optional. A runner that receives no marker (older controller) launches exactly as before. An older runner drops the marker at its environment boundary. The change saves no new run or session state. OpenCode configuration is regenerated on every launch, so sessions saved before this change resume under custody (covered by the cutover test). Warm-process reuse keys include the marker, so a warm process from before the deploy is not reused for a run that has secrets. - **Commands go through the wrapper.** Output arrives line by line. Output written after the shell exits is dropped, which matches how OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing, the launch is refused with `opencode_output_custody_unavailable` and does not run unprotected. - **Scope.** Only exact values are redacted. An agent that encodes a value (for example base64) is out of scope. The model-request layer applies when Paperclip routes the model through its proxy (OpenRouter and Paperclip connections). ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0986c3eb3a |
fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) use a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - #20923 keeps an unreachable remote Runner alive when the failed attempt closes, so that the same-run reattachment can adopt it > - The same close also leaves the session in the in-process cleanup quarantine. The quarantine refuses every new session of the task, and that includes the reattachment that the Runner was kept for > - Close retries cannot settle it. After the first close fails, the Runner transport returns the same rejected close for every retry > - On staging, this was the largest failure class: 14 Codex runs in one hour failed with `native_session_cleanup_quarantined`, while only 5 runs succeeded. Each task then needed Retry > - This pull request lets the same run's reattachment to that Runner, in the same remote sandbox, take over its own retained cleanup > - The benefit is that the task continues on its own, with no spent attempt, no failed run and no Retry ## Linked Issues or Issue Description Refs #20923, #21031 **What happened?** A remote native run (Codex on Daytona) failed a turn with a recoverable error, for example `runnerd did not report the provider turn identity` or `native_finalization_missing: session returned no semantic result`. The close then logged `runner did not prove durable suspension before checkpoint` and `remote runner left alive unsettled for same-run reattachment`. Each later same-run reattachment was refused at admission: - When the close error was retryable, admission waited out the grace and failed with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. That spent a same-run attempt. After the attempt budget, the run failed. All close retries replayed the same failed close within milliseconds (`runner transport close requested: native session quarantined cleanup recovery`), so they could never settle the quarantine. - When the close error was `native_session_close_unrecoverable` (operator-required), the next attempt failed at once with `native_session_cleanup_quarantined: the previous session is still shutting down; Paperclip finishes the cleanup automatically and the saved work is kept.` That code is a permanent failure. "Paperclip finishes the cleanup automatically" did not happen in the process. Only a server restart (a deploy) removed the in-memory quarantine, and by then the run had failed and the task waited for Retry. Staging restarts every 20 to 45 minutes, so attempts often end mid-cleanup and meet this gate. **Expected behavior** The same run's reattachment adopts the Runner that its earlier attempt kept for it. It does not wait for a close that cannot settle, and it does not spend an attempt. The task continues without Retry. Other runs of the task and other sandboxes stay gated. **Steps to reproduce** 1. Start a remote native run, and let its turn fail with a recoverable transport error while the Runner cannot prove suspension. 2. The close keeps the Runner alive and quarantines the session. 3. The server schedules the same-run reattachment 30 s later. 4. The reattachment fails with `native_session_cleanup_quarantined`, and the task needs Retry. **Paperclip version or commit** Deployed Core `c730872974` on staging. Reproduced on `master` at `49f0030f06`. **Deployment mode** Hosted cloud (staging), Daytona remote sandboxes. **Agent adapter(s) involved** `paperclip_runner` native runtime with the Codex app server backend. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: `executeNativeSession` takes a new option `retainedRunnerReattachment`. With a remote cleanup scope, admission retires quarantine entries that belong to the same run in the same remote scope. Removal stops later close retries. A close attempt that is still in flight stays an admission owner until it settles, so the old and new controllers do not overlap. Other runs, other sandboxes and local backends keep the gate. - `server/src/services/native-runtime/native-session-executor.ts`: sets `retainedRunnerReattachment` only for a `reattach_remote_runner` recovery of the same run. - `doc/execution-semantics.md`: records the takeover. - Tests: - `packages/paperclip-runner/src/native-session-runtime.test.ts`: four new cases. An operator-required close with a retained Runner. A retryable close whose retries replay one failure. A close that a deploy interrupts while it is in flight (the reattachment waits for it, then takes over). Controls: another run, a run without the takeover option and a local backend stay refused. - `server/src/services/native-runtime/native-session-executor.test.ts`: asserts that the option is set for a same-run remote reattachment and is not set for a fresh sandbox run. ## Verification - Red on `master` (runtime source restored): 3 of the 4 new runtime cases fail with exactly the two staging errors (`the previous session is still shutting down…` and `prior session cleanup remains incomplete`). The control case passes. - Red on `master` (executor source restored): the 3 executor assertions fail. - Green with this change: - `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts` (294 passed) - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "exact Stop cleanup|fresh sandbox|retirement"` (46 passed) - `cd server && npx vitest run src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed) - `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes. - Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence above in their events and logs. ## Risks - The takeover applies only when the server recovers the same run with `reattach_remote_runner` in the same remote cleanup scope. This is the case that #20923 keeps the Runner alive for. After a deploy the quarantine is already gone and the same reattachment runs today, so the takeover gives the in-process retry the same behavior as a restart. - A close attempt that is in flight is never overlapped. The reattachment waits for it within the existing admission grace. - The reattachment still verifies Runner ownership and its controller generation before it adopts the Runner. If the Runner is gone, the existing relaunch or reconciliation path handles it. - No persisted shape changes. Saved state from the previous release is not affected because the quarantine lives only in process memory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
22c9300dac |
fix(recovery): a run that fails before the agent is asked anything continues by itself on a fresh computer (#21297)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox opens a provider session and then
waits at a launch gate before the first prompt goes to the provider
> - On remote OpenCode sandboxes the gate saves the provider launch
files; when that save fails, no prompt was ever sent
> - Today that failure ends as `native_session_interrupted` with
ambiguous recovery evidence, so the task goes blocked and the user must
press Retry
> - Nothing was sent to the provider, so nothing can be duplicated: the
system can safely continue by itself
> - This pull request types that failure, proves it from the Runner turn
ledger at the failure point, and continues the task on a fresh sandbox
in the existing interrupted-preparation lane
> - The benefit is that a setup failure before the first prompt no
longer needs a human, and the user sees plain words instead of internal
codes
## Linked Issues or Issue Description
Refs #21287
Refs #21304
**What happened?**
On a remote OpenCode sandbox, the provider launch-files gate failed
before the first prompt (for example
`native_provider_artifacts_transfer_timeout`). The run ended blocked
with a Retry banner. The run error showed internal codes and raw byte
counts to the user.
**Expected behavior**
A failure that happens before any prompt reaches the provider continues
automatically on a fresh sandbox. The task shows as working while the
continuation is pending. The visible text is plain and has no internal
codes or raw numbers.
**Steps to reproduce**
1. Use an OpenCode agent on a Daytona sandbox environment with
controller persistence.
2. Arm the crash point `native.provider_artifacts.before_capture` in
fail mode (or let the launch-file upload time out).
3. Start a task. The first run fails at the gate before any prompt.
4. On master the task goes blocked with Retry. With this change a
successor starts on a new sandbox and the task completes.
**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).
## What Changed
- New typed cause `native_failed_before_provider_turn`
(`server/src/services/native-runtime/failed-before-provider-turn.ts`).
The executor raises it only when the launch gate fails, the run was not
cancelled, and the Runner turn ledger proves that no turn command was
issued. The proof is taken at the failure point, before the session
closes. In all other cases the original error is rethrown and today's
path stays.
- The executor quarantines the failed run's session root, so the
successor starts in a new root.
- `heartbeat/predecessor-retirement-fallback.ts` accepts the new cause:
it records the proof on `runnerProfileJson.failedBeforeProviderTurn`,
marks the run's reusable sandbox leases for replacement, and schedules
the successor in the existing `interrupted_preparation` lane (same cap
of 12 and the same backoff).
- `remote-workspace-unchanged-proof.ts` accepts the runner process,
runner command and pre-turn events only when that proof record is
present. It still refuses any turn evidence, an unsettled save by
another run on the same workspace, and setup writes. When the proof
refuses, the run's epoch is abandoned unapplied: the files are kept and
recorded, and the next run restores the last committed save into a fresh
root.
- When the proof refuses and the epoch is abandoned, the replacement
mark stays. The next acquisition replaces the sandbox. #21304's
retention hold makes `retireReplacedReusableSandboxLease` park the old
lease as `retained` instead of `pending_cleanup`, so the task continues
on a fresh sandbox and the sandbox that holds the abandoned files is
kept. When the proof holds (only the Runner's own session folder was
written), the old sandbox is replaced and cleaned up as usual.
- `task-interruption-presentation.ts` shows a pending
`interrupted_preparation` retry as `working` with no Retry until it is
due (plus the dispatch grace).
- The visible run error is now plain: "Setting up the computer took too
long, so the task is continuing on a fresh one." (for a setup timeout)
or "The computer could not be set up before the agent started, so the
task is continuing on a fresh one." The technical cause code stays in
`runnerProfileJson.failedBeforeProviderTurn.technicalCause`, the server
log and the run log.
- New crash point `native.provider_artifacts.before_capture` (fail mode
throws the typed transfer timeout at the gate).
`tests/runner-e2e/source-sdk-public-inputs.json` is refreshed for the
changed registry file.
- `tests/restart-chaos/main.ts`: `RESTART_CHAOS_EXTERNAL_CRASH_CONTROL`
lets an outside driver arm fail-mode crash points without a restart
fault.
## Verification
Local (targeted):
- `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: 8/8
pass. New cases: fresh remote sandbox and fresh plugin-task sandbox
continuation (now also asserts the plain visible text with no
identifier-looking tokens, and the technical cause in the proof record),
the ambiguous hold when the ledger cannot prove no turn, and backoff
plus the lane cap.
- `server/src/services/remote-workspace-unchanged-release.test.ts`:
27/27 pass. New cases: hand-on when the ledger proves no turn; refusal
on a recorded turn, an unsettled predecessor save, and setup writes. Two
more cases follow the marked lease through the next acquisition's
replacement decision (`readReplaceBeforeReuseMarker` +
`reusableSandboxReplacementPlan` + `retireReplacedReusableSandboxLease`)
and then through `sweepPendingCleanupLeases`. "replaces the marked
sandbox when the failed run provably left its files unchanged": the old
lease goes to `pending_cleanup` and the sweep destroys it. "continues on
a fresh sandbox and parks the one that holds abandoned files, which
cleanup leaves alone": the old lease is `retained` with the
abandoned-files reason and is held by the receipt, the sweep does not
destroy it, and the file is still there.
- Red without the product change (product files reset to the base, tests
kept): 8 failed, 25 passed of 33. All 8 failures are the new cases. The
visible-text assertion alone was red on the previous commit: the error
read "The computer could not be prepared before the agent started
(native_provider_artifacts_transfer_timeout: the upload of 536870912
bytes did not finish within 120 s). The task continues on a fresh
computer."
- `task-interruption-presentation.test.ts`,
`native-provider-artifact-gate.test.ts`,
`crash-point-call-sites.test.ts`: 79/79 pass.
`packages/shared/src/crash-points.test.ts`: 16/16.
`tests/restart-chaos/crash-points.test.ts`: 12/12.
- `tsc --noEmit` clean for `server`, `packages/shared` and
`tests/restart-chaos`.
Daytona (local servers, Daytona sandboxes, image `sha256:787c3e9c…`,
these commits rebased onto
|
||
|
|
41998d551c |
feat(plugins): send the claimant's credential on plugin task resume (#21335)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugin environment drivers run agent tasks on remote computers, and a warm follow-up run resumes a parked Runner through the `resume` task operation > - `resume` carries only the durable identity and a bootstrap ticket, which is enough while the parked Runner process is still alive > - A provider sometimes has to start the Runner again to resume the session: after it moved the parked session to a new sandbox, or after the computer rebooted > - A Claude Runner cannot start without its credential, and today only the portable handoff path (`handoffCredential`) delivers one > - This pull request lets a provider opt in to receiving the claimant run's harness credential on `resume` > - The benefit is that a provider can continue a parked session on a new computer without Core orchestrating a handoff ## Linked Issues or Issue Description Refs #21289 Related: #21305 **What existing behavior does this improve?** Warm resume for plugin task providers that restart the Runner to resume a session. **Subsystem affected** Plugin SDK environment task schema, the server's plugin Runner task owner, and the plugin environment task docs. **Current behavior** `submit` can carry the harness credential (`credential`), but `resume` cannot. A provider that must start the Runner again on resume has no credential for a Claude task. **Proposed behavior** A provider that returns `resumeCredentialVersion: 1` from `describe` receives `credential` on `resume` whenever Core would send it on `submit`, resolved for the claimant run. Other providers see no change. **Reason and benefit** Providers can restart a Claude Runner on resume without the Core-run handoff protocol. **Breaking changes** None. The field is opt-in: the SDK worker parses operations strictly, so Core sends it only to providers that advertise the capability. ## What Changed - `resume` gains an optional `credential`, with the same schema and validation as `submit`'s. - `describe` gains an optional `resumeCredentialVersion: 1` capability. - The plugin Runner task owner adds the credential on `resume` when the provider advertised the capability and Core resolved a credential, using the same resolution as `submit` (`preparePluginRunnerCredentials`). After a warm rebind, the claimant run's credential is sent. - `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` documents the field and its handling rules. - The runner-e2e SDK public-input pin is updated. `handoffCredential` on the session handoff path is unchanged. ## Verification - SDK vitest `tests/environment-task-release-resume.test.ts` and `tests/environment-tasks.test.ts` (68 passed): `resume` accepts and rejects the same 11 credential cases as `submit`; the capability is optional and rejects other versions; a resume credential reaches the plugin through the serialized worker RPC. - Server vitest `plugin-runner-task.test.ts` and `environment-task-runtime.test.ts` (151 passed): credential sent with the capability and a resolved credential; omitted without a credential or without the capability; the rebound claimant's credential is sent; the credential appears in no metadata write and not in the sanitized error, even when the worker error echoes it. - `native-session-executor.test.ts -t "uses a task-only"` (5 passed): the executor forwards the capability to the owner. - runner-e2e `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` (46 passed). - `tsc --noEmit` for `packages/plugins/sdk` and `server`. - Not run locally: the full suite and `pnpm verify`; required CI covers them. ## Risks - No server-side provider check on resume, unlike `submit`: the lease has no reliable harness at resume time. Protection relies on the owner, whose rebind refuses a changed harness, account or credential identity. - The credential is the claimant run's current one, which can be newer than the one that started the parked Runner. That is intended. - A provider must advertise the capability only after its SDK pin includes this schema; roll Core out first. - Rollback: revert this pull request. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs), implemented by a sub-agent and reviewed before opening. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e063587edd |
fix(recovery): repair stale pending recovery states and tell the fresh run what the cut-off run already did (#21257)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work invariant and its periodic sweep find open tasks that nothing will continue, and continue them once. > - The fresh-continuation fallback continues a cut-off run in a new run, with an opening note about what the cut-off run did. > - In release-swap waves, the invariant believed some "pending" states forever, so stalled tasks were reported OK and never repaired. > - The opening note did not say what the cut-off run had already created or read, so fresh runs created subtasks again and posted DONE twice. > - This pull request bounds the stale "pending" states and adds the cut-off run's work to the opening note. > - The benefit is that a stalled task gets repaired within minutes, and a fresh run does not repeat completed work. ## Linked Issues or Issue Description Refs #21135, #21203, #21234 **What happened?** 1. The invariant counted a task as pending without limit when its projection said "system recovery working", or when a wake was deferred behind a run that had already ended, or when a wake was claimed but no run was ever created. In release-swap waves, an undelivered continuation and follow-ups that were never promoted were reported OK for 17 minutes or more and not repaired. The suspect listing also excluded every task with a deferred wake, so the periodic sweep did not find them. 2. A fresh continuation's opening note listed only actions with unknown outcomes. Subtasks, comments and documents that the cut-off run had completed were not mentioned, and follow-ups it had already read were not mentioned. Continuations created the same subtasks again and posted the DONE comment twice. **Expected behavior** A stale pending state stops counting after a bounded time, and the sweep continues the task once. The fresh run knows what the cut-off run already created and read. **Steps to reproduce** 1. Run a release-swap wave where a run is cut off after it created subtasks and posted a comment, and where a follow-up waits behind it. 2. See the continuation create the subtasks again, or see the task stay "working" with no run while the invariant reports it OK. ## What Changed - Invariant bounds (`server/src/services/recovery/assigned-work-invariant.ts`, `server/src/services/recovery/assigned-work-sweep.ts`): - "System recovery working" counts as pending only for `SYSTEM_RECOVERY_WORKING_BOUND_MS` (17 min) after the task's last run ended: the fallback's progress bound (15 min) plus one disposition re-check. - A deferred wake with no live run counts as pending only for 5 minutes after it was created. A claimed wake with no run counts only for 5 minutes after it was claimed. - The suspect listing uses the same bounds, so the periodic sweep finds these tasks. - A held continuation (paced delivery from #21234) counts as the task's pending wake only for `PENDING_CONTINUATION_BOUND_MS` (7 min) after it was recorded: the pacing admits a held continuation within 5 minutes even when the queue is full (`RECOVERY_CONTINUATION_MAX_HOLD_MS`), plus 2 minutes of 15-second delivery passes. Inside the bound the task still shows "working" with no Retry, as #21234 intends. - The sweep's wake still goes through admission, which hands a held task to the fallback. It never starts a second writer. - A continuation whose delivery is stuck is repaired, not just no longer counted (`server/src/services/recovery-continuation-bounds.ts`, `execution-recovery-resolution.ts`, `fresh-continuation-fallback.ts`): - One module owns `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 min) and `PENDING_CONTINUATION_BOUND_MS` (that plus 2 min). The invariant, wake admission and delivery all read them from there. - Delivery re-drives first. A continuation held past 5 minutes was already admitted when the agent's queue was full, but the delivery page skipped every record of an agent that had another recovery run queued, at any age. That exclusion now applies only within the hold bound. - Admission never replaces the recorded continuation. The recorded continuation alone carries the isolation and fresh-session decision. A wake that `hasPendingFreshContinuation` coalesces into it now also notifies the delivery queue. Delivery then re-drives the same record, with the same isolation key, `forceFreshSession`, and one-successor idempotency key. ### Where the isolation decision is read, and every path that starts the successor Run preparation reads the decision in one place: `readFreshWorkspaceReadiness` (`server/src/services/fresh-workspace-continuation.ts`), called from `heartbeat/workspace-preparation.ts`. It reads `contextSnapshot.freshContinuationIsolation`, which names the record, and loads it through `readFreshContinuationReadiness`. Only a run that carries that key gets a new isolated workspace. | Path that can start the successor | How it keeps the isolation | Test | |---|---|---| | Delivery worker (`deliverReconciledExecutions`) | Sends the record's own wake with `freshContinuationIsolation` and `forceFreshSession` | `fresh-continuation-fallback.test.ts` "gives the continuation a new isolated workspace while the old local writer may be alive" | | Re-drive after the bound | The same delivery wake, admitted past the hold bound even when the queue is full or another recovery run is queued | "re-drives an isolated continuation pending past the bound itself; a repair wake never replaces it"; `recovery-wake-herd.test.ts` "re-drives a continuation held past the hold bound..." | | Sweep / monitor / timer wake | Never becomes the successor: coalesced into the record, which it re-drives | "records a pending continuation for an ordinary continuation wake and delivers it once"; the re-drive test above | | Saved input (a person's message) | Waits behind the isolated continuation and is promoted when that run is released (#21250) | `postgres.test.ts` "... (released, continuation pending)" (#21250) | | A person's Retry started before delivery | Delivery counts the Retry's run as the successor. The Retry run does not carry `freshContinuationIsolation`. On a task with a workspace binding, the fallback cleared the binding, so a new workspace is prepared. On a projectless task this path is **not yet covered** | "never starts a second run when a person's Retry already continued the task" (one successor only). Isolation on this path is a known gap, reported for follow-up | - Opening note (`server/src/services/fresh-continuation-fallback.ts`): - Names the subtasks, comments and document saves that the cut-off run created, and asks the agent not to create them again. The record keeps the list as `createdObjects`. - Lists the person's messages that the cut-off run had read (through `taskHistoryReadsForRun`, from #21203) before what that run then posted, so the agent answers a re-delivered follow-up once. Only the revision that was read counts as read. A message a person edited after the read is listed as changed ("treat them as new"). - The note is written when the continuation is recorded. Nothing is added before `turn.start`. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/services/recovery/assigned-work-invariant.test.ts` (unit) and `server/src/__tests__/assigned-work-invariant.test.ts` (sweep integration): stale "working", a stale deferred wake, and a claimed wake with no run become violations past their bound and stay pending inside it. - `server/src/services/fresh-continuation-fallback.test.ts`: "names the subtasks, comments and documents the cut-off run already created" and "names the follow-up the cut-off run had read and its own final comment". - `fresh-continuation-fallback.test.ts` "re-drives an isolated continuation pending past the bound itself; a repair wake never replaces it" (projectless task, writer possibly alive). It was red on |
||
|
|
7e7452e88f |
fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.
## Linked Issues or Issue Description
Refs #21314, #21272, #21161, #21023
**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).
**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.
**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.
**Paperclip version or commit**
master at
|
||
|
|
4fe34df748 |
Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task view shows a "Retry" notice when a task needs a person to start it again. > - The product rule is that a task that continues by itself never shows this notice, not even for a few seconds. > - A parent task that waits for its subtasks showed the notice for 3.4 seconds after its run succeeded. > - The cause is the queued-message grace in the task interruption presentation: it does not apply to `blocked` tasks. > - #21200 kept `blocked` out of that grace on purpose, so this pull request adds only the exact observed shape: a parent with open subtasks and a queued task message. > - The benefit is that a parent that waits for its subtasks no longer shows a false failure, and every other blocked task keeps its Retry. ## Linked Issues or Issue Description Refs #21264, #21200 (added the queued-message grace and kept `blocked` out of it) **What happened?** A Codex parent task ran across a two-server release swap. Its run waited for two subtasks and then succeeded with the disposition `blocked`. The task status became `blocked`. A user comment (a deferred wake, status `deferred_issue_execution`) had been queued 13 minutes earlier, while the run was active. For 3.4 seconds after the run finished, the task presentation was `state: "failed"`, `canRetry: true`. Then the release drain cancelled the wake as "Queued messages were already delivered to a completed run", and the presentation changed to `resolved`. The subtasks finished later. Their `issue_children_completed` wake continued the parent, and it completed without help. The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`. The `queuedDuringRun` check shows a succeeded run with user input queued before it finished as `working` for up to 2 minutes. It applies only to `todo`, `in_progress` and `in_review`. For a `blocked` task the presentation fell through to the last line: `failed` with `canRetry: true`. The workspace finalization deferral (`native_workspace_finalization_busy`) in the same window does not change the presentation. The presentation does not read it. **How this differs from what #21200 narrowed** #21200 first applied the grace to every status. Its commit `a3fae9bb67` removed `blocked` because input queued on a blocked task is not promoted by itself, and because a recovery that gives up hands its hold to the board and must end with Retry. This change does not add `blocked` back. It adds one narrow case, and all of these must be true: - The latest run **succeeded**. A failed, interrupted or given-up run is not affected. - The task is `blocked` **and has open subtasks of its own** (not `done` or `cancelled`). Their completion wakes the parent again (`issue_children_completed`). - The pending item is an ordinary **task comment** wake (`commentId` or `wakeCommentIds` in its payload), not a Retry request (`taskRetry.sourceRunId`). - The comment was queued before the run finished, the run finished less than 2 minutes ago, and saved-work recovery is not exhausted. These are the existing bounds. A blocked task that waits for a person or a board decision has no open subtasks, so it keeps Retry as before. **Expected behavior** After the run succeeds, the parent shows `working` with no Retry while its queued message is settled. Then it shows `resolved` (message already delivered) or `working` (message promoted to a new run). If nothing settles the message in 2 minutes, Retry shows. **Steps to reproduce** 1. Assign a parent task that creates subtasks and waits for them. 2. Post a comment on the parent while its run is active. 3. Let the run succeed with the disposition `blocked`. 4. Read the task presentation in the first second after the run finishes. It shows `failed` with Retry. **Paperclip version or commit** master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - The batched wait read also reads whether the wake carries a task comment (`queuedComment`). - One company-scoped read finds which `blocked` tasks with a succeeded latest run and a queued comment have open subtasks (`waitingOnOpenSubtasks`). It runs only for those tasks. - The `queuedDuringRun` grace also applies to such a task. The `todo`, `in_progress` and `in_review` behavior does not change. - `server/src/__tests__/task-interruption-automatic-path.test.ts`, new embedded-Postgres tests: - The exact observed state: a `blocked` parent with two open subtasks, a run that succeeded 1 second ago, a user comment wake queued 13 minutes earlier. Shows `working`, no Retry. - The same parent after the wake is cancelled as already delivered: `resolved`. - The same parent after the wake is promoted to a new run: `working` on the new run. - Guard: the same parent after the 2-minute grace: `failed` with Retry. - Guard: a `blocked` task with a queued comment and no subtasks, or only `done` or `cancelled` subtasks (waiting for a person or a board decision): `failed` with Retry, unchanged. - Guard: a parent with open subtasks whose queued wake carries no comment: `failed` with Retry, unchanged. - Guard: a parent with open subtasks and a queued comment whose run failed and whose recovery hold went to the board: `failed` with Retry. - Guard: a parent with open subtasks and a queued comment whose saved-work recovery is exhausted: `failed` with Retry. - A parent whose run succeeded with no queued message stays `resolved` when its subtasks are open, blocked or cancelled. - The existing #21200 assertion (a `blocked` task with queued input and no subtasks shows Retry) is kept unchanged. ## Verification - `cd server && npx vitest run src/__tests__/task-interruption-automatic-path.test.ts` - Red on master `bbafa78fe2`: "shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled" returns `state: "failed"`, `canRetry: true`. This is the value that the wave recorded. The other 15 tests pass on master. - Green with this change: 16 of 16. - A plain `blocked` inclusion fails 3 of the guards: the #21200 assertion, "still offers Retry on a blocked task with a queued message and no open subtasks" and "still offers Retry on a parent with open subtasks when its queued wake carries no message". This shows that the narrow condition is necessary. - `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`: green (116 tests with the file above). - `cd server && npx tsc --noEmit -p .`: 0 errors. - Not verified: no new chaos wave. The flash was seen once in a loaded release-swap wave. The server test reproduces its saved state. ## Risks - Low risk. The change is presentation only. It does not change wake promotion, dispatch or Retry. - A parent with open subtasks and a stranded queued comment now shows `working` for up to 2 minutes after its run succeeded, and then Retry. Before, it showed Retry at once. - One extra indexed read (`issues_company_parent_idx`) runs only for blocked tasks with a succeeded latest run and a queued comment. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |