mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
784bc06276d93c608ea7ee082e2f5cb5e8e8fe26
8166
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
784bc06276 |
perf(runner): reuse the sandbox's identical runnerd instead of uploading it on every cold boot (#21949)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote agent runs start `paperclip-runnerd` inside a sandbox, such as a Daytona sandbox > - Agent boot time is a measured product constraint, and cold boot on Daytona grew from about 11 s to 25–75 s in restart-chaos waves > - The span that grew is `runner.artifact.prepare`: the controller uploaded its 31 MB `paperclip-runnerd` to every new sandbox > - The controller uploads this file whenever `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` is set, even when the sandbox image already has the same bytes > - The upload time depends on the sandbox file-upload throughput. That throughput changed from about 1 s to 5–48 s for the same 31 MB, on the same image and with no runner change > - This pull request makes the controller use the image's `paperclip-runnerd` when its SHA-256 equals the configured artifact > - The benefit is a cold boot that does not depend on sandbox upload throughput: about 9–11 s instead of 33–45 s in the same hour ## Linked Issues or Issue Description No issue exists. This describes the bug, following `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** Cold `agent_boot` on Daytona became 2–7 times slower. In restart-chaos waves on master `c5218f5ec4`, the cold boot p50 was about 11 s for claude, about 10 s for codex and 15–25 s for OpenCode. On master `3452639a81` it was 25 s, then 55 s, then 70–78 s for claude and codex over five waves. OpenCode reached 238–265 s. The run spans show that the extra time for every adapter is in `runner.artifact.prepare` (codex example: 42.7 s, before it was 1.2 s). In this span the controller uploads its own 31 MB `paperclip-runnerd` to the new sandbox. `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` is set, so `mayUsePreinstalledRunnerArtifact` returned `false` and the controller never used the identical `paperclip-runnerd` in the sandbox image. The upload time follows the sandbox upload throughput, which changes over time. None of the 17 commits between the two masters changed this path. **Expected behavior** When the sandbox already has a `paperclip-runnerd` with the same SHA-256 as the configured artifact, the controller links it and does not upload it. Cold boot does not depend on upload throughput. **Steps to reproduce** 1. Set `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` to the `paperclip-runnerd` from the Daytona runner image. 2. Start a cold agent run in a new Daytona sandbox from that image. 3. Look at the run spans. `runner.artifact.prepare` holds a full upload of the runner (5–48 s in our waves), and there is no `runner.artifact.discover` span. **Paperclip version or commit** `3452639a81` (also current master `6205b24d54`). Image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:710deab7375a…`. **Agent adapter(s) involved** All remote adapters: claude, codex and OpenCode. ## What Changed - `mayUsePreinstalledRunnerArtifact` accepts a preinstalled `paperclip-runnerd` when an artifact is configured and the controller has that artifact on disk. - In that case the existing SHA-256 check (`runner_remote_preinstalled_artifact_mismatch`) is mandatory. A different, missing, or unreadable image copy still falls back to the upload. - When an artifact is configured, a failed discovery probe counts as "no preinstalled runnerd", so the controller uploads the configured file as before. - Tests: a unit case for the gate, and three cases in the runner wiring test: identical configured artifact (no `syncIn`), mismatched configured artifact (upload), and failed discovery with a configured artifact (upload). - `docs/deploy/environment-variables.md` describes the new behavior. ## Verification - Red/green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "server-owned replacement artifact|contract-compatible runnerd|configured artifact the controller holds"`. Without the fix 3 tests fail. With the fix 27 pass. - `pnpm verify` passes. - Restart-chaos waves on Daytona, image `710deab7375a`, 6 tasks (2 claude, 2 codex, 2 opencode), A1/K0, run back to back in one hour: | Wave | Fix | claude cold boot p50 | codex | opencode | `runner.artifact.prepare` | | --- | --- | --- | --- | --- | --- | | w-20261117T005138 | no | 38.4 s | 33.2 s | 36.0 s | not recorded | | w-20261117T005627 | yes | 9.5 s | 8.8 s | 10.6 s | 0.5–0.6 s, all runs use the image copy | | w-20261117T005956 | no | 44.1 s | 13.1 s (max 47.1 s) | 45.5 s | 4.9–37.8 s upload | All 18 tasks passed. ## Risks - Low risk. The controller uses the image copy only when its SHA-256 equals the configured artifact, so the runner bytes do not change. Every other case keeps the current upload. - One extra `command -v` probe and one `sha256sum` probe run when an artifact is configured. Each takes about 0.1 s. - Saved run state is not changed. A run saved by the previous release resumes with the same runner identity, because the bytes are the same. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1117.0-canary.816600 |
||
|
|
d258f478fc |
test(ci): stop the live-run census hanging server shards, and name any stalled test file (#21941)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI splits the server tests across 12 shards; `ci / verify` fails when any shard fails > - Since 2026-11-16 the server shards were cancelled at their 20-minute job limit with no output (shard 2/12 five times, plus shards 1, 3 and 9), which blocked unrelated pull requests until reruns > - Turbo prints a task's output only when it ends, so a cancelled shard never names the stuck file > - Recomputing each cancelled shard's file list at its own merge commit put `issue-recovery-live-run-agent-census.test.ts` in 12 of 14 of them; it is also the slowest file in passing shards (235-300 s) > - That census started 9-14 real agent runs per file while `setTimeout` was faked, waited on them with no bound, and had 30-minute test timeouts, so a stuck wait could never fail inside the job > - This pull request stops the census from starting real runs, bounds every wait, and adds a CI reporter that fails a run naming any file that stops making progress > - The benefit is that this file cannot hang a shard, and the next stuck file fails in minutes with its name and step in the log ## Linked Issues or Issue Description Refs #21867 (the same fix for the sibling agent-row census), #21831 (the behavior this census guards) **What happened?** `ci / General tests (server (2/12))` was cancelled at 20m16s in actions runs 43865258195, 43926799428, 43932238377, 43934278368, 43944290677 and 43935722497 (both attempts), and other server shards hit the same limit (for example 43901003914 shard 1, 43945768035 shard 3, 43845343831 shard 9). The hung jobs show CPU near 0 for the last 9 minutes and printed nothing after `Running //#test:run:general`. Shard membership changes with every merge commit. To compare the shards, I recomputed each job's file list at its exact merge commit with `GITHUB_WORKFLOW=PR` (the PR workflow removes the native-runner suites from these shards, so a local dry run without it gives a different list). `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` was in 12 of the 14 hung shards. Why that file: - During its 10-minute fake-timer warm-up, the "live" task was `in_progress` with no run. That is real assigned work, so recovery woke its agent and a real `process`-adapter run started while `setTimeout` was faked. The run lost its lease ("Legacy controller lease lost"), recovery retried, and a local run of the file showed 13 real run executions. - No wait in the file had a bound, and each test had a 30-minute timeout, so a delivery pass waiting on a faked timer or a run holding a lock never failed inside the 20-minute job. - Under local CPU contention, 3 of 4 copies of the old file failed the census assertion (39 settled checks, caused by the stray runs), and each copy took about 270 s with 9-14 real runs. Fixing the runs exposed a second problem in the fixture: the seeded tasks were stamped "now". The first 5-minute backstop after the live run's first hint re-checks rows changed within 5 minutes before the last full pass, so it re-checked all 860 seeded tasks. The old file passed only because the stray runs happened to trigger that backstop during the warm-up. **Expected behavior** The census never starts a real run, every wait has a bound that names what it waited for, and a CI test file that stops making progress fails the job quickly with its name. ## What Changed - `issue-recovery-live-run-agent-census.test.ts` - The live task is a settled review until its run starts; `startLiveRun` now writes the checkout (`in_progress` + `executionRunId`) the way execution start does. - Population agents and the held agent take no on-demand wakes (`wakeOnDemand: false`), so a recovery check cannot dispatch a run. The census counts queue checks, so this does not change what it measures. - The seeded population is backdated 2 hours ("settled long before the window", as the file's comment already said). - Each step (seed, worker start, run start, run-event tick, fake-timer advance, pass settlement, direct writes) has a real 20-second bound captured before the timers are faked, with a message naming the step. Test timeout is 4 minutes instead of 30. - New `scripts/ci/stalled-file-reporter.mjs`, enabled in GitHub Actions from the root `vitest.config.ts`: when a running file starts no test or hook and reports no result for 8 minutes (`PAPERCLIP_VITEST_FILE_STALL_MS` overrides), it prints a `::error file=...` annotation naming the file and the step it was in, cancels the run and exits 1. The slowest healthy server file runs about 5 minutes in total, so 8 minutes without any progress means a stuck file. Tests: `scripts/ci/stalled-file-reporter.test.mjs` (added to `test:release-registry`). ## Verification - Census, red: the old file runs 13 real agent runs locally (`heartbeat execution setup failed`, `LegacyControllerLeaseLostError`); with 4 copies under CPU contention, 3 failed with 39 settled checks at about 270 s each. - Census, green: 4/4 tests pass, 0 real runs, about 80 s locally. - Census still guards #21831: with the `ag.status <> 'running'` line removed from `changed-tasks.ts`, the first test fails with 534 settled checks and 540 backstop checks, the numbers #21831 recorded. - Reporter, red/green: the real-Vitest case (a file awaiting a promise that never settles, with a 10-minute test timeout) is still running when the 20-second test bound ends without the reporter; with the reporter it exits 1 in about 8 s and prints the annotation naming the file and the test. - Reporter on a real server file: `PAPERCLIP_VITEST_FILE_STALL_MS=3000` on `issue-recovery-recorded-decision-idle.test.ts` prints the annotation naming the file and step and exits 1. - `node --test scripts/ci/stalled-file-reporter.test.mjs`: 5/5 pass. ## Risks - Test and CI-tooling change only. No product code changed. - The census's agents no longer take on-demand wakes. The census measures recovery queue checks, not dispatch, and still fails without the #21831 fix. - The reporter stops the whole CI run when one file stalls for 8 minutes. A legitimate single test or hook that runs more than 8 minutes with no other progress in its file would now fail; none exists today (the slowest whole file is about 5 minutes). The limit is configurable. - Other fake-timer recovery files still have long test timeouts and unbounded pass waits (for example `issue-recovery-activity-hints`, 30 minutes). The reporter now caps any of them at 8 minutes without progress and names the file; bounding them is follow-up work. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d8a5db36e6 |
fix(heartbeat): name the blocking database sessions when reattached runs are never reached (#21946)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a server restart, the new server reattaches to the live runs
of the old server (the reattach phase)
> - In a restart-chaos release swap with a hard kill, six runs
authenticated in under 10 s but no run was reached
> - The whole API then stopped answering for about 480 s: plain GETs
timed out, CPU was idle, and no run events were saved
> - The process identity write of each run started and did not commit,
so the reattach phase ended at its 180 s cap with 0 of 6 reached
> - The wave kept no `pg_stat_activity` or `pg_locks` data, so the head
of the lock chain cannot be proven from the evidence
> - This pull request records that data on the server at the moment the
stall is detected
> - The benefit is that the next occurrence names the blocking session
and its query, so the real fix targets one proven holder
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos K15 release swap, the outgoing server got SIGKILL.
The incoming server reattached six remote runs after their leases
expired. All six runs authenticated. None was reached. The reattach
phase ended at the cap. The server did not answer any API request from
about 22:35:24Z to 22:43:23Z. Then all buffered events arrived in the
same second. The task finished without help, but the runs were silent
for about 8 minutes.
**Expected behavior**
The runs are reached within seconds, as in the hand-off swap of the same
wave (6 of 6 reached in 13 s). If a stall occurs, the server log names
what holds the database.
**Steps to reproduce**
Run a release swap that kills the outgoing server with SIGKILL while six
remote runs are live. Wait for their controller leases to expire.
**Paperclip version or commit**
master at
canary/v2026.1117.0-canary.816400
|
||
|
|
6205b24d54 |
fix(ci): review the instance settings cache reset in the server test setup (#21943)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - CI selects only the server tests that a pull request can affect. The Vitest selection graph trusts a setup file only while its content matches a reviewed hash, and it follows every local import of that setup > - #21768 added a `beforeEach` to `server/src/__tests__/setup-supertest.ts` that resets the instance settings cache, through `../services/instance-settings-cache.js` > - The reviewed hash was not updated, and the cache module was not reviewed > - So server test selection fails with "Opaque setup dependency", and CI runs every server test instead. The qualification test also fails on every pull request that changes CI files > - This pull request reviews the new setup content and the cache reset > - The benefit is that server test selection works again, and pull requests that change CI files pass `qualify_test_selection` ## Linked Issues or Issue Description Refs #21768. **What happened?** `qualify_test_selection` ("Incremental CI qualification") fails on pull requests that change CI files, for example `perf/canary-git-numbering` and `fix/server-shard2-hang`: ``` ✖ real graph retains direct and indirect filesystem, process and opaque dependencies Error: Opaque setup dependency: .../server/src/__tests__/setup-supertest.ts ``` With only the hash updated, selection of the real server project stops at `Opaque setup package: drizzle-orm`, which `packages/db/src/work-signals.ts` imports through the cache module. **Expected behavior** The reviewed setup permits file selection again. A change to the setup, its helpers or the cache module still selects every server test. **Steps to reproduce** 1. On `master`, run `node --test scripts/ci/test-selection.integration.test.mjs`. 2. The real graph test fails with "Opaque setup dependency". ## What Changed - `scripts/ci/vitest-selection-graph.mjs`: - Pin the current `setup-supertest.ts` content. - Add `server/src/services/instance-settings-cache.ts` as a reviewed setup dependency. The setup calls only `invalidateInstanceSettingsCache`, which increments an in-memory generation. The module performs no IO when it loads. Its `drizzle-orm`, `drizzle-orm/pg-core` and `@paperclipai/db/work-signals` imports serve the read-through path, which the setup does not call. The review lists them as external, like `supertest/lib/test.js` for the setup itself. - `scripts/ci/test-selection.integration.test.mjs`: the fixture copies the cache module with the reviewed setup files. It also checks that a change to the cache module selects every test. ## Verification - `node --test scripts/ci/test-selection*.test.mjs scripts/ci/vitest-impact-report.integration.test.mjs` (the `qualify_test_selection` step): 42 passed. On `master`, the real graph case fails. - The "Qualify selection against the real Paperclip workspace" script from `ci-incremental-qualification.yml`, run locally with `NODE_ENV=test VITEST=true`: passes. - Real server project selection (`selectProjectTests` with the `@paperclipai/server` project): a change to `server/src/services/heartbeat-run-summary.ts` selects 2004 of 2188 tests. A change to `instance-settings-cache.ts` sets `setupChanged` and selects all 2188. ## Risks - A later edit to `instance-settings-cache.ts` or `setup-supertest.ts` disables the review again until the hash is renewed. Selection then fails closed and runs every server test, as it does today. - Changes to `packages/db/src/work-signals.ts` no longer select every server test through the setup. The setup's cache reset does not call it. Tests that import it directly are still selected. - CI scripts and tests only. No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1117.0-canary.816300 |
||
|
|
5e53fec10c |
feat(adapter-claude): add Claude Opus 4.5 to the static model fallback (#12379)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each adapter supplies model choices for agent configuration.
> - The Claude adapter combines live discovery with a static fallback.
> - The fallback lacks Claude Opus 4.5.
> - This PR adds that model and a regression assertion.
> - Users can select it when live discovery is unavailable.
## Linked Issues or Issue Description
**What happened?**
The Claude adapter model selector omits Opus 4.5 when live discovery is
unavailable.
**Expected behavior**
The static fallback should include claude-opus-4-5.
**Steps to reproduce**
1. Use the claude_local adapter without working live model discovery.
2. Open the agent configuration model selector.
3. Look for Claude Opus 4.5.
Related model-list work: #10280 and #10327.
Searched open and closed PRs for Opus 4.5; no competing title match was
found.
## What Changed
- Add claude-opus-4-5 after Opus 4.6 and before Sonnet 5 in the static
fallback.
- Assert that the fallback returned by the server includes this model.
- Preserve the existing first model, claude-fable-5-1.
## Verification
- At head
|
||
|
|
537b0408b3 |
fix(connections): retire satisfied connection cards on waiting tasks (#21940)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent needs a service that is not connected, it raises a "Connect" card for the responsible person > - #15856 added a sweep that retires such a card when the connection is usable again, and wakes the agent to continue > - The sweep only checks tasks that are in progress or in review > - A task that is blocked or not yet started keeps its card forever, even after the person reconnects the service > - The card then asks the person to connect a service that is already connected > - This pull request retires those cards too, without waking the agent > - The benefit is that a reconnect clears every card that it satisfies, and people do not see stale connection requests ## Linked Issues or Issue Description **What happened?** A shared Sentry connection lost its authorization. Over three days, agents raised about 40 "Connect Sentry" cards on different tasks. One task's run failed after it raised its card, and the task became blocked. When the person reconnected Sentry, the sweep retired 21 cards on in-progress and in-review tasks. Twelve cards on blocked and todo tasks stayed pending, and they kept asking the person to connect Sentry. **Expected behavior** When the requesting agent can use the connection again, its card is retired on every task that is not done or cancelled. A blocked or unstarted task does not wake because of this. **Steps to reproduce** 1. Have an agent raise a tool connection card on a task. 2. Move the task to `blocked` (for example, its run fails and recovery blocks it). 3. Make the connection usable for the agent. 4. The sweep does not retire the card. It stays pending. Related: #15856 added the sweep. #21355 adds AI repair cards to the same sweep. It does not change which task statuses the sweep checks. ## What Changed - `server/src/services/satisfied-connection-intents.ts`: - The sweep selects pending cards on all tasks that are not `done` or `cancelled`. Before, it selected only `in_progress` and `in_review` tasks. - On `in_progress` and `in_review` tasks, nothing changes. The card is retired and one continuation is saved. - On `backlog`, `todo` and `blocked` tasks, the sweep uses the same access checks (identity, membership, grants, installs, tool policy). It records the same system outcome and audit event. It does not save a continuation and does not notify the delivery worker. The delivery gate wakes only `in_progress` and `in_review` tasks, so a continuation for these tasks could never settle and would keep the worker busy. - `server/src/__tests__/satisfied-connection-intents.test.ts`: two new cases, described below. ## Verification - New tests in `satisfied-connection-intents.test.ts`: - For `blocked`, `todo` and `backlog` tasks: the card is retired with the connection, there is no delivery row, the agent is not woken, the task status does not change, and the delivery worker has no pending work. - A blocked task whose grant is revoked keeps its card pending. After the grant is active again, the next sweep retires it without a delivery row. - Fail before, pass after: with the `master` version of `satisfied-connection-intents.ts`, all 4 new cases fail. With this change, they pass. - `pnpm exec vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/idle-connection-history.test.ts src/__tests__/satisfied-connection-intents.test.ts` (in `server/`): 187 passed. - `pnpm exec vitest run` on the other suites that cover connection cards and their delivery (`connection-intents-service`, `ai-connections`, `agent-setup-connection-intents`, `runner-connection-eval-fixtures`, `server-startup-feedback-export`, `run-dispatch/domain/wake-context`, `execution-blocker-read`, `recovery/reconciliation-run-read`, `chat-interaction-continuation`, `heartbeat/run-state`): 472 passed. - `pnpm exec tsc --noEmit -p .` in `server/` passes. ## Risks - A card on a blocked or unstarted task is retired without a human answer. The person no longer sees it. If the connection breaks again before the task runs, the agent raises a new card in its next run, as it does on active tasks today. - Retiring a card on a waiting task does not wake it. Today, answering a tool card on such a task does not wake it either, so continuation behavior does not change. - The sweep now checks more cards. It still reads at most 50 per pass and rotates past cards that are not ready. - The worker's pending-work check does not change. Cards on waiting tasks are retired by the next sweep that runs for any reason, including at startup. - No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
ee6922f705 |
perf(tasks): open recent tasks without waiting on a slow ten-task preload (#21939)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person opens a task from the sidebar's Recent Tasks, the Inbox or the Tasks list, and the task chat shows its first frame from a `chat-bootstrap` read > - On staging, a task opened from Recent Tasks soon after a page load took 15.3 s to show its chat. The same task from the Inbox took 0.19 s > - Two defects cause this. Every task's bootstrap paid a fixed ~1.8 s for its latest runs. Recent Tasks preloaded ten tasks in one request (17.7 s), and an open waited for that whole request > - Recent Tasks also warmed its tasks only once per page load. Unread cache entries are dropped after five minutes, so its tasks were cold for the rest of the session > - This pull request makes the runs read use indexes, lets an open read its own task instead of waiting for a ten-task preload, and re-warms Recent Tasks on navigation > - The benefit is that a task from Recent Tasks opens as fast as from a task list ## Linked Issues or Issue Description Refs #19818, Refs #16713 (earlier proposals of the same `runsForIssue` rewrite; this PR adds a query-plan regression test and the client fixes). **What happened?** On staging, a click on a Recent Tasks link about 6 s after a page load took 15.3 s until the chat thread showed. The Inbox opened the same task in 0.19 s. **Steps to reproduce** 1. Load a page with a populated Recent Tasks sidebar. 2. Within a few seconds, click a recent task. 3. The chat stays on its loading state for many seconds. **Expected behavior** The task opens about as fast as from the Inbox or the Tasks list. **Root cause (measured on staging)** - `GET /issues/:id/runs?view=recent` took 1.7–1.9 s of server time for tasks with 1 to 24 runs. Every other first-frame read took 80–350 ms. The query filtered with `issue_id = $1 OR EXISTS (activity_log … run_id = heartbeat_runs.id)`, so Postgres walked every run of the company. - The bootstrap reads two tasks at a time, so the Recents preload of ten tasks took 17.7 s. Each task in a batch settles only with the whole request, and the page's reads wait for it (up to the 5 s share deadline, then read again under load). - `useRecentTaskOpenWarmup` remembered preloaded task ids for the life of the page and never preloaded them again. ## What Changed - `activityService.runsForIssue`: select the task's runs by `id IN (runs with issue_id = task UNION run ids from the task's activity)`. Each half uses its own index. The results are the same. - `startIssueChatBootstrap`: when the only request for a task is a background batch that is still loading, send a request for that task alone. All refs of the task point to the new request. The later batch answer only fills keys that are still empty or older. - `loadIssueChatBootstraps` (background): a task whose answer did not arrive is not held back for five minutes, so the next preload can retry it. - `useRecentTaskOpenWarmup`: use the background preload (five-minute repeat, the same as the cache GC time) and run again after each navigation (`navigationKey`). It no longer keeps a per-page-load set. `SidebarRecentTasks` and `CoordinatorSidebar` pass the pathname. ## Verification - `server/src/__tests__/issue-runs-read-cost.test.ts` (new): a company with 20,003 runs, and a task with 3 runs. EXPLAIN ANALYZE of the runs reads examined **40,011** run rows before the fix and **15** after (red/green). The test also checks the run order and the run that is linked only through activity. - `ui/src/lib/issue-chat-bootstrap.test.ts`: a new test opens a task while a ten-task preload is still loading. Before the fix it timed out waiting on the batch. After the fix it reads its own request. A second new test checks that a single-task request is still shared. - `ui/src/hooks/useRecentTaskOpenWarmup.test.tsx`: a new test re-warms Recents on a navigation after five minutes (red on the old hook). Another new test sends nothing for navigations inside five minutes. - Existing suites pass: activity service/routes, task cold open, agent conversations (197 tests); sidebar, Recents, navigation prefetch, issue detail cache, IssueDetail (341 tests). - `tsc --noEmit` is clean for `server` and `ui` (apart from the pre-existing Storybook index errors). - `pnpm verify` exits 2 (an opaque server setup dependency), so CI supplies the broad coverage. - After deploy: on staging, time a Recent Tasks click a few seconds after a cold load, and time `runs?view=recent`. ## Risks - Low risk. The SQL change keeps the same filter. It changes only how Postgres finds the candidate runs. The existing activity tests cover retry, credential and conversation cases. - An open during a background preload now sends one more bootstrap request for that task. This happens only on an open or a resting hover, never for each row. - Recents now sends at most one preload per five minutes while the person navigates. It sends nothing while the tab is idle or hidden. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser measurement on staging, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1116.0-canary.816000 |
||
|
|
c938bdfee4 |
perf(ci): stop printing Postgres TRUNCATE notices in server test shards (#21903)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server test suites run on 12 CI shards. Each suite gets a disposable Postgres database from `packages/db`. > - Many suites reset their state with `TRUNCATE ... CASCADE`. Postgres sends one `NOTICE` for each cascaded table. > - postgres.js prints each notice with `console.log` when a client sets no `onnotice`. Vitest then carries each line from the worker to the reporter. > - On the slowest shard of a recent run, about 600k of the 760k log lines were these notices. The noise makes the shard slower and the log hard to read. > - This pull request sets `client_min_messages = warning` on each disposable test database. Every client that a suite opens then gets no notices. > - The benefit is a shorter slowest server shard and a much smaller log. ## Linked Issues or Issue Description Refs #21879 (a separate cause of the same slow shard: missing shard-duration weights). **What existing behavior does this improve?** Disposable test databases send every Postgres `NOTICE` to the test clients. The test logs print each notice. **Subsystem affected** Test database fixtures in `packages/db` (`test-embedded-postgres.ts` and `test-postgres-admin.ts`). This change touches only test infrastructure. **Current behavior** On PR #21872 (run [43858567594](https://github.com/paperclipai/paperclip/actions/runs/43858567594)), the longest check was `ci / General tests (server (10/12))` at 768s. The Vitest step took 722s of that. The other server shards took 597–729s. The shard log had 760k lines. About 600k lines were `truncate cascades to table "…"` notices (about 77k notice objects): | suite | notices | |---|---| | heartbeat/workspace-preparation.test.ts | 19,089 | | agent-timer-deadlines.test.ts | 16,445 | | idle-incoming-wake.test.ts | 13,846 | | public-mcp/events-work.test.ts | 13,156 | | execution-control-work.test.ts | 4,485 | | 6 more suites | about 10k | About 20 suites already stop the notices one session at a time (`SET client_min_messages = warning` or `onnotice = () => {}`). **Proposed behavior** The fixture sets `client_min_messages = warning` on the database itself. It does this for the embedded-cluster path and for the `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` path. Warnings and errors still show. A suite that needs notices can lower the level for its own session. **Reason and benefit** The notice-heavy suites ran 15–50% over their manifest weights in this run (for example, workspace-preparation took 193s against a weight of 173s, and idle-incoming-wake took 40s against 26s). Other suites were near their weights. Estimate: about 0.5–1 min less on the slowest server shard. The log on that shard goes from about 760k lines to about 160k lines. If #21879 also merges, the server lane's critical path should go from about 12.8 min to about 10–11 min. **Breaking changes** None. Only disposable test databases change. ## What Changed - `packages/db/src/test-postgres-admin.ts`: add `QUIET_TEST_DATABASE_SQL`. Run it on each new database on the admin-URL path. - `packages/db/src/test-embedded-postgres.ts`: run the same statement on the `paperclip` database at each embedded start. It runs on each start, not only when the template is built. Thus a template that was cached before this change also gives a quiet database, and the template key stays the same. - `packages/db/src/test-postgres-admin.test.ts`: new test for the admin-URL path. - `packages/db/src/test-embedded-postgres.test.ts`: new tests for the embedded template path and the fresh path. - Each new test opens a new client, checks `SHOW client_min_messages`, and runs a `TRUNCATE ... CASCADE`. The test fails if the client gets a notice. ## Verification - `cd packages/db && npx vitest run src/test-postgres-admin.test.ts src/test-embedded-postgres.test.ts`: 2 files, 25 tests pass. - I changed the statement to `RESET client_min_messages` and ran the new tests again. All 3 new tests failed. Then I restored the statement. - `cd packages/db && npx tsc --noEmit -p .`: no errors. - To measure the time savings, compare the server shard timings of this PR's CI run with run 43858567594. ## Risks - Low risk. The change affects only disposable test databases. Production databases do not use these fixtures. - A test that expects to receive a `NOTICE` on a new client would now not get it. Such a test can set `client_min_messages = notice` for its session. No current suite depends on notices: the full CI run on this PR passed. - The time savings is an estimate. The shard planner and other suites can change the slowest shard. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, used through Claude Code as a Paperclip agent with tool use (shell, file edits, GitHub CLI, CI log inspection). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b298f5ce1d |
docs: allow the chat webhook ingress hostname in private mode (#18301)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native chat connectors receive provider webhooks at `/api/chat-webhooks/<publicId>/<provider>` > - `PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL` lets the webhook ingress use a different host than the board > - In `authenticated` + `private` mode, the private-hostname guard runs on every request, also on the webhook route > - The docs say that the ingress URL is not a trusted host. They do not say that the operator must allow the ingress hostname > - So a private instance behind a webhook-only tunnel answers 403 to every provider callback, and the operator gets no hint in the docs > - This pull request adds that step to the `PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL` row > - The benefit is that operators of private instances can connect Teams (or another webhook provider) without reading the guard source ## Linked Issues or Issue Description Refs #13468 (the next failure on the same path, after this step is done). Refs #15185. No issue exists for this docs gap. I searched open and closed issues and PRs for "hostname is not allowed webhook", "PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL", "private hostname guard webhook", "allowed-hostname tunnel 403" and "chat webhook 403". Only #14492 mentions the variable, and it is about Telegram long polling. **What happened?** Microsoft Teams callbacks through a webhook-only tunnel got `403 {"error":"This hostname is not allowed for this Paperclip instance. ..."}`. The request did not reach the Teams route. A `Host` rewrite at the tunnel did not help, because the guard reads `X-Forwarded-Host` first. **What did you expect?** The docs for `PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL` tell the operator to add the ingress hostname to `PAPERCLIP_ALLOWED_HOSTNAMES` in private mode. **Steps to reproduce** 1. Run `authenticated` mode with `private` exposure. 2. Set `PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL=https://<ingress-host>`. Do not add `<ingress-host>` to `PAPERCLIP_ALLOWED_HOSTNAMES`. 3. Send `POST https://<ingress-host>/api/chat-webhooks/<publicId>/microsoft-teams` through the tunnel. 4. The response is 403 from the private-hostname guard. **Paperclip version** v2026.1023.0 (Docker image), and the same code on `master`. **Deployment mode** `authenticated` + `private`, Docker, Postgres, Cloudflare tunnel that forwards only the Teams webhook path. ## What Changed - `docs/deploy/environment-variables.md`: the `PAPERCLIP_CHAT_WEBHOOK_PUBLIC_URL` row now tells private-mode operators to add the ingress hostname to `PAPERCLIP_ALLOWED_HOSTNAMES`. It also says that the guard reads `X-Forwarded-Host` before `Host`. ## Verification - Read the guard: `server/src/middleware/private-hostname-guard.ts` reads `x-forwarded-host` first, then `host`. `server/src/app.ts` mounts the guard for all routes, also `/api/chat-webhooks`. - Live check on our instance (v2026.1023.0, Microsoft 365 commercial tenant, Cloudflare tunnel): before the allowlist entry, `POST /api/chat-webhooks/<publicId>/microsoft-teams` returned 403 with the guard message. After we added the ingress hostname to `PAPERCLIP_ALLOWED_HOSTNAMES`, the same request reached the Teams route (`404 {"error":"Chat endpoint not found"}` for an unknown publicId). - `node scripts/check-forbidden-tokens.mjs`: `No forbidden tokens found.` - Docs-only change. No code or test change. ## Risks - Low risk. Docs only. No behavior change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected - check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5` (Opus 5.5), in Claude Code. Tool use (shell, GitHub CLI) and code execution. Default reasoning mode. A human operator owns the Paperclip instance and the evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c8dd2e400f |
fix(tasks): confirm Stop after preparation settles (#21906)
## Thinking Path > - Paperclip lets people start and stop work by AI agents. > - Stop must end the current turn and preserve the task for a later message. > - A person can press Stop while the server still prepares the turn. > - The preparation owner can prove that no model work started. > - That path saved the proof but left the Stop receipt at `requested`. > - The page then reported an unverified Stop and a failed attempt. > - This PR completes the receipt after proof and shows the confirmed Stop without a failure notice. ## Linked Issues or Issue Description **What happened?** A person submits a task by mistake and presses Stop during preparation. The run ends before the adapter starts. The server records that no provider work started, but the Stop receipt stays at `requested`. The page waits for a receipt that will never change. It then reports that it cannot verify Stop. The task also shows a failed-attempt notice. **Expected behavior** Stop ends the current turn without a failure notice. A later message starts one new turn through normal admission. Stop does not restart the task by itself. **Steps to reproduce** Hold session preparation before adapter dispatch. Press Stop through the run API. Release preparation and wait for the owner to finish. Read the run and the task's interruption state. Send a new task comment. Related: #20644 covers saved Retry after a committed native cancellation. #21382 covers a provider failure after native Stop. This PR covers Stop before adapter dispatch and its page presentation. ## What Changed - Complete the Stop receipt in `settleCancelledLegacyPreparation`. Keep its existing checks for process, session, events, leases and saved files. - Require no saved `budgetDispatch.handedOffAt` marker under the run lock. This marker can exist before an adapter records its start. A stale controller must keep the recovery hold after this handoff. - Write and require the `preparation_settled` proof before clearing the failed-attempt notice. Both page queries use one bounded SQL projection. Unknown and foreign proof labels keep the notice. - Clear an obsolete automatic-recovery marker only for a user's operator Stop. Keep system cancellation delivery metadata so saved comments still reach admission. Cover both cases with persisted recovery records and the stale-comment journey. - Give the bounded SQL result an explicit column name. Keep its real-query tests for typed and bounded Stop evidence. - Give the Retry test an active company membership and remove it during cleanup. The test uses normal authorization. - Record an expected cancellation as information in the run log. - Add a regression through the real Stop, run-read and comment routes with a real test database. Only the adapter boundary is simulated. Check that preparation cannot confirm Stop early, Stop does not restart work, and one later message starts one turn. - Document preparation Stop settlement in the heartbeat module notes. ## Verification - Head: `372ad381f4b60258e17b6b05eda133ed875e7de3`. This revision includes master `20b0d4e5a5`. [Full CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43910771322/attempts/2) completed successfully: all 52 executed jobs passed, including Build, Typecheck, all Runner lanes, all nine serialized suites, all twelve server shards and all eight browser shards. Greptile and Paperclip Review are both 5/5, and Superagent passed on this exact head. All three inline threads are resolved. The final commit inventory has 57 successful checks, one neutral review and three expected skips, with no pending or failing checks; Snyk also passed. - All 339 focused tests pass at this head across 11 suites. The Stop, saved-comment and fresh-continuation suites pass 132 tests. Eight other Stop contract suites pass 207 tests. - Red: the preparation regression receives `executionCancellation.state = requested` after the owner finishes. The presentation regressions report a failed attempt after a confirmed Stop. - Red: automatic recovery resolves the hold first, then preparation confirms Stop, but the run still shows a recovery notice. Green: the confirmed operator Stop clears the obsolete marker and projects `completed` / `Stopped`. - Red: the previous repair also changed system cancellations. Both stale-comment and Retry cases pass on the base and fail on the previous PR head. A direct regression also catches a preparation Stop receipt written for a system cancellation. Green: only a user's operator Stop gets this receipt; system cancellations keep their recovery delivery metadata. - Red: the real-query tests read PostgreSQL's implicit expression-column name after the expression changed. Green: the query has the explicit `result_json` alias, and all typed and bounded evidence assertions pass. - Red: a saved dispatch handoff marker was present before any adapter event. Both new cases returned an acknowledged Stop instead of keeping the hold. Green: the current locked row must have no marker, including a marker with JSON null. The receipt stays requested and the hold stays active. The tests use a stale owner snapshot. - The HTTP Stop/read/comment test uses real routes and PostgreSQL. It proves that pending preparation cannot confirm Stop, the owner confirms it after cleanup, Stop does not restart work, and a later message starts exactly one turn. Only the adapter boundary is simulated. - All source typecheck layers and the changed `test-tree-0` layer pass in low-memory mode. The first full typecheck attempt lost the compiler to the workspace memory limit while tests ran in parallel. The affected layers then passed alone. Other test layers passed on the previous head. The full current-head CI Typecheck + Release Registry job passed. `git diff --check` passes. - `pnpm verify` returned exit 2 and requested broader CI. No full local suite was started. - Environment: Linux x64, Node 25.9.0, fresh embedded PostgreSQL. This does not execute a live model or change staging. CI supplies Rust and full-suite coverage. - Previous-head CI found the SQL assertion and continuation regressions above. Its cancelled workspace lane lost its runner before a test result and was retried once. Previous-head results do not count as final-head passes. ## Risks - Only a requested, expected operator Stop with an acknowledged receipt gets neutral presentation. Unknown termination and file recovery remain visible. - The preparation owner writes acknowledgement only after all existing evidence checks pass. Terminal status alone is insufficient. A retained local lease test checks both refusal before release and confirmation after release. - The receipt does not grant replay authority. Budget, approval, dependency and cleanup gates still control later messages. - No schema or migration changes are required. ## Model Used OpenAI Codex, GPT-6, with reasoning and code tools. The exact served model build and context size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
f33c4cf6f7 |
feat(links): cache rich link status in the database and load it on idle (#21746)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task text contains links to pull requests and other work. > - A link has useful status that should be visible when a task opens. > - The current cache lives in the server process and the browser. > - A new browser or server restart has no stored status to show. > - This pull request stores link status in the database and reads it in batches. > - Idle page work updates stale status without a hover or a delay to task data. ## Linked Issues or Issue Description **Problem or motivation** PR status is unknown in a new browser until the user points to each link. The process cache cannot supply saved status after a server restart. Related work: #20593 and #20654. A search found no other PR for database link caching. **Proposed solution** Store status by company, viewer and canonical resource URL. Read saved status in one indexed batch query. After document load and core request settlement, use browser idle time to update missing or stale status. Use the same query for hover and idle work. **Alternatives considered** Adding provider calls to task reads would delay task loading. Company-wide reuse of personal GitHub results would cross viewer access boundaries. A periodic server worker would create work when no page needs the result. **Roadmap alignment** This extends the shipped rich-link feature. It does not duplicate a roadmap project or add a new provider integration. ## What Changed - Add an additive, repeat-safe `link_previews` migration. Store channel, resource kind, free-form status, JSON metadata and freshness. The table has no task relationship or GitHub status enum. - Bound each company/viewer cache to 1,000 rows across concurrent server instances. Remove rows expired for over 30 days during writes. Limit provider lookups to 600 requests per minute per scope/process. Keep cached batch reads available during backoff. - Add a database store owned by the link service. Reuse canonical results across tasks. Keep company and viewer isolation. Save hover and idle results. Preserve last known status during an outage and cache unavailable results. - Add a board-only batch GET endpoint, including viewer-role access. It reads at most 50 URLs with one indexed select and makes no provider calls. Both endpoints use private, no-store HTTP responses. - Add a page coordinator for mounted rich links. Reuse browser data, share pending reads, and defer provider work until document load, core request settlement and browser idle time. Limit automatic work to 200 unique links and three requests in flight. Remove queued work when links leave the page. Keep encoded batch URLs below 6 KB. - Keep exact navigation URLs, status styling, anchored cards and keyboard access. Add a page-load Storybook case and browser regression. The link service owns cache policy and provider calls. The browser coordinator owns page timing. The shared contract adds only the batch limit and response shape. The database adds storage because process and browser caches cannot serve a new browser session. Plugin resolver registration is outside this change; storage accepts future provider channels and status values. ## Verification - **All 93 affected tests pass:** 40 link service, real PostgreSQL and HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case failed before the storage repair. Concurrent writers now retain at most 1,000 rows per company/viewer; cleanup removes long-expired rows and keeps recent stale status. Request backoff preserves cached reads and isolates companies/viewers. The earlier eight concurrent hydration regressions also pass. Focused store/service/unit types pass. Commands: `pnpm exec vitest run server/src/__tests__/link-previews.test.ts server/src/__tests__/link-preview-store.integration.test.ts server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and `pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts --maxWorkers=1`. - **All 50 affected server tests passed before the later server-only repairs** on the master-integrated source: 20 recovery/deadline cases, one authenticated review-counts HTTP case, 10 link-service cases, 13 real PostgreSQL cache cases and six link-route cases. Command: `pnpm exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts server/src/__tests__/tool-connection-review-counts-http.test.ts server/src/__tests__/link-previews.test.ts server/src/__tests__/link-preview-store.integration.test.ts server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`. - Reproduced the CI watchdog assertion locally twice before the test repair. Ordinary assignment recovery can win `wakeReason` while the watchdog wake coalesces into the same successful run with its full mandate. The test now verifies the persisted watchdog identity, watched issue and stopped fingerprint, plus exactly one review run. All 21 recovery/HTTP cases passed before integrating master; all 50 affected cases pass after integration. No runtime behavior was changed for this repair. - **Fresh static Storybook build and all 22 browser cases pass**, with two workers and retries disabled. They cover states, anchored hover, keyboard access, exact navigation, duplicate requests, reload persistence, expiry and saved-status hydration followed by one idle refresh. The first browser attempt could not launch because this session lacked Chromium; installing the expected browser resolved setup. That attempt had no behavioral verdict. - The cache repair remains covered by four PostgreSQL red/green regressions for expired memory and newer durable status during provider failure/timeout, plus check-time precedence and failed-read recovery. Earlier UI/database/focused server type and migration safety/compatibility/replay checks remain unchanged-input evidence. - Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK source validation checks 545 inputs with no stale source identities or changed runtime/lock pins. Whitespace validation passes. - `pnpm verify --explain` returns exit 2 and requires broad CI. No full local suite or deployed acceptance is claimed. Earlier local full-server type checks exceeded memory; current-head CI supplies broad type/build/test coverage. - [Previous CI 43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895) passed full types/build but failed on embedded PostgreSQL startup (`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task browser cleanup (activity-log foreign key on company deletion). One failed-jobs retry was requested. Its browser shard 7 now passes without code changes; the exact deletion-race cause remains unproven. The watchdog assertion is repaired in this revision. The previous failures remain recorded, not counted as an overall pass. - [Prior CI 43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848) passed full types, build and all eight browser shards. Several test jobs were cancelled; the run has no overall pass. Its concurrent update finding is repaired. [Next CI 43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530) also passed types, build and all browser shards, but several server jobs were cancelled. Greptile and Paperclip Review gave that revision 5/5. Its security scan found unbounded retained cache rows. The quota, retention and request-limit repair above addresses that finding. All ten review threads are resolved. - [Current CI 43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278), fresh code reviews and a fresh security scan are pending on `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until the current revision clears its gates. Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head: `b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree `643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation verified equality. ## Risks - Each company/viewer retains at most 1,000 rows. Writes remove excess rows and entries whose freshness expired over 30 days ago. Inactive scopes remain bounded and are cleaned on the next write. Company deletion removes all rows. No periodic worker runs. Scope writes use a transaction lock; the existing deadline and concurrency budget cover that work. - Last known status can be stale until idle or hover work completes. Provider failures keep that status through the negative-cache window. - Automatic enrichment adds bounded provider traffic. It runs only for mounted supported links, once per mount, with the existing timeout and server concurrency limit. - Cache storage is viewer-scoped because provider results can be private. New viewers load their own results. This prevents reuse across users with different repository access. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution and browser automation. The runtime does not expose the exact serving model ID or context-window size. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com> |
||
|
|
0e8447fe77 |
fix(release): retry the finalize install while a large tarball lags its npm metadata (#21925)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Since #21917, each canary's npm confirmation, git tag, Docker build and onboarding smoke run in `canary-finalize.yml` > - The first canary through it, `2026.1116.0-canary.814700`, passed the master-position, visibility, provenance (31 of 31) and dist-tag checks > - It then failed the clean-prefix install: the 524 MB `@paperclipai/server` tarball returned 404 while npm already listed the version > - Without a tag, nightly cannot select that canary, and no Docker image or smoke runs for it > - This pull request retries that install for a bounded time > - The benefit is that a large tarball's normal lag no longer stops a canary from finishing ## Linked Issues or Issue Description **What happened?** Canary finalize run 43927092510 for `2026.1116.0-canary.814700`: `npm error 404 Not Found - GET https://registry.npmjs.org/@paperclipai/server/-/server-2026.1116.0-canary.814700.tgz`. npm's packument records that version at 18:31:41Z. The install ran at 18:32:03Z, and the tarball still returned 404 at 18:34:39 and 18:35:15. `@paperclipai/shared` and `paperclipai` tarballs at the same version returned 200. **Expected behavior** Finalize waits, within a bound, until the published tarball downloads, then tags the canary. Refs #21917 ## What Changed - `scripts/release-lib.sh`: `wait_for_npm_installable` retries `verify_npm_installable` up to N attempts with a fixed delay. - `scripts/confirm-canary-publication.sh`: uses it with 40 attempts × 30 s (`NPM_INSTALL_VERIFY_ATTEMPTS`, `NPM_INSTALL_VERIFY_DELAY_SECONDS`). - `.github/workflows/canary-finalize.yml`: `confirm_canary` timeout 45 → 75 min, to fit the existing 30-minute visibility budget plus the 20-minute install budget. - Test and `doc/RELEASING.md`. ## Verification - `node --test scripts/release-lib.test.mjs`: 23 of 23 pass, including a new test: immediate success, recovery after 3 failures (4 attempts, warning names attempt 3/5), and failure after the bound (4 of 4 attempts). - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 73 of 73 pass. - After merge: the next canary's finalize run passes the install and pushes its tag. `canary.814700` itself is recovered by rerunning its finalize run once its tarball downloads. ## Risks - Low. A tarball that never appears still fails finalize, after 20 minutes instead of at once. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b816c95f96 |
chore(lockfile): refresh pnpm lockfiles (#21936)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
|
fb288f5666 |
fix(acpx): only treat an error result as a Claude sign-in failure (#21921)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the Claude ACP bridge (`claude-agent-acp` 0.73.0), which Paperclip patches in `patches/` > - The bridge reads Claude Code's final result for each turn. If the result text contains "Please run /login", the bridge fails the turn as a sign-in failure > - That check does not look at `is_error`. A successful reply that quotes the CLI's sign-in hint also fails the turn > - Agents quote that hint when they report the output of a `claude` command that they ran. The work is done, but the run fails with "ACP agent reported a terminal access failure." > - Upstream fixed this in [agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103) (commit [`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f), released in 0.81.1), "fix: gate login result text on error state" > - This pull request ports that upstream change into the 0.73.0 bridge patch > - The benefit is that a correct, finished run is not reported as a failed sign-in ## Linked Issues or Issue Description Upstream fix: [agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103) (commit [`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f), released in 0.81.1). This pull request ports it. Refs #14089. That pull request fixes the same class of problem in the Claude CLI lane (`execute.ts`). This pull request fixes the ACP lane. Related: #14093 shows the same error text, but it has a different cause. This pull request does not fix #14093. **What happened?** An agent ran a script that calls the `claude` CLI without credentials. The CLI printed "Not logged in · Please run /login". The agent finished its task and quoted that text in its final reply. Claude Code ended the turn with a successful result (`is_error: false`). The bridge saw "Please run /login" in the result text and failed the turn with `auth_required`. Paperclip then failed the run with "ACP agent reported a terminal access failure." **Expected behavior** A successful result completes the turn. Only an error result with the sign-in hint is a sign-in failure. **Steps to reproduce** 1. Run a Claude agent on the ACP lane. 2. Make the model's final reply contain the text "Please run /login" (for example, quote the output of `claude -p hi` without credentials). 3. See that the run fails with "ACP agent reported a terminal access failure." The new test in this pull request does these steps against a local API stand-in. **Agent adapter(s) involved** `claude_local` on the ACP engine, and the Runner's ACPX Claude profile. Both use the same patched bridge. ## What Changed - `patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch`: port [agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103) (commit [`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f), released in 0.81.1). It has two parts, the same as upstream: - The `success` result check is now `message.is_error && message.result.includes("Please run /login")`. A successful result that quotes the sign-in hint no longer fails the turn. - The goal-command check also requires `message.is_error`. A `/goal` command whose successful result quotes the sign-in hint no longer rolls the session goal back. - Both patch comments name the upstream change. All other patch hunks are unchanged. - `packages/paperclip-runner/src/drivers/acpx/acpx-runtime-adapter.claude-login-text.test.ts`: a new test. It runs the real bridge and the bundled Claude Code against a local API stand-in. The model's reply quotes the sign-in hint, and the test expects the turn to complete. ## Verification - Red: on `master`'s patch, the new test fails with `{"status":"failed","error":{"message":"ACP agent reported a terminal access failure."}}`. - Green: with this patch, the new test passes (`status: "completed"`, `stopReason: "end_turn"`). - I ran Claude Code 2.1.286 (the bundled version) with no credentials. Its real sign-in failure is a result with `is_error: true` and the text "Not logged in · Please run /login". The guard still catches it. The separate synthetic sign-in message check is unchanged. - The goal-command part has no new test here. The upstream change adds a test for it. The runner's `session-goals.test.ts` passes with the new patch. - `node --test scripts/acpx-patch-packaging.test.mjs`: 26 passed. - `node --test packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs`: 9 passed. - Runner real-bridge Claude tests (`acpx-runtime-adapter.claude-*.test.ts`, `acpx-runtime-adapter.test.ts`, `claude-scheduling-tools.claude-acp.test.ts`, `runner-tool-bridge.claude-*.test.ts`): 139 passed. - `claude-local` ACP tests (`acp.test.ts`, `acp.auth.test.ts`, `acp.quota.test.ts`): 95 passed. - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`: passed. - I did not commit `pnpm-lock.yaml`. The patch hash changes, and CI resolves the lockfile for this run. ## Risks - Low risk. The change adds one condition to each of two checks in the bridge patch. Both match the upstream change. - The bridge's `commandDigest` does not change, because the patch does not touch `dist/index.js`. Saved sessions are not reset. - A sign-in failure that Claude Code reports as a successful, non-error result is no longer caught by this check. Claude Code 2.1.286 reports sign-in failures as error results, and upstream uses the same rule. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, subagents) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cc9ff75cb8 |
fix(runtime): keep managed GitHub access across a controller restart (#21928)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs give agents managed GitHub access through `gh` and `git` launchers on the provider's `PATH` > - A provider process can keep running while the server restarts, and the next controller adopts it > - The detaching controller deleted the launcher directory, and the adopting controller staged launchers in a new random directory > - The running provider still names the old directory, so every later `gh` or `git` call ran without credentials > - This pull request keeps the launcher directory stable for one provider session and task, and re-arms it after the restart > - The benefit is that a long run keeps its GitHub identity across a server deploy or restart ## Linked Issues or Issue Description No public issue exists. This describes the bug in-PR, with the bug template labels. **What happened?** An agent run on a Daytona sandbox used `gh` with managed GitHub access. The server restarted during the run (a rolling deploy), and the controller re-attached to the live provider process. After that, every `gh` call failed with `You are not logged into any GitHub hosts`. `git credential fill` returned no token. The agent reported that its token had "expired" and used the unauthenticated public API for the rest of the run. **Steps to reproduce** 1. Start a native run with managed GitHub access on a remote sandbox. 2. Run a `gh` command in the run. It succeeds. 3. Restart the server while the provider process is still running. The new controller re-attaches to the process. 4. Run a `gh` command again in the same run. It fails with "not logged in", and no `Paperclip: GitHub ...` diagnostic appears. **Expected behavior** After the new controller adopts the provider, `gh` and `git` get credentials from the new controller, as they did before the restart. **Root cause** - `createNativeGitHubAccess` staged the launchers in `native-session-<random UUID>` and put that path in the provider's `PATH`, `BASH_ENV`, and `GH_CONFIG_DIR`. - On a restart detach, `githubAccess.stop()` ran `rm -rf` on that directory (`detachReleasedWarmNativeSession`, and the run's final `github_stop` while `nativeRunsDetachingForRestart` is set). - The adopting controller made a new random directory. The provider environment cannot change after spawn, so the provider never saw it. - The shims were gone. `gh` resolved to the real binary with an empty `GH_CONFIG_DIR`, so no Paperclip diagnostic appeared. Refs #15354. That PR keeps ACPX sessions across launcher rotation in the legacy adapter path. This PR covers the native runtime path. ## What Changed - `createNativeGitHubAccess` takes an optional `launcherKey`. The launcher directory is `native-github-<sha256(key)>`, so the same provider session and task always get the same directory. - The native session executor passes a key built from `nativeSessionScopeKey` and the issue id. - `prepareGitHubOperationLaunchers` takes an optional `broker`. It then stages broker-file launchers and an owner-only `broker.json`. The launchers already read `broker.json` on each operation in this mode (`githubLauncherSource({ brokerFile: true })`). A restaged directory therefore gives the running provider the new endpoint. - Native access passes `broker` only for remote sandbox targets. A sandbox provider is the one that outlives a controller restart. Local launchers live in the shared host temp directory, so they keep the capability in the provider environment only, as before. - `stop({ preserveLaunchers: true })` keeps the launcher directory. A restart detach of an active run and of a released warm session use this option. A normal stop still removes the directory. - A final stop does not remove a directory that a newer access in the same process has already restaged. - The callback bridge runtime stays in its own per-access directory. A new bridge never shares a queue directory with an old bridge. - `syncRemoteTextFiles` takes a `private` flag. It sets `umask 077` for the whole staging script, so the upload copy, the partial file, and the final file are never readable by group or other users. Local launcher files are written to a temporary file and renamed, so a running launcher never reads a partly written file. ## Verification - New `native-github-access` test on a sandbox target: a live parent process gets credentials, the access stops with `preserveLaunchers`, and a new access with the same key re-arms the directory. The same parent PID then gets the new controller's credentials, with empty stderr. `broker.json` has mode `600`. The final stop removes the directory. - New previous-release cutover test: a provider launched with the old random-directory layout keeps working without managed GitHub after its controller removes the directory. The new controller does not resolve credentials for it. A provider that the new controller launches gets credentials. - New local regression: local launcher files do not contain the broker capability, and there is no `broker.json`. - New local restart test: a local provider adopted after a restart gets a Paperclip diagnostic and no credentials. The new controller resolves nothing for the old endpoint. - New `sandbox-callback-bridge` test: staging a private file runs with umask `077` and leaves the file at mode `600`. - New `native-github-access` test: when a superseded access stops, its successor's launchers stay usable. A different key gives a different directory. - New `native-session-executor` test: a run detached for restart while active stops its GitHub access with `{ preserveLaunchers: true }`, and creates it with a `launcherKey`. The existing overlapping-failure test now asserts `{ preserveLaunchers: false }`. - The new tests fail on the old behavior. I checked this by forcing `brokerFile: false`, forcing `preserveLaunchers = false`, removing the `umask 077` line, and writing `broker.json` for local targets. - Passed: `native-github-access` (26), `native-session-executor` + `github-operation-credentials` + `heartbeat-github-launchers` + `native-stop-accounting` + `heartbeat-project-env` (1640 passed, 7 skipped), `native-runner-restart-recovery.integration` and `heartbeat-native-restart-suspension`. - Passed: adapter-utils `execution-target`, `github-launcher`, `github-launcher-environment`, `sandbox-callback-bridge`; runner `acpx/github-launcher`, `acpx/environment`, `codex/codex-security-config`. - `pnpm --filter @paperclipai/adapter-utils typecheck` and the server `pnpm typecheck` pass. - Not run locally: `native-warm-checkpoint-ownership.integration` needs the compiled `paperclip-runnerd` binary. CI builds it. ## Previous-release story - No persisted run, session, checkpoint, or database state changes. Only the in-sandbox launcher layout changes. - A provider launched by the previous release has a random `native-session-*` launcher directory in its environment. The previous release's controller removes that directory at the cutover restart. The new controller cannot find it, so it does not re-arm it. - That provider carries forward without managed GitHub until its next spawn, which is the current behavior. The run is not refused and is not retried. `gh` and `git` run without managed credentials, as they do today after a restart. The cutover test covers this. - A provider that the new release launches gets the stable directory and survives later restarts. ## Risks - On sandbox targets, the provider can now read the bridge capability in `broker.json`, as well as in its environment. The file holds the same per-access transport token that the environment already holds, not a run token or a GitHub credential. The file and its staging copies are mode `600`, and the broker still checks the active run and grants on each request. Local targets write no `broker.json`. - Local providers that outlive a restart still lose managed GitHub. This is not a regression: master deletes the launcher directory at the detach, so the old local provider also runs without credentials there. With this PR, the local launchers stay in place and print a `Paperclip: GitHub ...` diagnostic instead of a silent "not logged in". Re-arming local providers needs a broker channel that other local runs under the same user cannot read. That is out of scope here, because `broker.json` in the shared temp directory was a security finding on this PR. - On sandbox targets, `git` and `gh` children no longer inherit `PAPERCLIP_API_URL`, `PAPERCLIP_API_KEY`, or `PAPERCLIP_GITHUB_BRIDGE_TOKEN`. Broker-file mode removes them, as it already did in its existing tests. - After a restart detach, a launcher directory stays in the workspace runtime directory. If no controller adopts the provider, that directory stays until a later access with the same key stops, or until the workspace is removed. It contains no GitHub credential. - Between the detach and the adoption, `gh` gets a transport error from the old endpoint. It then prints `broker_transport_unavailable` and continues without credentials, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1116.0-canary.815200 |
||
|
|
0308108531 |
fix(native): pipeline tool progress events with text deltas (#21533)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs store every relayed runner event in `heartbeat_run_events`. Each stored event also locks and updates the run's `heartbeat_runs` row. > - Text deltas already commit in batches. Every other event, including each `tool.execution.progressed` event, commits in its own transaction. > - Tool receipts and usage accounting lock the same run row. Under parallel tool calls, accounting failed with "other updates to the same run kept conflicting", and a provider-pack receipt save hit a lock timeout. > - #21520 made events that arrive while a batch commits join the next batch. This pull request adds tool progress events to the pipelined types, so they use those batches too. > - The benefit is fewer run-row transactions in tool-heavy turns, without changing event order. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents that made about 190 tool calls in about 10 minutes. Three tool calls failed with "could not save this run's budget or usage records because other updates to the same run kept conflicting", and a provider-pack admission receipt failed with a lock timeout (55P03). Every tool progress event of the run was a separate transaction that updated the same run row. **Expected behavior** Observational events such as tool progress commit in batches, so they hold the run row lock fewer times. **Steps to reproduce** 1. Run a turn that makes many tool calls in parallel. 2. Each `tool.execution.progressed` event commits in its own transaction and updates `heartbeat_runs`. **Relevant logs or output** ``` could not save this run's budget or usage records because other updates to the same run kept conflicting native provider-pack admission receipt could not be saved (cause=55P03) ``` Related: #21531 tried letting the runner relay events faster, which would make this contention matter more. It is closed. ## What Changed - `paperclip-control-plane-port.ts`: - `pipelinedEventTypes` names `item.delta` and `tool.execution.progressed`. Both go through the batched append path and its group commit from #21520. - The text-delta boundary is now tracked when an event is appended: a pipelined event that is not a delta ends it, and a delta with text starts it. The first text delta after any other event still commits at once. - `contracts/control-plane-port.ts`: optional `pipelinedEventTypes`, and the `streaming` option documents that it applies to those types. - `native-session-runtime.ts`: the runtime pipelines the port's `pipelinedEventTypes`, and only `item.delta` when the port does not name them. - Tests: - `paperclip-control-plane-port.test.ts`: three progress events appended while a batch commits share one transaction, and a following event commits after them. - `native-session-runtime.test.ts`: progress appends start without waiting only when the port pipelines them. ## Verification - After rebasing onto current master: `pnpm exec vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts` (in `server/`): 108 passed. - `npx vitest run src/native-session-runtime.test.ts` (in `packages/paperclip-runner`): 298 passed, 1 failed. `settles a governed wait through real Runner shutdown after a missing Codex terminal` needs the built Rust test binaries (`fake-codex-app-server`), which my checkout does not have. CI builds them. - `pnpm exec tsc --noEmit -p .` in `server/` (after building its workspace dependencies) and in `packages/paperclip-runner` pass. - `npx vitest run src/__tests__/chat-channels.integration.test.ts src/__tests__/heartbeat-process-recovery.test.ts src/services/native-runtime/native-plan-wait.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts` (in `server/`): 2,586 passed, 2 failed. `binds Slack tools to the admitted linked request...` reads a skills checkout that is not next to my worktree. `materializes direct external-chat finals once...` failed in the full file run but passes alone, with and without this change. ## Risks - Progress events now commit up to one transaction later than before. Their receipts and live UI updates follow their batch. - A batch that contains progress events still writes rows one by one inside its transaction, because progress events register chat-progress wakes. The gain is one transaction and one lock wait per batch, not one row update per batch. - If a batch fails, every event in it fails, and the runner resends them. That is the same as for text deltas today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
fdf66c7989 |
test(server): give the blocker history case room on a slow shard (#18720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its CI runs the server suite in shards so a change finishes in minutes > - One case in the attention-retention suite inserts 2100 issues and their relations, then runs EXPLAIN ANALYZE > - A single-file run takes about 90 seconds of local work, and the case itself fits in 15 seconds there > - On a two-core CI runner beside 110 other files, the case exceeds the 15s default > - This pull request gives that one case a 60s budget > - The benefit is that a slow shard stops failing for timing reasons ## Linked Issues or Issue Description **Summary** `ci / General tests (server (2/12))` fails intermittently on the same case. **Steps to reproduce** 1. Open a pull request that does not touch `server/`. 2. Wait for the shard that contains `src/__tests__/attention-retention-work.test.ts`. **Expected behavior** The case passes, as it does locally. **Actual behavior** `Error: Test timed out in 15000ms.` at `attention-retention-work.test.ts:443`. The rest of the shard passes (2197 tests). ## What Changed - Raise the timeout of the `it.each(["running", "run", "confirmation", "completed"])` case to 60 seconds. Nothing else changes: same assertions, same 2100-row fixture. ## Verification ```sh cd server npx vitest run src/__tests__/attention-retention-work.test.ts # 53 tests pass ``` The case still exercises the same query plan and the same bounded retention checks; only the wall-clock budget changed. ## Risks - Low risk. A slower failure now takes up to 60 seconds to surface on that one case. - No production code changes. ## Model Used - Provider: DeepSeek - Model ID: `deepseek-flash` - Harness: OpenCode - Capabilities used: tool calls (shell, file read/write, search) - Context window: not published by the provider ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting mergecanary/v2026.1116.0-canary.815000 |
||
|
|
d216e413f3 |
test(recovery): stop the idle-backstop suite starting real runs and bound every wait (#21924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task recovery suites drive real recovery workers against an embedded database with a faked clock > - `issue-recovery-idle-backstop.test.ts` seeds 50 agents that take on-demand wakes, so recovery starts real `process` runs while timers are faked. A run then waits on timers that only move when the test advances, and the test waits for the run's passes with no time limit > - On master's readiness run for |
||
|
|
3452639a81 |
fix(runner): reject only the overflow call when parallel tool calls exceed the result reserve (#21913)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs call Paperclip tools through the runner's provider tool bridge > - The bridge reserves 768 KiB for each pending call against an 8 MiB budget per turn > - When an agent sends many tool calls in parallel, the reserve for the pending calls can go over the budget, although the bytes the turn really holds are small > - The bridge then reports the durable receipt limit, and the backend stops the whole turn. The run ends as cancelled with `needs_review`, and the task stays blocked with a Retry button > - This pull request rejects only the overflow call as retryable, and keeps the turn stop for real held bytes. It also adds a server backstop: a turn stopped at the receipt limit continues by itself in a fresh session > - The benefit is that a parallel burst of tool calls no longer leaves a task waiting for a person ## Linked Issues or Issue Description **What happened?** In a restart-chaos wave, an opencode agent sent 10 parallel `get_task` calls. The turn already held about 0.7 MB of results. 10 × 768 KiB plus 0.7 MB is more than 8 MiB, so `begin_call` returned `active_turn_receipt_limit()` for the 10th call. This branch does not set `durable_run_receipt_limit_reached`, so it is a temporary reserve overflow. `handle_tool_call` sends every receipt-limit error to `stop_turn_at_tool_receipt_limit`, which interrupts the turn. The run ended as cancelled with `needs_review` and a human review request. The release then blocked the task (`native_continuation_requires_reconciliation`) and showed Retry. No continuation was scheduled. The task moved again only because a person posted a comment ten minutes later. **Expected behavior** The overflow call gets a normal, retryable rejection, and the turn continues. The agent can send the call again after a pending result arrives. If a turn does stop at the receipt limit, Paperclip continues the task in a fresh session, as it does for `agent_tool_call_loop` (#21734). The task does not show Retry. **Steps to reproduce** 1. Start a native turn that holds about 600 KiB of settled tool results. 2. Send 10 tool calls in parallel. 3. On master, the 10th call stops the turn, and the task becomes blocked with Retry. **Paperclip version or commit** `master` atcanary/v2026.1116.0-canary.814800 |
||
|
|
d16c06308d |
perf(release): number canaries by master position and confirm them outside the release lock (#21917)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push runs a canary, and only one canary at a time may run the release workflow > - After #21554 and #21555 a canary held that lock for 35.5 min, and 13-16 min of it was waiting for npm to show the new packages > - The wait was required: the next canary computed `canary.N` from npm's version list, so starting before npm showed the previous canary would reuse its number > - This pull request numbers a canary by its commit's position on master and moves the registry confirmation, the git tag, the Docker dispatch and the onboarding smoke to a separate `canary-finalize.yml` run > - The benefit is that the release lock is held for about 20-25 min per canary instead of 35, with the same checks before a canary is tagged ## Linked Issues or Issue Description **What existing behavior does this improve?** Canary throughput. In run 43715037497 (2026-11-15) `publish_canary` took 23.7 min, and 13.3 min of that was "Waiting for registry visibility" (13:08:50 to 13:22:11). The next canary could not start until that run ended. **Current behavior** `release.sh canary` sets `N` to (highest published `-canary.N` for today) + 1, then publishes, waits for npm to show every package, checks dist-tags, installs `paperclipai@canary`, and creates `canary/v<version>`. `publish_canary` pushes the tag, dispatches `docker.yml`, and `smoke_canary_onboarding` runs in the same run. **Proposed behavior** `N` = `git rev-list --count HEAD` (full history required), or the registry number + 1 when that is higher (a rerun of the same commit). `release.sh canary --defer-registry-confirmation` ends after npm accepts every publish. `publish_canary` dispatches `canary-finalize.yml`, which does the confirmation, tag, Docker dispatch and smoke outside the release lock. Nightly, beta and stable numbering and flow are unchanged. Refs #21554, #21555 ## What Changed - `scripts/release-lib.sh`: `next_canary_version_from_source`. A publishing canary refuses a shallow checkout; a dry run (the PR Canary Dry Run job) falls back to the registry number. - `scripts/release.sh`: canary uses the new number; new canary-only `--defer-registry-confirmation` flag skips the registry wait, the dist-tag check and the local tag, and writes the version to `PAPERCLIP_RELEASE_VERSION_FILE`. - `scripts/confirm-canary-publication.sh` (new): waits for registry visibility, checks provenance, checks dist-tags with `--allow-newer-canary`, installs `paperclipai@<version>`, then creates and pushes `canary/v<version>`. A tag already on the same commit is accepted, so reruns are safe. - `scripts/verify-canary-provenance.mjs` (new): every package's npm SLSA provenance must name `.github/workflows/release.yml` on `refs/heads/master` in `paperclipai/paperclip` at the source commit. A dispatch with invented inputs cannot tag anything. - `scripts/verify-release-registry-state.mjs`: `--allow-newer-canary` accepts a `canary` dist-tag that a newer canary already moved; older, foreign or missing tags still fail. - `.github/workflows/canary-finalize.yml` (new, `workflow_dispatch`, master only): `confirm_canary` (contents: write, actions: write) and `smoke_canary_onboarding` (downloads the publisher's lockfile artifact by run id). Inputs reach shell steps only through a validating step. - `.github/workflows/release.yml`: `publish_canary` hands off to `canary-finalize.yml` and drops to `contents: read`; its tag push, Docker dispatch and smoke job move to `canary-finalize.yml`. - Tests and `doc/RELEASING.md`. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/release-lib.test.mjs scripts/verify-release-registry-state.test.mjs scripts/verify-canary-provenance.test.mjs`: 846 pass. `scripts/__tests__/release-verify-workflow.test.mjs`: 73 pass. - New tests: numbering by master position, rerun moves past a published number, shallow refusal and dry-run fallback; newer/older/foreign dist-tags; provenance from another commit, workflow, branch, repository or package is rejected; finalize wiring, permissions and input handling. - Real npm data: `verify-canary-provenance.mjs 2026.1116.0-canary.7 b4fa1c7f... paperclipai @paperclipai/shared @paperclipai/server` passes; the same version with another commit fails. - `confirm-canary-publication.sh 2026.1116.0-canary.7 b4fa1c7f...` from a checkout of that commit ran all steps against npm in 61 s and ended with "canary/v2026.1116.0-canary.7 already points at b4fa1c7f...", pushing nothing. - `release.sh canary --dry-run --defer-registry-confirmation` and `release.sh nightly --defer-registry-confirmation` both stop with "only applies to a published canary". - `actionlint` reports no new findings compared with master. - Not verified before merge: a full publish through `canary-finalize.yml`. It can run only from master. The first canary after merge is the check. ## Risks - Canary numbers jump from `canary.7` to about `canary.8144` (master has 8,143 commits). This is valid semver and still higher than every earlier canary; users install through the `canary` dist-tag. - The `canary/v` tag, Docker image and smoke now finish after the release run ends. Nightly selects only tagged canaries, as before. If finalize fails, that canary is on npm without a tag, so nightly skips it, as it skips any canary whose old in-run verification failed. - A canary can now publish while the previous one is still being confirmed. Publishing itself is still serialized, so the `canary` dist-tag only moves forward. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1116.0-canary.814700 |
||
|
|
93111b175b |
fix(wake-queue): keep the child-completion reason when a person's comment merges into its held wake (#21918)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The wake queue holds new work for a task while an agent run is
active. A later wake for the same task merges into the held wake
> - A parent task must not finish while a finished child's result is
still waiting for a parent turn. The finish gate checks for this pending
child result
> - The gate finds the held child-completion wake by its context wake
reason
> - When a person's comment merged into that held wake, the merge
replaced the wake reason with `issue_commented`. The gate then did not
see the pending child result
> - This pull request keeps every merged reason in the held wake and
makes the gate read that list
> - The benefit is that a merge keeps the meaning of both wakes: the
comment is delivered, and the pending child result still blocks a
premature finish
## Linked Issues or Issue Description
Refs #21914
**What happened?**
A restart test held a child-completion wake behind a running parent run.
A person then posted a comment. The comment wake merged into the held
wake. After the merge, the held wake's
`_paperclipWakeContext.wakeReason` was `issue_commented`, not
`issue_children_completed`. `hasPendingNativeChildCompletion` matches
held wakes by that field only. So the finish gate could no longer see
the pending child result.
**Expected behavior**
The held wake keeps both reasons. The finish gate still sees the pending
child result. The next run still receives the person's comment.
**Steps to reproduce**
1. Start a run for a parent task.
2. Hold a child-completion wake behind the run.
3. Post a person's comment on the parent. The comment wake merges into
the held wake.
4. Call `hasPendingNativeChildCompletion` for the running parent run. It
returns `false`.
**Paperclip version or commit**
`master` at
canary/v2026.1116.0-canary.10
|
||
|
|
4da6902cb3 |
fix(dispatch): start work on a task tree a person created in the person tier (#21916)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue chooses which queued run starts when an agent at its concurrency limit frees a slot > - The queue puts runs a person asked for in a "person" tier ahead of agent "work" > - Subtasks of a person's task, and the parent's resume after its children complete, ranked as agent work > - Under load these runs started only through the 60-minute oldest-run share, so a half-finished task waited about 55 minutes per step behind newer tasks > - This pull request puts every non-recovery run on a task tree a person created in the person tier > - The benefit is that a person's started work finishes before newer work starts, and the fairness bounds stay in place ## Linked Issues or Issue Description **What existing behavior does this improve?** The order in which an agent at its concurrency limit starts queued runs. **Subsystem affected** Server heartbeat queue: `server/src/services/heartbeat/queued-run-dispatch-order.ts` and `queued-run-line.ts`. **Current behavior** A board user's new assignment ranks as person work. An agent's subtask of that person's task, and the parent's `issue_children_completed` resume, rank as agent work. In a restart chaos run with agents at capacity, the median queue wait of a parent resume was about 1,640 s. The median wait of a new assignment was about 190 to 290 s. One parent waited while 14 newer tasks started first. Subtasks and parent resumes started only near the 60-minute aging bound. **Proposed behavior** A non-recovery run on a task tree that a person created goes in the person tier. A task is in such a tree when the task, or any task above it, has `createdByUserId`. In the tier, runs keep the existing order: readiness, then priority, then age (oldest first). **Reason and benefit** A person waits on the whole tree, not only on the top task. Work that a person started now finishes before newer tasks take its slots. **Breaking changes** None. There is no schema or API change. ## What Changed - `queuedRunDispatchClass` puts a run with `personTask` in the person tier. System recovery stays in the recovery tiers, also on a person's tree. - `personTask` now means "the task, or a task above it, was created by a person". The existing `createdByUserId` marker is the source. - New `listPersonTreeIssueIds`: one recursive query per dispatch pass. It walks each queued task's parent chain through the `(company_id, id)` unique index. It stops at the first person-created task, at most 32 levels. It runs only when the agent has more than one queued run. The start-history read uses the same query in place of its old direct `createdByUserId` read, so the query count does not change there. - Recovery's 1-in-4 share is held only while a run a person asked for (`isPersonRequestedRun`) is ready, not for the rest of a person's tree. So a deep tree cannot hold recovery indefinitely. The dispatcher and the place-in-line replay use the same rule. - The claim backoff for a declined claim keeps the short person backoff only for runs a person asked for (unchanged behavior for tree runs). - `doc/heartbeat-internals.md`: new "Queued run dispatch order" section with the tiers, the order and the fairness bounds. Fairness bound: the oldest-run share is unchanged. Every 4th start goes to the longest-waiting ready run past 60 minutes, preferring runs people wait on. So another person's new task starts within 60 minutes plus 4 x (aged person-tier runs queued before it + 1) starts, even while a deep tree keeps adding in-progress or higher-priority work. A newer tree run never passes an older, equally ranked person task. Queues are per agent, and an agent belongs to one company, so no agent or company gets ahead of another. ## Verification - Red on master, green with this change: - `queued-run-dispatch-order.test.ts`: an older parent resume (`issue_children_completed`, person tree) starts before a newer person assignment. The place-in-line replay agrees. - `queued-run-dispatch-order.test.ts`: an older subtask of a person's task starts before a newer person assignment. - `queued-run-dispatch-order.test.ts`: fairness simulation. A deep tree adds two in-progress, higher-priority runs per start. Another person's new task starts within `QUEUED_RUN_AGING_MS` + `OLDEST_DISPATCH_SHARE` starts. - `execution-projection-agent-queue.test.ts` (embedded Postgres, real SQL): line is subtask (grandchild of a person's task), parent resume, newer person assignment, then work on an agent-created tree. Master gives the person assignment first and the tree runs last. - Negative and guard tests (green on both): work on an agent-created tree stays in the work tier behind a newer person assignment; system recovery on a person's tree stays in the recovery tier; newer tree work does not pass an older equally ranked person task; recovery still gets its share while only tree work waits. - Query plan on embedded Postgres with 300,000 issues and 200 queued tasks six levels deep: every step is an index scan on `issues_company_id_uq`, 1.5 ms execution. - `pnpm verify` exited 2 (a doc is a global CI input). Focused run: `queued-run-dispatch-order`, `queued-run-line`, `execution-projection-agent-queue`, `fresh-continuation-retry-delivery`, `assigned-work-stalled-queued-run`, `automatic-run-start-gate`, `automatic-run-brake-recovery-passes`, `heartbeat-stale-queue-invalidation`: 8 files, 173 tests passed. `tsc --noEmit` for the server passed. ## Risks - Behavior shift: in a company where people create most top-level tasks, most issue runs are now person tier. The "work" tier then holds agent-started trees, routines and runs without a task. Recovery keeps its share because only runs a person asked for hold it. - An in-progress run on a person's tree now ranks ahead of a new person task that is not in progress (readiness rank). The oldest-run share bounds that wait at about 60 minutes plus a few starts, the same bound agent work had before. - A tree deeper than 32 levels above a queued task is treated as not a person tree. It falls back to the work tier and the aging share. - The extra query runs only when an agent has more than one queued run, and only for queued tasks without `createdByUserId`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
db73178bcf |
fix(heartbeat): keep saved sessions across config edits the provider thread cannot see (#21862)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A heartbeat run resumes the agent's saved task session, which holds the provider thread and its history. > - Before resume, Paperclip compares a fingerprint of the effective run configuration with the fingerprint that the session saved. A difference discards the thread. > - The fingerprint hashed the agent config revision row (id, changed keys, time). Every saved agent edit therefore discarded the thread, also an edit that set a value back to what it was. > - The fingerprint also hashed settings that never reach the thread or the native execution input, for example turn timeouts, wake scheduling, and the environment row's update time. > - This pull request removes those inputs from the fingerprint, classifies every input with its reason, and adds backstop tests so that a new input cannot join or leave the fingerprint without a decision. > - The benefit is that agents keep their thread history across edits that cannot change what the thread saw or may do, while real changes still start a new thread. ## Linked Issues or Issue Description Related: #9977 (semantic task-session fingerprints, open) and #8797 (added the effective run config fingerprint). **What happened?** An operator changed an agent's warm idle window from 60000 to unset, and then back to 60000, between two turns on the same issue. The next run logged `Skipping saved session resume for task "<issue>" because effective run configuration changed: adapter.` Only the `adapter` category changed, because the new config revision row was in it. The adapter config itself was identical. The run started a new provider thread. The environment lease was reused, so the run stopped the retained remote task before it started a new one. That stop raced the task's exit, and the run failed with `Plugin Runner termination is unconfirmed; reconcile the saved task`. An automatic retry then succeeded on a fresh environment. **Expected behavior** An edit that cannot change what the provider thread saw or may do, and cannot change the native execution input, keeps the saved session. An edit to the model, provider, active permission mode, lifecycle mode or warm idle window, instructions, skills, environment, secrets, or another thread input still starts a new session. **Steps to reproduce** 1. Run an agent turn on an issue so that the task session saves its fingerprint. 2. Save an agent config edit that does not change a thread input, for example set `adapterConfig.idleTimeoutMs` to another value and back, or rename the agent. 3. Wake the agent on the same issue. On master the run log shows "effective run configuration changed: adapter" and the run starts a new thread. ## What Changed - New `server/src/services/heartbeat/session-fingerprint-inputs.ts`. It classifies each fingerprint input as kept or excluded, with the reason: - agent config revision fields (`CONFIG_REVISION_FIELDS`); - `paperclip_runner` adapter config keys; - top-level environment driver config keys; - agent runtime config keys; - selected-environment and environment-selection fields. - The session fingerprint no longer hashes these inputs: - the config revision row (id, changed keys, time); - for `paperclip_runner`: `timeoutSec`, `graceSec`, the permission keys of providers that are not selected, and `search` when the provider is not Codex; - `runtimeConfig.heartbeat` (wake scheduling, concurrency, daily caps, max-turn continuation); - the selected environment's `configRevisionAt` (row update time) and the `selectionSource`; - environment config `archiveOnRelease`, `streamRunLogs`, and `timeoutMs` (sandbox RPC timeout). - Unknown adapter and runtime config keys are still hashed. A missing classification can only keep the old reset behavior; it cannot skip a reset. - Lifecycle inputs stay in the fingerprint: `lifecycleMode`, `idleTimeoutMs`, and the environment's `runnerLifecycleMode`, `runnerIdleTimeoutMs`, and `reuseLease`. They become `session.lifecyclePolicy`, which the native session digest compares before a follow-up turn attaches to its warm Runner. A kept session with a different policy would be refused (`warm_recovery_refused=execution_mismatch`). - Session preparation accepts a saved fingerprint built with the previous composition (`legacyComposition`) for the same inputs. The upgrade therefore does not discard every saved session once. The run then saves the new fingerprint. - Backstop tests in `session-fingerprint-inputs.test.ts`: - every agent revision field, runner config schema field, provider permission key, and environment driver config key has a classification; - the shape of each hashed category is pinned, so a new field cannot reach the fingerprint without a decision; - every runner key classified as excluded leaves the native provider input and the lifecycle policy unchanged (this test fails if `idleTimeoutMs` is reclassified as excluded); - behavior tests for each exclusion and for the thread and lifecycle inputs that must still reset; - a pinned fingerprint proves that the legacy composition reproduces the master fingerprint exactly. - `ai-connection-session.test.ts`: two tests used a rename or a `heartbeat` edit as the "unrelated change". They now use a model change or an unclassified runtime setting, and two new tests pin that a rename and a scheduling change keep the session. - `doc/heartbeat-internals.md` (Config Freshness) describes the rule and the backstop. The PR adds no code comments, as `CONTRIBUTING.md` requires; the reasons live in the classification tables and the doc. ## Verification - Red/green: on master, the exclusion cases, the category-shape pin, and the legacy acceptance case fail. With the change, all pass. - Backstop check: with `idleTimeoutMs` temporarily reclassified as excluded, the native-input test and the idle-window reset test fail. - Legacy bridge: I computed the fingerprint for one runner fixture and one `codex_local` fixture with master's `workspaces.ts` and with `legacyComposition: true` on this branch. The output files were byte-identical, category fingerprints included. The pinned test uses those values. - `cd server && npx vitest run src/services/heartbeat/session-fingerprint-inputs.test.ts src/__tests__/ai-connection-session.test.ts src/__tests__/heartbeat-workspace-session.test.ts src/__tests__/heartbeat-execution-configuration.test.ts src/__tests__/ai-connection-router.test.ts src/__tests__/native-session-resumption.test.ts src/__tests__/environment-service.test.ts`: 7 files, 397 tests, all pass. - `npx tsc --noEmit -p server`: no errors. - `pnpm verify` exited 2 (broader coverage needed), so the focused checks above and required CI give the coverage. ## Risks - Behavior change: an agent rename, role, title, icon, appearance, reports-to, capabilities, budget, or metadata edit no longer starts a new thread. Agent tools read these fields live at call time (`runner-semantic-authority.ts`, `paperclip-runner-tool-authority.ts`), so the thread never holds a frozen copy. - Behavior change: wake-scheduling and turn-timeout edits no longer start a new thread. Neither value is part of the native execution input, so warm Runner attachment is unaffected. - A warm idle window edit still starts a new thread. Setting it and setting it back between turns no longer does, because the revision row is not hashed. - The legacy bridge computes extra candidate fingerprints only when the saved fingerprint differs. It accepts a legacy fingerprint only for the same current inputs; it never widens what a reset protects. - During the transition, a real change on a session that still holds a legacy fingerprint can list extra categories in the reset reason. The reset decision is correct. - No schema or API change. The fingerprint version stays 1. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1116.0-canary.9 |
||
|
|
20b0d4e5a5 |
fix(wake-queue): reopen a finished task for a person's comment merged into a system wake (#21914)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The wake queue holds new input for a task while an agent run is active, and releases it when the run ends > - A person's comment on a task always reopens the task when the run that closed it did not receive the comment > - A restart test found a person's comment that was posted while the run was finishing. The run marked the task done and no later run started. The comment was lost > - The comment joined a deferred child-completion wake that a system actor requested. The release step only looked at the wake requester, which was "system", so it did not see the person > - This pull request makes the release step look at who wrote the undelivered comments in the wake > - The benefit is that a person's comment is never dropped because it shares a queued wake with system work ## Linked Issues or Issue Description **What happened?** A person posted a comment on a task while the assigned agent's run was active. The run then marked the task done. The task stayed done, and no run received the comment. The deferred wake that held the comment was cancelled with "Deferred execution wake no longer applies to a terminal task". **Expected behavior** The task reopens and a new run receives the person's comment. **Steps to reproduce** 1. Start a run for a task. 2. While the run is active, let the system defer a wake for the same task (for example, a child task completes). 3. Post a comment as a person. The comment wake merges into the deferred system wake. 4. Let the run mark the task done. 5. The task stays done. No run receives the comment. **Paperclip version or commit** `master` at |
||
|
|
77ddf260bf |
fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid Runner Full-Stack E2E workflow proves chat features against a real source server > - Since master |
||
|
|
9de343b3fe |
perf(recovery): keep a repeatedly deferring task's saved row; stop agent settlement from re-checking its backlog (#21905)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues check assigned tasks for work that has no
live run and no queued wake. Since #21872 they load only changed rows,
and they save pending deferrals so a replaced server keeps their
deadlines
> - Two costs remained. A running task's check defers to now + 30 s, and
each new deadline rewrote its saved deferral row: about two writes a
minute per running task per queue
> - Also, two writers that settle an agent after its runs end set the
agent's status and `updated_at` but not `last_heartbeat_at`. The
changed-rows agent rule then reads the write as a pause or resume and
re-checks all of that agent's tasks for about 5 minutes
> - This pull request keeps a repeatedly deferring task's saved row
until its deadline is well past, and makes both writers record a
heartbeat as `finalizeAgentStatus` does
> - The benefit is near-zero deferral writes for running tasks and no
whole-backlog re-check after an abandoned or stranded run settles an
agent
## Linked Issues or Issue Description
Refs #21872, #21861
**What happened?**
1. In `recovery/work.ts`, a deferring check upserted its
`issue_recovery_deferrals` row whenever the deadline changed. A deferral
to now + 30 s changes on every check, so a task deferring for 20 minutes
wrote its row 41 times. At 1,000 concurrent agents that is roughly 30 to
70 writes a second.
2. `execution-control-reconciliation.ts` (abandoned run past its
execution-control deadline) and `heartbeat/agent-status.ts`
`reconcileStrandedAgentStatuses` (agent left "running" with no live run)
set the agent to idle with `updated_at` but without `last_heartbeat_at`.
The agent clause of `changedTaskCondition` matches an agent that is not
running and whose `updated_at` is newer than its last heartbeat. So each
such write made the backstop re-check every task of that agent for about
5 minutes.
**Expected behavior**
A task that keeps deferring writes its row a small fixed number of
times. Settling an agent after its runs end does not re-check the
agent's other tasks; the run's own task is still checked promptly.
**Steps to reproduce**
1. Defer one task's check to now + 30 s on every check for 20 simulated
minutes and count inserts into `issue_recovery_deferrals`: 41.
2. Seed an agent left "running" with 140 settled reviews, settle it with
`reconcileAbandonedExecutionControl` or
`reconcileStrandedAgentStatuses`, and count stranded checks of those
reviews over the next backstop: 280 per agent.
**Paperclip version or commit**
`master` at
|
||
|
|
ac22f7f42d |
chore(lockfile): refresh pnpm lockfiles (#21915)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
|
5561a4068c |
fix(recovery): keep a running run visible when a dropped continuation is newer (#21907)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task shows a Retry banner when its latest run failed and nothing
else continues the task
> - The banner reads "the latest run" by creation time only
> - The queue cancels a queued system continuation before it starts when
the task no longer needs it. That run is newer than the run that is
really working
> - So a task showed "failed" with Retry for about 79 seconds while its
run was working
> - The stranded-work sweep also queued one of those extra
continuations, because it did not see a native status wake that was
still waiting for dispatch
> - This pull request ignores never-started, superseded continuations in
the banner, and makes the sweep see a pending native status wake
> - The benefit is no false Retry offer while an agent works, and no
duplicate continuation after a slow finalization
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave (6 rolling deploys, 207 tasks), one task showed
state `failed` with Retry for 75–79 seconds while its run was running.
Two queued system continuations for that task were cancelled before they
started with `recovery_no_longer_needed`. That cancel is correct. But
the newest cancelled run became the "latest run" in the interruption
projection. The projection then fell through to `{ state: "failed",
canRetry: true }`.
The first of the two extra continuations came from a double queue. The
finishing run asked for its own continuation through a native status
wake (`agent_tool_call_loop`). Its finalization did not apply at once
("native result persisted but finalization did not apply; the
reconciliation loop will retry"), so the dispatch of that wake waited
about 17 seconds. In that window, the stranded-work sweep saw an
`in_progress` task with a succeeded run and no live path, and queued its
own continuation (`issue.productive_terminal_continuation_recovery`).
The sweep's live-path check counted only runs and
`deferred_issue_execution` wakes. It did not count a native status wake
that was still `queued` or `claimed` without a run. When the native
status wake was dispatched later, the queue correctly dropped it. The
dropped wake also carried the better instruction (the tool-loop
guidance).
**Expected behavior**
A task with a running run shows "working" with no Retry. One finished
run produces one continuation.
**Steps to reproduce**
1. Give a task a running run R1.
2. Add a newer run R2 for the task with status `cancelled`, no
`startedAt`, and error code `recovery_no_longer_needed`.
3. Read the task interruption. Before this change: `failed`, `canRetry:
true`, observed run R2.
For the sweep: give an `in_progress` task a succeeded native run and a
`native-status-committer` wake in `queued` or `claimed` with no run. Run
`reconcileStrandedAssignedIssues`. Before this change it queues a second
continuation.
**Paperclip version or commit**
master `c5218f5ec4` (wave), reproduced on `b4fa1c7fb1`.
## What Changed
- `task-interruption-presentation.ts`: the latest-run candidates skip a
run that is `cancelled`, never started (`startedAt` is null), and has
error code `recovery_no_longer_needed` or `issue_closed_before_start`.
The filter is in each per-arm `limit 1` index scan, so the query shape
and cost stay the same.
- `recovery/service.ts`: `hasActiveExecutionPath` also counts a
`native-status-committer` wake in `queued` or `claimed` with no run for
the issue. The native status wake dispatcher owns that wake, so the
sweep does not queue a second continuation and does not escalate the
task while it waits.
- Tests (red on master, green here):
- `task-interruption-query.test.ts`: a running R1 plus a newer
never-started R2 cancelled with each code shows `working`, `canRetry:
false`, observed R1. Negatives keep their behavior: a cancelled run that
had started, and a run a person cancelled, still show `failed` with
Retry.
- `heartbeat-process-recovery.test.ts`: a succeeded native run with its
own status wake still `queued` or `claimed` gets no second continuation
and no escalation.
## Verification
- `pnpm verify` exited 2 (opaque setup dependency), so I ran focused
suites:
- `server/src/__tests__/task-interruption-query.test.ts`,
`task-interruption-automatic-path.test.ts`,
`issue-confirm-run-stopped-routes.test.ts`: 83/83 pass. This includes
the query-plan test that bounds `heartbeat_runs` rows read.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`,
`server/src/services/recovery/*.test.ts`, the interruption suites and
`task-interruption-presentation.test.ts` together: 18 files, 871/871
pass.
- Red/green: the new interruption tests fail on master with `state:
"failed", canRetry: true` and the dropped run as observed run. The new
sweep tests fail on master with `continuationRequeued: 1`.
- `tsc --noEmit` for `server` passes.
## Risks
- Low. The banner change only hides runs that never started and were
dropped as superseded. Person-cancelled runs, started runs, and other
cancel codes behave as before.
- The live-path change makes the sweep wait while a native status wake
is pending. The dispatcher always settles such a wake: it dispatches it,
or marks it skipped. A throttled wake stays queued only until its
throttle time, so the sweep waits at most that long.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
698c371b1b |
fix(ui): keep new Recents tasks ready to open from cache (#21904)
Track warmed tasks per query client, company, and account so tasks added to the top Recents rows receive the existing idle preload. Preserve the batch limit, cache ownership, and retry-on-revisit behavior. Add unit and browser regressions for later additions, held API responses, editor retention, and message position. Extend controlled performance coverage and document timing results. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1116.0-canary.8 |
||
|
|
991f773d7f |
fix(ui): align recent task status icons and move fold controls right (#21859)
## Thinking Path > - Paperclip helps people manage AI agents for work. > - The sidebar shows recent tasks and their subtasks. > - A left fold control moves parent tasks away from the status column. > - Status icons must stay in one column so people can scan tasks. > - This pull request moves the fold control to the right. > - The child guide now starts below the parent status icon. ## Linked Issues or Issue Description Follow-up to #21649. Related sidebar alignment work: #20926. Related experimental Recents work: #12314. This change corrects the status column in the shipped subtask group. **What happened?** Parent tasks with subtasks moved 20 pixels to the right. Their status icons did not align with tasks that had no subtasks. **Expected behavior** All top-level task status icons align. The child guide is centered below the parent status icon. The fold control is on the right. **Steps to reproduce** 1. Open a sidebar with a recent parent task and a task that has no subtasks. 2. Compare their status icons. 3. Expand the parent and compare the child guide with its status icon. ## What Changed - Remove the left offset from parent task rows. - Put the fold control in a separate right slot. - Center the child guide below the parent status slot. - Add browser checks for alignment, narrow rows, unread parents, both themes, and separate Classic menu and fold controls. ## Verification - The browser regression failed before the fix: the status icons differed by 20 pixels. - Final browser run: 44 checks passed. It covers 20 states in both themes, folding, paging, unread state, light and dark alignment, and the Classic action menu. - The focused component and request-performance suites passed: 46 tests. - Final source is based on master `29903bed59b1`. UI typecheck and the focused browser-test typecheck passed. The published head is `b8287aec1f55aa182b44622d72b20fe7ed7ce0e6`. Its tree matches the tested local tree `958e8d901ed1fec9ae17c164271eda4d25edca45`. - `pnpm verify --base origin/master` returned exit 2. It requires broad CI because the browser suite changed. Focused checks pass. [CI 43803650911](https://github.com/paperclipai/paperclip/actions/runs/43803650911) passes typecheck, build, all eight E2E shards, policy, Runner checks, all nine serialized server suites, all twelve general server shards, and [final verification](https://github.com/paperclipai/paperclip/actions/runs/43803650911/job/133095032900). All 62 check runs are clean: 57 success, 3 skipped, and 2 neutral. No checks are pending or failed. The legacy Snyk status also passes. - [Greptile review](https://github.com/paperclipai/paperclip/pull/21859#issuecomment-6907272673) is 5/5 on head `b8287aec1f55aa182b44622d72b20fe7ed7ce0e6`. The check passed. There are no review threads or actionable findings. The separate source review also reports 5/5. - Screenshots use production components with HTTP and session fixtures in Chromium on Linux. No deployed-instance or image-baseline comparison is claimed. - No database or API behavior changes. ## Risks Low risk. The change adjusts layout in the shared task group. The fold button remains separate from the task link and the Classic action menu. Exact unread counts and request limits keep their current behavior. ## Model Used OpenAI Codex, GPT-6. Exact deployment model ID was not supplied by the runtime. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work (this is a sidebar layout bug fix) - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and has no internal task ID - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no command or subsystem behavior change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
b4fa1c7fb1 |
perf(recovery): load changed rows, not every task, for company hints, boots and reconnects (#21872)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery queues (stranded assignments, assigned-work
invariant, resolved dependencies, stale issue locks) check tasks to find
work that has no live run and no queued wake
> - A plain company hint, or an unscoped `"*"` reload, made each queue
load and check every eligible task of the company, at about 22 read-only
statements per task
> - Many writers send that hint (issue creation, documents, budgets,
work intents, agent status changes), every server boot reloads every
task, and the notification connection publishes a `"*"` reload each time
it opens or reconnects
> - On staging this shows as bursts of about 1,400 to 1,550 checks for
one company roughly every 9 minutes, with almost no real work
> - #21861 removed one writer. This pull request fixes the class: a
company hint, a boot and a reconnect now load only changed rows since
the queue last looked, and the hourly full pass is saved so a replaced
server keeps its schedule
> - The benefit is that settled tasks are checked by the hourly full
pass and by real changes, not by every write, deploy or reconnect
## Linked Issues or Issue Description
Refs #21861, #21831, #21867
**What happened?**
A company hint `{ companyId }` on a task queue loaded every eligible
task of the company with no changed-rows filter. The same happened for a
`"*"` reload. Three sources send these all the time:
1. Each server boot starts every task queue with a `"*"` reload.
2. `reconcileDatabaseWorkNotifications` publishes a `"*"` reload to
every queue each time the deployment notification connection opens or
reconnects. The connection opens at boot and again whenever a native
execution starts after an idle period.
3. Many writers send a bare company hint: issue creation, document lock,
unlock, restore and delete, budget pause and resume, work intent
promotion, agent status changes, watchdog writes, and others.
In a staging-shaped fixture each server replacement caused two full
reloads (boot and connection open).
**Expected behavior**
A company hint, a boot or a reconnect re-checks the tasks whose rows
changed. A change that makes tasks eligible without writing any of their
rows (a company or project resume, a tree hold release) still re-checks
every task of the company. An agent status change re-checks that agent's
tasks. The hourly full pass stays the safety net, also across server
replacements.
**Steps to reproduce**
1. Seed one agent with settled in-review tasks and a live native run.
2. Start the recovery workers with
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Restart the workers, call `reconcileDatabaseWorkNotifications(db)`,
or create an issue.
4. See one `check` event per settled task with `wake: "reload"` or
`wake: "company"`.
**Paperclip version or commit**
`master` at
canary/v2026.1116.0-canary.7
|
||
|
|
278660149c |
test(recovery): bound every wait in the agent-row census and stop it starting real runs (#21867)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery census tests prove that recovery queues do not re-check
settled tasks without a reason
> - The agent-row census from #21861 stopped producing output in a
server test shard and ran until the 20-minute job limit
> - Its tests let real agent runs start while `setTimeout` was faked,
and no wait in the file had a bound (the test timeout was 30 minutes)
> - This pull request stops the census from starting real runs, bounds
every wait with a real timer, and makes the file much faster
> - The benefit is that the file cannot hang a shard, and a stuck step
fails in seconds with a message that names the step
## Linked Issues or Issue Description
Refs #21861
**What happened?**
In a 10-shard server test run,
`server/src/__tests__/issue-recovery-agent-row-write-census.test.ts`
finished its third test and then wrote no more output until the job
reached its 20-minute limit. The file had a 30-minute timeout per test,
so the hang never failed as a test. Several tests let real agent runs
start while `setTimeout` was faked:
- the idle-agent guard marked a live run succeeded while its task stayed
in progress, so recovery continued the task;
- the hold-release guard released an agent that had a `todo` task;
- the pause and terminate guards stopped agents with live native runs.
A run execution that waits on a faked timer, or holds a lock that a
delivery pass then waits on, does not end. A local run of the old file
showed 4 run executions ("heartbeat execution setup failed").
**Expected behavior**
The census never starts a real run. Every wait has a bound and a message
that names what it waited for. The file finishes well under 60 seconds.
**Steps to reproduce**
1. Run the old file and search its output for "heartbeat execution setup
failed": 4 run executions.
2. Run it in a slow, shared CI shard: the file can stop with no output
until the job limit.
**Paperclip version or commit**
`master` at
canary/v2026.1116.0-canary.6
nightly/v2026.1116.0-nightly.0
|
||
|
|
94cb7ad359 |
fix(recovery): stop an agent configuration write from re-checking every task of the company (#21861)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery queues check each assigned task to find work that has
no live run and no queued wake
> - On a nearly idle hosted instance, these queues still re-check almost
every task of a company in bursts, at about 22 read-only statements per
task
> - #21831 stopped a run start from re-checking all of its agent's
tasks, but bursts continued after writes to an agent row during a long
run
> - The cause is not the changed-rows backstop. An agent configuration
write publishes the company-wide task recovery hint, so every task queue
reloads and checks every assigned task of the company
> - This pull request makes an agent configuration write signal only the
company-level recovery queues and the run dispatch queues, not the
per-task queues
> - The benefit is that a configuration write during a long run no
longer re-checks hundreds of settled tasks
## Linked Issues or Issue Description
Refs #21831
**What happened?**
A write to an agent's configuration (for example `PATCH /agents/:id`, an
instructions save that rebinds the entry file, a skill or connection
change) calls `notifyRunDispatch({ companyId })`. That function sends
the company hint to the task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`,
`stale-issue-locks`). A company hint is not a backstop hint, so the
queue loads every eligible task of the company with no changed-rows
filter and checks each one. In a staging-shaped fixture (780 settled
reviews), each configuration write caused 780 checks and about 16,000
statements. The agent row keeps status `running` and its old
`last_heartbeat_at`, which matches the writes seen on staging.
**Expected behavior**
A configuration write does not re-check tasks. Only a change that can
strand a task (pause, resume, hold release, termination) re-checks
tasks, and those changes keep their own lifecycle hint.
**Steps to reproduce**
1. Seed one agent with 700 settled in-review tasks and a second agent
with 80, each with one live native run.
2. Start the recovery workers with the debug flag
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:1`.
3. Update the agent's `runtimeConfig` or `adapterConfig` through
`agentService(db).update`.
4. See 780 `check` events with `wake: "company"`.
**Paperclip version or commit**
`master` at
canary/v2026.1116.0-canary.5
|
||
|
|
29903bed59 |
fix(ui): add a space after composer autocomplete (#21754)
Preserve an ordinary space after composer autocomplete selections and place the caret after it. Cover all seven completion types and queries with trailing spaces. Co-Authored-By: Paperclip <noreply@paperclip.ing>canary/v2026.1116.0-canary.4 |
||
|
|
f7a3497e9d |
fix(evals): compare browser zones by clock, wait for the reloaded profile name, bind accounting reads by returned run (#21760)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog runs real agent conversations and grades the saved outcome > - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three failures that were not product failures of the chat agent > - One setup check compared time zone names, one observer read a form before it filled, and one grader accepted only exact query digests > - An eval that fails valid behavior hides real regressions behind noise, so each check must grade the outcome, not an incidental detail > - This pull request fixes the three checks, and it fixes a first-frame empty display name on the Profile page > - The benefit is that these three cases now fail only for the behavior they test ## Linked Issues or Issue Description **What happened?** Three chat-eval cases failed in cohort 16 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1) `activity.routine-audit`: setup threw "Browser timezone override did not apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects `America/Argentina/Buenos_Aires`, and Chrome reports that zone by its alias `America/Buenos_Aires`. The check compared names. (2) `navigation-preferences.edit-profile`: the name saved correctly, but the observer read the display-name field once, right after reload, and got an empty value. (3) `budgets-costs.spend-breakdown`: the answer was correct, but the grader accepted company cost reports only for `period=all` or `period=month`. The agent read `costs/by-agent` and `costs/summary` for the requested week, and that range covers the unpriced run. **Expected behavior** Each case passes when the product and the agent do the requested work. Each case fails when they do not. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.activity.routine-audit` between 11:00 and 11:59 UTC. 2. Run `chat-features.runner-opencode.local.navigation-preferences.edit-profile` and look at `profileAfterReload`. 3. Grade the retained cohort 16 observation of `budgets-costs.spend-breakdown` with the old grader. **Additional context** The run tool inputs are saved only as SHA-256 digests, so a grader binds a read by recomputing the digest of inputs it can check. For the cohort 16 spend-breakdown reads, the inputs were recovered this way: `from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`. ## What Changed - `internal-routine-audit.ts`: the browser override check compares the UTC offset of the requested and reported zones, now and 182 days later, instead of the zone names. An unknown zone or a different clock still fails setup. - `identity-profile.ts`: after reload, `settledFieldValue` waits up to 15 s for the display name to reach the saved value, then records what the person sees. The grade still fails a name that does not arrive. - `ProfileSettings.tsx`: unedited fields show the session's saved name and image from the first frame. Before, a session restored from the saved read cache rendered an empty name until an effect filled it. - `identity-accounting.ts`: `readSavedAccounting` also accepts a company cost report whose checked `from`/`to` covers the unpriced run's cost event. The run log keeps only the input digest, so the grader recomputes digests for covering ranges: UTC day boundaries (with or without milliseconds or seconds, date-only, end of day), local midnights at each whole-hour offset, and the run's own `startedAt` and cost-event instants. A pending run (no cost event) counts only through `costs/summary`, `costs/data-quality` or its own run record, because the other reports sum cost events. A link to the run does not count, because the run id is in the task description. The run-log redaction is not changed. - Oracle versions bumped: `routine-audit.v7`, `spend-breakdown.structural.v6`, `identity-profile.v4`. ## Verification - Red/green: each new test fails on the old code and passes on the new code. - `internal-remaining.test.ts`: every candidate zone, reported by its own name, its ICU name and known browser aliases, passes at every hour in January and July. A different clock, a zone with daylight saving time and an unknown name fail. - `identity-personal.test.ts`: the settled read returns the saved name after empty reads, and returns a wrong name when the saved name never arrives. - `ProfileSettings.test.tsx`: the first rendered frame with a cached session shows "Alex Chen", not `""`. - `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as do other covering ranges. Negative controls fail: no reads, run-list reads with unchecked inputs, ranges that exclude the run, another company, a denied read, a read by another run, and a read of a different run record with a link to the target run. - The retained cohort 16 spend-breakdown observation passes the new grader, and fails with its reads removed. - `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed. `scripts/check-token-gates.mjs` is clean. - One narrow paid run of the three cases on this branch (result in a comment below). ## Risks - Low risk. The changes are in eval code, plus a small Profile page state change. - A covering range in a form outside the recomputed set (for example 08:00 at -03:00) is not recognized. That is a false negative, not a false pass. - The Profile page now keeps an unedited field in step with the session. A refetch still resets drafts, as before. ## Model Used - Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c5218f5ec4 |
test(restart-chaos): count distinct warm close failures and let R5 wait out environment acquire (#21852)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) runs zero-downtime
gate waves and grades each wave and each task
> - Gate wave A1 of rerun 6 (w-20261116T001158-s7106) failed. Its "warm
sessions that did not close" count was 28, but only 2 sessions failed to
close. Task t010 also failed R5, but its run was healthy.
> - The harness's environment-delete retry loop re-ran one failing close
every 3 s, so one failed session was logged 27 times. The harness
counted log lines, not failures. Also, no sandbox capture existed from
the time of the failed idle close, so the cause was not recorded.
> - R5 saw 253 s of silence on t010's run aedc1f6e. The run spent 251 s
in `environment.acquire` (a Daytona restore of an archived sandbox, with
one 90 s plugin call timeout). The "run started" event is written only
after acquire, so the run had no event during that time. The run then
succeeded.
> - This pull request makes the warm-shutdown grade count distinct (run,
reason) failures, captures the sandbox at the first idle-close failure,
and lets R5 wait up to 10 minutes while a running run has no run event
yet.
> - The benefit is a wave grade that shows the real number of failed
sessions, records why a close failed, and does not report a slow
environment acquire as a dead run.
## Linked Issues or Issue Description
**Bug description**
Three findings from gate wave A1 of rerun 6 (w-20261116T001158-s7106):
1. `warm-shutdown.ts` reports one failure per log line. The wave showed
29 shutdown close failures (28 "did not close" lines and 1 "cleanup
failed before controller shutdown" line) for 2 sessions: run 300f0d61 (2
lines) and run 70d909cc (27 lines from the environment-delete retries).
No sandbox capture was taken when the idle close of 70d909cc failed, so
why the Runner did not suspend is not known.
2. R5 reported `silent_dead_run aedc1f6e… 181s` for task t010. The run
was in environment acquire for 251 s. It then succeeded.
3. Task t024 failed its content check `subtask_1_done`. The child saved
its ledger with Read-tool line-number prefixes (`1\tROUND 1 ...`). The
grader requires lines that start with `ROUND`. This is model behaviour:
a ledger that the agent corrupts is a real content failure. This pull
request does NOT change that check.
**Steps to reproduce**
1. Grade the A1 server log lines (fixture
`fixtures/a1-warm-close-failures.log`) with `gradeWarmShutdown`. The
result has 29 `shutdownCloseFailures` entries.
2. Grade the t010 samples (fixture
`fixtures/t010-slow-environment-acquire.json`) with `gradeR5`. The
result is `fail: silent_dead_run aedc1f6e…`.
**Expected behavior**
1. The grade shows 2 distinct shutdown close failures, each with its
line count, and the verdict stays "fail".
2. R5 passes a run that has no run event yet for up to 10 minutes. R5
still fails a run with no event past 10 minutes, and a started run that
is silent past 3 minutes.
## What Changed
- `warm-shutdown.ts`: `shutdownCloseFailures` and `idleCloseFailures`
are now lists of distinct `{ runId, reason, lines, firstLine }` entries,
keyed by run and failure reason. The reason drops the JSON detail after
the error text (for example timings), so retries of the same failure
merge. A line with no parsable run keeps its own message as the reason.
Each distinct shutdown failure still adds a problem, so the verdict is
unchanged in meaning: any distinct failure fails the wave.
- `warm-shutdown.ts`: new `watchFirstIdleCloseFailure` splits server
output into lines per stream and reports the first idle-close failure
once. New optional `sandboxCaptures` input and output, listed in the
markdown section.
- `supervisor.ts`: new `onServerOutput(chunk, source)` hook, called for
the primary and the second server's stdout and stderr. A hook error is
recorded in the timeline and never breaks logging.
- `main.ts`: on a Daytona warm wave, the first idle-close failure
triggers one `captureRunnerLogs("idle-close-failure-<run>")` (the class
R probe: processes, sockets, Runner runtime files including
`runner-state.json`, and logs). The harness waits for that capture
before the environment cleanup and before the warm-shutdown grade.
- `observer.ts` / `types.ts`: a running run whose events read back empty
(the read succeeded and there is no event yet) gets `preparing: true` in
its sample. A failed events read does not set it.
- `grader.ts`: new `PREPARATION_LIMIT_MS` (10 minutes). `deadRunningRun`
uses it when the run sample has `preparing`.
- Why 10 minutes: the observed acquire took 251 s, including one 90 s
plugin call timeout. The Daytona provider bounds each start or restore
at 5 minutes by default (`DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS`), so 10
minutes covers one timed-out restore and its retry. This is the same
bound #21730 gave an active provider turn.
- New fixtures: `fixtures/a1-warm-close-failures.log` (the 34 real
close-failure lines of the A1 server log) and
`fixtures/t010-slow-environment-acquire.json` (the 87 real t010 samples
of run aedc1f6e, with the time of its first run event).
- `README.md`: document the distinct-failure count, the idle-close
capture and the preparation bound for R5.
- t024: no change. Decision: the ledger check stays strict, because a
ledger that the agent saved with line-number prefixes is a real content
failure.
## Verification
- Red on `origin/master` sources, green with this change (9 tests):
- `warm-shutdown.test.ts` › "reports each distinct (run, reason) once,
not each log line (w-20261116T001158-s7106 A1: 2 sessions, 29 lines)": 2
failures (300f0d61 with 2 lines, 70d909cc with 27 lines), 2 problems,
verdict fail.
- `warm-shutdown.test.ts` › "still fails on one distinct failure, and
keeps different reasons for the same run apart" (guards: the 27 lines of
one session still fail; two reasons for one run give two problems; an
unparsable line still fails).
- `warm-shutdown.test.ts` › "names the first idle-close failure once,
across chunk boundaries and interleaved streams" (a forced-stop line
does not trigger it).
- `warm-shutdown.test.ts` › "lists the sandbox captures taken at the
first idle-close failure", plus the two updated shape tests.
- `triggers.test.ts` › R5 live evidence › "does not flag a run that is
still acquiring its environment before its first run event
(w-20261116T001158-s7106/t010)": the real samples still fail without
`preparing`, and pass with `preparing` on the samples before the first
run event.
- `triggers.test.ts` › R5 live evidence › "still fails a run silent past
the preparation bound, and a started run silent past the stall bound"
(guards: 9.5 min preparing passes; 10.5 min preparing fails; 4 min quiet
without `preparing` fails).
- `triggers.test.ts` › R5 live evidence › "marks a running run as
preparing only while its events read back empty" (the flag clears after
the "run started" event; a failed events read never sets it).
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71
files, 833 tests pass.
- `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- No wave was run. The live idle-close capture path in `main.ts` is not
exercised by a unit test.
## Risks
- Low risk. The change touches only the test harness.
- `warm-shutdown.json` changes shape: `shutdownCloseFailures` and
`idleCloseFailures` are now objects, not strings. Scripts that read
these fields as strings must read `firstLine`. Summary counts now show
sessions, not lines, so they are lower than before for the same log.
- A run that dies before its first run event now fails R5 after 10
minutes instead of 3. The fault is still caught, but later.
- The idle-close capture can add up to 45 s (the capture deadline)
before the environment cleanup on a wave that had an idle-close failure.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1116.0-canary.3
|
||
|
|
d149e30323 |
fix(native): stop a finished warm owner's retained Runner when suspend is refused, and release a suspended Runner whose process left (#21853)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes an idle session after 5 minutes, and closes it again for an environment delete, idle sleep, or a restart. > - In a zero-downtime release-swap test, two warm closes failed and did not recover. > - In the first case, the Runner of a finished run refused `runner.suspend`. The server kept the Runner alive and the session registered. An environment delete returned 409 thirty-five times, and only an outside sandbox destroy released the session. A live Runner and a paid sandbox were held, and idle sleep was blocked. > - In the second case, the Runner durably suspended and exited, but its process check returned exit 44 (`process_identity_changed`) on every attempt. Exit 44 also covers "a live process without `--runner-id`", so no proof could release the owner. > - Also, the bounded idle retries from #21782 did not run for the first case. > - This pull request stops the retained Runner of a finished owner with the verified port-holder retirement, releases the owner only when a fresh check reports `process_not_running`, splits exit 44, accepts a durably suspended Runner with no live process that carries its runner id, and keeps the idle retries alive after a failed sweep. > - The benefit is that a finished warm session can no longer hold a live Runner and a sandbox until someone destroys the sandbox. ## Linked Issues or Issue Description Refs #21782 Refs #21835 **What happened?** Release-swap gate wave, two warm-session close failures: 1. A finished run's idle close failed with `native_session_close_unrecoverable`: `runner.suspend` failed, `runnerRetained: true`, Runner lifecycle `ready`. Only close attempt 1 was logged. An environment delete with `forceStopUnsettled` could not discard the session, because `closeWarmNativeSession` discards only with a Launcher task owner or a suspended Runner. The delete got 409 thirty-five times. The session was released only after an outside sandbox destroy. 2. A Runner's `runner-state.json` showed lifecycle `suspended`, with drain and suspend `completed`. The process check exited 44 on every attempt, and the Runner process was gone later while the sandbox stayed started. **Expected behavior** A finished owner whose Runner refused suspend is stopped and released, and a failed stop keeps it held. A durably suspended, settled Runner with no live process that carries its runner id releases its owner. A bare `process_identity_changed` never releases an owner. **Steps to reproduce** 1. Keep a warm sandbox session idle after its run succeeds. 2. Make its close fail with `NativeSessionCloseUnrecoverableError` with `runnerRetained: true`. 3. Call `closeWarmNativeSessionsForEnvironment({ forceStopUnsettled: true })`. On master the result is `failed: 1`, every time. **Why the retries did not run** - The transport memoizes `close()`. A second close returns the same rejected promise and does no sandbox work. The 27 environment-delete failures have identical timings (`totalMs: 6055`), and the other Runner's daemon log shows no sandbox commands during its three retries. So a retry can only succeed through a fresh proof, not through the close. - `closeIdleWarmNativeSessions` (environment delete, idle sleep, restart) clears the entry's idle timer, which is also the pending retry timer, and never re-arms it after a failed close. The test "keeps retrying a failed idle warm close after a failed environment sweep cleared its retry timer" is red on master for this reason. - The log shows one more `close()` request on the same transport in the same second as attempt 1, and no attempt 2. Every path that closes an idle owner from outside the timer clears that timer. The log does not show which caller it was, so this part is not proven. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - A warm entry keeps its sandbox runner and remote directory when the run used a sandbox that the server reaches directly (no Launcher task). - `warmCloseFailureProvesOwnerGone` adds two proofs: - The close failed with `NativeSessionCloseUnrecoverableError`, `runnerRetained: true` and not suspended. The server stops the session's own Runner with `retireSandboxRunnerPortHolders` (owner run must be `succeeded` or `cancelled`, same agent, idle state; TERM, then KILL). It releases the owner without a checkpoint only if a fresh process check then exits 45 (`process_not_running`). A stop that cannot be confirmed keeps the owner. - The close failed with the new `process_runner_id_missing`. A fresh sandbox inspection must show lifecycle `suspended`, settled state (empty outbox, no pending terminal delivery, provider cleanup, or warm transition), matching run, session and runner ids, and no process that carries `--runner-id <id>`. - Both the idle-timer failure path and `closeIdleWarmNativeSessions` (environment delete with `forceStopUnsettled`, idle sleep, restart) use these proofs. - The first Runner-check or refused-suspend close failure for a sandbox owner logs one evidence line: stage, reason, Runner lifecycle, suspend command status, proof window, and an inspection of the sandbox (durable lifecycle, settled, identity match, marker pid, and pid/state of each `paperclip-runnerd` or runner-id process). It logs no nonce and no command line. - A sweep that fails to close an owner whose idle close already failed re-arms the bounded idle retry. - `server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`: `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT` exits 48 (`process_runner_id_missing`) for a live process whose command line lacks the runner id. Exit 44 now means only an identity file mismatch. - `server/src/services/native-runtime/sandbox-runner-port-holders.ts`: `retireSandboxRunnerPortHolders` takes `only: { sessionDirectory, runId }` to retire one session's Runner. New `sandboxSessionRunnerExited` and `inspectSandboxSessionRunner` (a read-only `node` script). ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts src/services/native-runtime/sandbox-runner-port-holders.test.ts src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts src/services/runner-lost-mid-turn-continuation.test.ts src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts` - Red on master (master sources with these tests): "stops the retained Runner of a finished warm owner that refused suspend ..." returns `failed: 1`. "keeps retrying a failed idle warm close after a failed environment sweep ..." sees 2 closes, not 3. "releases at controller restart a warm owner whose Runner durably suspended ..." returns `failed`. - Negative tests: a Runner that survives TERM and KILL stays `failed: 1` with no exit check. A run that is still `running` is never signalled. A live process that still carries the runner id, and a bare `process_identity_changed`, keep the owner. - Script tests: the process check exits 48 for a live process without `--runner-id`, 0 with it, and 44 for a changed marker. The inspection script reports state, marker pid and runner processes, never the nonce or a command line, and fails closed without `/proc` or through a symlink. The port-holder integration test stops only the named session's finished Runner (a real local process) and then proves it exited. - `pnpm verify` exited 2 (opaque setup dependency), so focused checks and required CI give the coverage. `npx tsc --noEmit -p server`: no errors. ## Risks - A finished owner's retained Runner is now stopped when its suspend was refused. The session is released without a new checkpoint. The last saved checkpoint stays the recovery source, which is the same as an environment delete today. - The stop uses the same rules as a successor's port-holder retirement: finished owner run, same agent, idle Runner state, and a verified process identity. Any doubt keeps the owner. - The stop can take up to about 20 s (TERM and KILL waits) inside a sweep, while the entry stays fenced. - The first close failure of a sandbox owner runs one read-only inspection command (5 s limit). - Previous-release story: no persisted state or launch identity changes. The scripts are sent with each command, so a server always runs its own exit-code table. Exit 44 from the previous release maps to `process_identity_changed`, which still never releases an owner. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1116.0-canary.2 |
||
|
|
49a2b7b245 |
test(recovery): record a live process's OS-observed start time in owner-proof tests (#21847)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery continues a cut-off task automatically. It first lets a previous process that still runs on this host settle, for about ten minutes at most. > - A server test proves that wait. It failed once on master atcanary/v2026.1116.0-canary.1 |
||
|
|
03309f1650 |
fix(native): release an idle warm owner whose sandbox Runner port a later run took over (#21835)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes idle sessions, and closes them again before it exits. > - A sandbox has one Runner port. Before a new run launches its Runner on a reused sandbox, it retires every other Runner on that port. > - In a zero-downtime release-swap test, a run finished on the outgoing server and stayed warm. 77 ms later, a follow-up run started on the incoming server. It reused the same sandbox, started a fresh provider session, retired the old Runner, and launched its own Runner on the same port. > - The outgoing server's idle close could not suspend the old Runner. Its checkpoint was correctly refused with `native_runner_checkpoint_owner_lost`. The idle timer retried 3 times and then kept the owner registered. That phantom owner blocked idle sleep for 4.8 minutes, and the shutdown close failed again. > - The existing proofs that a warm owner is gone (#21782) do not count "a later run launched its own Runner on this sandbox". > - This pull request adds that proof. When it holds, the server forgets the owner without a checkpoint. The checkpoint fence does not change. > - The benefit is that a superseded warm owner no longer blocks idle sleep or fails a clean shutdown. ## Linked Issues or Issue Description Refs #21627 Refs #21782 **What happened?** An idle warm owner finished on the outgoing server. A follow-up run on the incoming server reused the same sandbox and launched its own Runner on the sandbox's single Runner port. The owner's idle close failed with `native_runner_checkpoint_owner_lost` 4 times. The owner stayed registered, held the idle-work flag, and blocked idle sleep. At SIGTERM the restart sweep logged "idle warm native sessions did not close" with "runner did not durably suspend before checkpoint". No task was affected. **Expected behavior** A failed idle close releases its owner when a later run proves that the owner's Runner is gone. The checkpoint stays refused while the successor is live. **Steps to reproduce** 1. Finish a warm run on a sandbox and keep it idle. 2. Start a successor run for the same task on another server. Give it a lease on the same sandbox, its own `native.process_identity_recorded` event and its own `session.started` event. 3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays `present`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: new `warmRunnerPortTakenBySuccessor`, used by `warmCloseFailureProvesOwnerGone` after the existing sandbox-termination proof. It holds only when another run has all of these: - an `environment_leases` row with the same environment and `providerLeaseId` as the owner; - its own `native.process_identity_recorded` event (`source_event_id is null`, so an inherited identity does not count); - a `session.started` event from its own Runner instance (`source_instance_id` equals the run's `runner_instance_id`); - both events created after the owner run's `finishedAt`. - When the proof holds, the idle-timer path and the controller-restart path log "warm native session's Runner or sandbox is gone; released its owner" and forget the owner without a checkpoint. Both paths already call `warmCloseFailureProvesOwnerGone`. - `validateNativeRunnerCheckpointOwner` does not change. A checkpoint still cannot land under a live successor. - Cost: the proof runs only after a failed close. It does one primary-key read of the owner run and one query on `environment_leases.provider_lease_id` (indexed), joined to `heartbeat_runs` by primary key, with `heartbeat_run_events` read through `(company_id, run_id)`. - New real-database test file `server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`. - `native-session-executor.test.ts`: one environment-delete test counts the extra proof read when the first proof does not hold. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`: 6 passed. - "releases an idle warm owner whose checkpoint is refused because a successor on another server launched its own Runner on the same sandbox: own_runner": the close is refused with `native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed: 0`, the idle-work flag is `none`, the owner has no checkpoint row, and `assertNativeRunnerCheckpointOwner` still throws `native_runner_checkpoint_owner_lost`. - These stay failed and keep the owner registered: the successor is on another sandbox; the successor has only `native.process_start_requested`; the successor has only inherited events (`source_event_id` set); the successor's events come before the owner finished. - "releases a superseded idle warm owner from its idle timer without writing a checkpoint": the idle timer releases the owner after the first refused close. - Red on master (test file only, fix reverted): `own_runner` fails with `closed: 0, failed: 1`; the idle-timer test fails with the idle-work flag still `present`. The 4 negative cases pass on master and with the fix. - `cd server && npx vitest run src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`: 9 passed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: all passed. - `cd server && npx tsc --noEmit -p .`: no errors. - `pnpm verify`: see the comment below for the result. - Not run: a restart-chaos wave. The release-swap timing is reproduced in the database test, not on a live sandbox. ## Risks - A superseded owner is now released without a checkpoint. Its old provider conversation cannot be resumed. The successor had already started a different conversation and holds the task, so no one would resume the old one. - The proof needs the successor's own Runner identity and own session start after the owner finished, on the same sandbox lease. A successor that reused the owner's Runner, only requested a start, or recorded its events earlier does not count. In those cases the close fails as before. - The proof adds two small indexed reads after a failed close only. The normal close path does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1116.0-canary.0 |
||
|
|
da8cfda42e |
perf(recovery): stop counting a run start as a change to every task of its agent (#21831)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues. The `stranded-assignments` queue checks assigned tasks that have no live run. One check costs about 22 database statements. > - #21729 made the 5-minute backstop re-check only tasks whose own rows changed. A change to the assignee agent's row counts as a change to every task of that agent. #21765 left out the agent's heartbeat writes. > - Execution start in `heartbeat.ts` writes `agents.status = 'running'` and `updated_at`, but not `last_heartbeat_at`. So each run start still makes every task of that agent "changed". The backstop window overlaps the previous one by 5 minutes, so each run start re-checks those tasks in two windows. > - On an almost idle hosted instance with one live run and a new run every few minutes, this gives bursts of about 1,450 checks for about 855 settled tasks, every few minutes. > - This pull request stops counting an agent row whose current status is `running` as a change to its tasks. > - The benefit is no re-checks of settled tasks beside a live run: 540 backstop checks in 30 simulated minutes drop to 6 (only the live task itself), and statements drop from 15,280 to 3,984. ## Linked Issues or Issue Description Refs #21729, #21765, #21795 No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The changed-tasks backstop of the `stranded-assignments` and `assigned-work-invariant` recovery queues (`server/src/services/recovery/changed-tasks.ts`). **Subsystem affected** Server: issue recovery background work. **Current behavior** On staging at near idle (one live native run, about six new runs in 30 minutes), the stranded-check statement bundle runs in bursts of about 10 checks per second for about 2.5 minutes, then stops, then starts again about 4 minutes later. The tasks do not change. The cause: every run start writes the agent row without a heartbeat time. The backstop then loads every open task of that agent as "changed", in two consecutive windows. **Proposed behavior** An agent row counts as a change to its tasks only when the agent is not `running`. A pause, a resume, a hold or a configuration change of an idle agent still counts. A run's own task still gets its run hints and an immediate targeted check. **Reason and benefit** Recovery work becomes proportional to activity, not to the number of tasks held by an agent that just started a run. ## What Changed - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent clause of `changedTaskCondition` now requires `ag.status <> 'running'`. A comment explains why run lifecycle writes are left out. - `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` (new): a staging-shaped census on the embedded test database with the real recovery workers and a fake clock. It seeds 800 settled in_review tasks with a succeeded latest run (3 workers, about 266 tasks each), 30 in_progress and 30 todo tasks without live runs, and one live native run. The run's agent gets the execution-start write, and the run publishes a run hint every 10 seconds for 30 simulated minutes. It also has three correctness guards: a task of the live agent whose own row changes without a hint is checked within the backstop bound; a resumed agent's task is checked within the backstop bound when the resume sends no hint; and the hourly full pass still checks every settled task while the run stays live. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: a fast test that a run-start write is not a change and a pause during the run is a change. ## Verification Red on `origin/master` (fix reverted), green with the fix: | 30 simulated minutes beside one live run | before | after | | --- | --- | --- | | stranded checks, backstop wake | 540 | 6 | | stranded checks, hint wake (the live task's own run hints) | 178 | 174 | | checks of settled tasks | 534 | 0 | | database statements | 15,280 | 3,984 | | work signals beyond the run's own hints | 0 | 0 | - `npx vitest run src/__tests__/issue-recovery-live-run-agent-census.test.ts` (4 tests): the census fails on master with `expected 534 to be less than or equal to 50` and passes with the fix. The three guards pass. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "run start"`: fails on master, passes with the fix. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts src/__tests__/issue-recovery-idle-backstop.test.ts src/__tests__/issue-recovery-activity-hints.test.ts src/__tests__/issue-recovery-signal-fanout.test.ts`: 31 passed, 1 skipped. - `tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Low risk. The change narrows one clause of the backstop query. No schema change. - If an agent is resumed and then starts a run in the same window, the resume no longer counts through the agent row. A resume already sends its own `pause_lifted` hint (`agent-lifecycle-driver.ts`), and the hourly full pass still covers every task. Before this change, a run finish already hid a resume the same way, because the finish writes a heartbeat time. - A configuration change of an agent while it runs is caught by its next idle write or by the hourly full pass, not by the 5-minute backstop. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8299f7c389 |
perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner that listens (`--listen-address`). > - In listen mode, the Runner checks for an incoming connection only once per `--reconnect-delay-ms` (default 250 ms). It waits that delay again after each disconnect, and it backs off after a failed handshake. > - That delay exists to pace redials in dial mode. A listener never dials. Core's connection waits in the kernel backlog until the next `accept`. > - Every listen-mode connect pays part of that wait. A warm run attachment pays it in full: the Runner disconnects at the authority rotation, and Core's new-authority connect then waits up to 250 ms or more. > - This pull request makes a listening Runner check for a connection every 10 ms. Dial mode keeps the configured reconnect delay. > - The benefit is about 0.4 s less on each warm run attachment and about 0.1 s less on each cold start, for every listen-mode Runner. ## Linked Issues or Issue Description **What happened?** A warm follow-up turn on a listen-mode remote Runner reached its first turn later than a cold turn on the same agent. In traces, a cold start makes one Core→Runner connect of 134–178 ms. Each warm attachment makes an old-authority reconnect (72–233 ms) and then a new-authority connect of 411–504 ms. This was 6 of 6 warm runs. **Expected behavior** A connect to a listening Runner completes in about one network round trip plus the WebSocket handshake. It does not wait for a reconnect timer. **Steps to reproduce** 1. Start `paperclip-runnerd` in listen mode with the default `--reconnect-delay-ms`. 2. Let it reach the disconnected wait (startup, a park claim, or a `run.attach` authority rotation). 3. Connect to the listener. The upgrade is not answered until the next 250 ms check. **Root cause** `durable/runner.rs` schedules the next `accept` at `now + config.reconnect_delay` when no connection is pending, after a disconnect, and as the base of the failure backoff. The Launcher-style hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies. The tests use a 1 ms delay, so they did not show the wait. ## What Changed - `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`. A listener returns `min(configured, 10 ms)`. A dial endpoint returns the configured delay without change. - `durable/runner.rs`: the four places that waited `config.reconnect_delay` (no pending connection, failure backoff base, two disconnect paths) now use `endpoint.retry_delay(config.reconnect_delay)`. - `DURABLE_TRANSPORT.md`: document the listen-mode connection check interval. - Add a unit test for the listener and dial delays. - Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in listen mode with the default reconnect delay. One connection drops before authentication, as a superseded route's redial does. The test then checks that the next upgrade is answered in under 120 ms. Before this change, that upgrade waited 188–313 ms. ## Verification - `cargo test -p paperclip-runner-core --lib` passes locally. - `cargo clippy -p paperclip-runner-core --all-targets` reports no new findings in the changed files. - `cargo fmt --all -- --check` passes. - New tests: `durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects` and `tests/listener_accept.rs`. - `tests/listener_accept.rs` fails on the previous `runner.rs` and passes with this change (3 runs each). - Existing listener and warm-attachment tests pass (`listener_*`, `warm_run_attachment_*`, `warm_external_attachment_*`). - I did not measure the change on a live host. The effect is visible in the next deployment as a shorter `runner.transport.registration` on warm attachments. ## Risks - Low risk. Dial mode does not change. - A disconnected listener now wakes every 10 ms instead of every 250 ms. Each wake is one non-blocking `accept` and a non-blocking host-control read. The loop already slept in 25 ms steps. - Failed handshakes on a listener now back off from 10 ms (10, 20, 40 … 320 ms) instead of from 250 ms. Core's own reconnect loop still paces its redials, and the ingress requires credentials, so this does not create a hot loop. - The change needs a Runner release before hosts pick it up. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, trace queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.20 |
||
|
|
304e40fe9c |
fix(auth): report effective Cloud owner access to the board UI (#21817)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - Cloud owners can receive instance-admin rights for their own dedicated instance through the existing owner setting. > - Authentication computes these rights for each request without storing an instance-admin role. > - The board UI reads `/api/cli-auth/me` to decide whether to show administration controls such as Tailscale. > - That response used only stored roles, hiding controls that the Cloud backend already authorized. > - This change reports the effective Cloud request rights so the UI agrees with the backend. ## Linked Issues or Issue Description **What happened?** A Cloud owner with `enableOwnerInstanceAdmin` enabled can successfully read Tailscale connections, while `/api/cli-auth/me` reports `isInstanceAdmin: false`. The Connectors page therefore hides Tailscale. The same stale summary can hide other instance administration controls. **Expected behavior** The access summary should reflect the Cloud request's effective privileges. Turning off owner elevation must remove those privileges on the next request. Session and board-key callers must continue to use the fresh stored-role snapshot. **Steps to reproduce** Sign in as a Cloud stack owner with owner instance-admin elevation enabled and no stored instance-admin role. Open Connectors and compare the access summary with a Tailscale connections request: the summary reports no admin access even though the backend permits the request. Searched existing PRs for `"cli-auth/me" cloud` and `tailscale access`; no duplicate fix was found. Related: [#21526](https://github.com/paperclipai/paperclip/pull/21526), which added device sign-in and exposed this hosted permission-summary gap. ## What Changed - `/api/cli-auth/me` uses the trusted Cloud actor's computed `isInstanceAdmin` value for `cloud_tenant` requests. - Session and board-key requests retain fresh stored-role checks. No roles are persisted and no authorization gates or owner settings change. - Regression tests cover Cloud elevation and de-elevation, non-Cloud stored access, and rejection of anonymous/agent callers. Tailscale documentation explains Cloud owner access. ## Verification - Reproduced the deployed mismatch in an authenticated Cloud owner browser session: the access summary reported false while the Tailscale list request succeeded. No feature connection or configuration was changed. - Both Cloud access regression cases failed before the fix. - `pnpm exec vitest run server/src/__tests__/cli-auth-routes.test.ts server/src/middleware/cloud-tenant-actor.test.ts ui/src/pages/apps/TailscaleConnector.test.tsx` passes: 88 tests. - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify --explain` exits 2 and selects broad CI. All 57 checks pass on final head `6044d5097396de95f49bcbb7ce7bbb1e445a80fa`; no checks remain pending. Greptile scored this head 5/5 with no unresolved comments, and the branch is conflict-free. - `git diff --check` and a local credential/PII scan pass. - The fix has not been deployed or accepted in a hosted browser yet. ## Risks The summary exposes administration controls to Cloud owners already authorized by the backend. Non-owner access and disabled elevation remain denied by the existing authentication policy. No migration, provider configuration, or deployment change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added regression tests - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no unresolved comments - [x] I will address actionable reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1115.0-canary.19 |
||
|
|
f71aa135a0 |
fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair
## Linked Issues or Issue Description
**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.
**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.
**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.
**Paperclip version or commit**
Reproduced on `master` at
canary/v2026.1115.0-canary.18
|
||
|
|
b4f71fc276 |
perf(recovery): stop re-checking settled workspace flags and snapshot ledger rows every pass (#21795)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs all background recovery on one Node main thread, and
each database statement costs time on it
> - The ownership reconciliation pass settles in-progress records every
30 s while it has records to watch
> - Two of its kinds listed records that can never change again: dirty
remote workspaces whose writer reached runtime, and snapshot ledger rows
that are uploaded or still held by a stale writer's epoch
> - Each such record cost a transaction, a row lock and up to four reads
on every pass, forever, and kept the 30 s watch armed on an idle
instance
> - This pull request lists only the records that the settle step can
still act on, locks the writer run only when the settle will write, and
skips one stranded-task read that can never match a native run
> - The benefit is that recovery work follows activity, not history
## Linked Issues or Issue Description
**Describe the problem**
On a hosted instance with 7 native runs, the ownership pass ran about
7.6 + 3.2 + 3.5 + 2.8 statements per second over records that cannot
change, and took a row lock on `heartbeat_runs` for each stale dirty
workspace on every pass. At 1,000 agents this becomes thousands of
statements per second.
**Steps to reproduce**
Seed about 100 dirty remote workspaces whose writer run reached runtime
and ended, 50 pending ledger rows still held by those writers, and 100
uploaded ledger rows. Run the ownership reconciliation worker. The new
census test
`server/src/__tests__/recovery-settled-records-census.test.ts` does this
on the embedded database.
**Expected behavior**
A settled record costs zero statements on later passes. An idle instance
runs no SQL from this pass.
**Actual behavior**
On master, one pass costs 714 statements and 150 row locks. An idle half
hour costs 40,800 statements and 9,000 row locks (the settle watch
re-checks every 30 s).
Refs #21765 (the stranded-assignment hint scoping that this rebases on)
## What Changed
- `remote_workspace_unsaved_files` lists a dirty workspace only when its
writer run can still be a preparation-only writer: stage `preparing`, no
`runtime_mode_resolved_at`, no process, session or runner id, no
finalization row and no work-started event (or no writer at all). The
SQL lives with the settle step in
`remote-workspace-unsaved-files-settlement.ts`.
- The unsaved-files settle reads the writer run without a lock. It takes
`for update` only after every check passes, just before it writes, and
checks the locked row again.
- `remote_snapshot_ledger` lists only `pending` rows with no object key,
whose operation is a workspace save, and whose workspace epoch has moved
past the row's writer. Uploaded rows belong to snapshot retention, as
before.
- `terminalRunSettlingUntil` in `recovery/service.ts` skips its
`heartbeat_runs` read when the latest run already in memory is not a
legacy run of another server boot. `getLatestIssueRun` now also selects
`runtime_mode` and `controller_boot_id`. The live-execution check still
runs first.
- New census test with a staging-shaped fixture, a statement counter
grouped by statement shape and caller, and prompt-pickup checks. New
red/green test for the stranded-task lease read.
Census (embedded PostgreSQL, same fixture, base
|
||
|
|
c2e7df2dcc |
fix(workspaces): use project storage for a fresh retry after unowned task storage (#20756)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project can use a storage provider plugin. That plugin prepares each task's git worktree > - A board Retry after a failed run can start a fresh attempt. The fresh attempt starts from the project source > - `preparePluginWorkspace` returns `null` when the task is still bound to a workspace that no storage provider owns. It does this also for a fresh attempt > - On a plugin task environment, the host then reserves an empty adapter-managed "Fresh task workspace" with no repository. The runner fails in it. The next Retry finds that empty workspace bound to the task, returns `null` again and reserves another empty workspace. The task can never start again > - This pull request lets a fresh attempt use the project storage provider even when the previous workspace is not provider-owned > - The benefit is that one Retry gives the task a normal provider worktree again ## Linked Issues or Issue Description **What happened?** Tasks moved off an older execution environment kept a binding to a workspace that no storage provider owned. A board Retry selected `fresh_isolated_workspace`. `preparePluginWorkspace` returned `null` because the bound workspace was not plugin-owned. The host fell back to `reserveFreshPluginTaskWorkspace`, which made an empty `adapter_managed` workspace (strategy `project_primary`, no repository, no recovery snapshots). The run failed in about one second with "Plugin-provided Runner execution operation unavailable; reconcile the same task before retrying". Each later Retry made another empty workspace, chained to the last one. **Steps to reproduce** 1. Use a project with a storage provider plugin that supports fresh workspaces. 2. Bind a task to a workspace that the provider does not own (for example an adapter-managed workspace). 3. Call `preparePluginWorkspace` for a fresh attempt with that workspace as the previous workspace. **Expected behavior** The storage provider prepares a new worktree for the fresh attempt. **Actual behavior** `preparePluginWorkspace` returns `null` and never calls the provider. ## What Changed - `server/src/services/plugin-workspaces.ts`: the early `return null` for a previous workspace that no provider owns no longer applies to a fresh attempt. The fresh attempt then uses the project's storage provider, as for a task with no bound workspace. - `server/src/__tests__/plugin-fresh-workspaces.integration.test.ts`: new test "prepares provider storage for a fresh retry that follows a workspace the provider does not own". ## Verification - The first commit adds only the test. CI on that commit (`d09ad235`, [failed job](https://github.com/paperclipai/paperclip/actions/runs/42976154239/job/130393285731)) shows only this test fail on the current code (`preparePluginWorkspace` returns `null`). - The second commit adds the fix. CI shows the test pass. - The other tests in the file keep the fresh-attempt guarantees: an older provider still fails closed, a changed binding revision still stops the call, and a lost reply still reconciles to the same operation. - I did not run the suite locally. The test needs embedded Postgres, and this environment does not install packages. CI runs it. ## Risks - Low to medium risk. A fresh attempt in a project with a storage provider now uses that provider even when the previous workspace was local or adapter-managed. Before, such an attempt used the fallback path. This matches the project's storage policy, and a fresh attempt does not reuse the previous workspace's files. - Projects without a storage provider are not affected: the function still returns `null` for them. - Non-fresh preparation is not affected. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (shell, file edits). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c6cdd57f7a |
perf(native-runtime): reuse a same-content image provider pack across builds (#21784)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote agent run needs the provider pack (Node, OpenCode and the Runner bridge) inside its sandbox before the provider can start > - The server reuses a pack that is installed in the sandbox image only when its full manifest equals the server's own manifest > - The manifest includes `runnerSourceRevision`, which is the commit that built the server, so an image pack fails this check on every later server build, also when every file in the pack is the same > - The run then uploads the whole pack again, which cost about 15 s per task in a recent load test > - This pull request accepts an image pack whose sealed content equals the expected pack and that differs only in build provenance, and caches verified archive hashes in the server process > - The benefit is that a sandbox image stays usable across server releases that do not change the pack, and the server does less hashing on its main thread ## Linked Issues or Issue Description Refs #21781 (merged), which names the failed check on the run spans. **What existing behavior does this improve?** Selection of the remote provider pack for a new sandbox (`provider_pack.select`). **Subsystem affected** Server: native runtime, remote provider pack verification and upload. **Current behavior** `provider_pack.verify_preinstalled` compares the complete manifest of the image pack with the server's manifest. Any difference fails, including `runnerSourceRevision` and `runnerInputsDigest`, which only record where the pack was built. The run then uploads the full pack. The server also hashes the same archive or pack directory again for every task. **Proposed behavior** When the run is not bound to a saved pack and the operator did not select a custom pack, the remote verification accepts an image pack when (1) its own digest seals its payload, and (2) its schema and payload equal the expected pack after the two provenance fields are removed. Every artifact, the `dist` tree, the candidate trees, the package versions, the Node version, the target platform and the OpenCode version are still checked in the sandbox, against the same values. The server checks the reported manifest again and selects it through the pack authority, as the compatibility path already does. An upload still happens for different content, a corrupted or partial pack, or no pack. The server also remembers a verified transport archive (by path, inode, size and times), so it does not hash it again for the next task. **Reason and benefit** In the harness (fake sandbox, 150 ms per command, 300 ms + 10 MiB/s per transfer, 24 MiB pack), the cross-build case goes from 4.1 s and 24 MiB uploaded to 1.1 s and 0 bytes. Other cases do not change. **Breaking changes** None. A saved or reattached session, and an operator-selected custom pack, still require the exact pack identity. ## What Changed - `remote-provider-pack-verification.ts`: new `acceptProvenance` option. The remote script accepts a sealed pack with the same content and prints its manifest. The server checks that manifest (`sameContentManifest`) and returns it. A forged or different report fails with `installed_manifest_invalid`. - `native-session-executor.ts` (`prepareRemoteProviderPackOnTarget`): staged and preinstalled verification use the option when `acceptProvenanceDifferences` is set. The function returns the selected manifest. The run caller sets the option only through `providerPackProvenanceReuseAllowed` (not retained, no explicit custom pack, selectable authority), and selects a different manifest through the pack authority. - `provider-pack-layers.ts`: `verifyArchive` skips the hash for an archive it already verified with the same file identity. - `remote-provider-pack.ts`: `providerPackProvenanceReuseAllowed` gate, with tests for each case. - `provider-pack-selection-harness.test.ts`: new harness. Uploads go through the workspace confinement check of the Daytona sync guard. It runs the real remote verification in local processes, with modelled sandbox latency and transfer speed, for five cases, before and after. ## Verification - `cd server && npx vitest run src/services/native-runtime/provider-pack-selection-harness.test.ts src/services/native-runtime/provider-pack-archive.test.ts src/services/native-runtime/remote-provider-pack-verification.test.ts src/services/native-runtime/remote-provider-pack.test.ts src/services/native-runtime/retained-provider-pack-source.test.ts src/__tests__/kimi-connection-probe.test.ts` passes. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-artifact-preparation.test.ts src/services/native-runtime/remote-provider-pack-authority.test.ts` passes (1541 passed, 7 skipped). - Red/green: with the source changes reverted, the harness case `same_content_other_build` and the archive cache test fail. With the changes, they pass. - Harness output (load average about 9 on a shared machine; modelled time = wall time + modelled sandbox latency): | Case | Before | After | | --- | --- | --- | | Exact image pack | preinstalled, 1.1 s, 0 MiB | preinstalled, 1.0 s, 0 MiB | | Same content, other build | uploaded, 4.1 s, 24 MiB | preinstalled, 1.1 s, 0 MiB | | Different content | uploaded, 4.1 s, 24 MiB | uploaded, 4.0 s, 24 MiB | | Corrupted image pack | uploaded, 4.0 s, 24 MiB | uploaded, 4.1 s, 24 MiB | | No image pack | uploaded, 3.7 s, 24 MiB | uploaded, 3.8 s, 24 MiB | - The pack step runs no database statements. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - This changes which provider pack the server accepts. It accepts a different manifest digest only when all content fields are equal and every file is hashed in the sandbox, and only for a run that is not bound to a saved pack. The selected manifest goes through the existing pack authority, so launch profiles and receipts bind the installed pack's own identity. - The archive hash cache trusts each archive file's identity (inode, size, times) on the server. The sandbox still checks the uploaded bundle digest and verifies every uploaded file, so a changed local file still fails closed. - This does not help when the image pack has other content (for example a different contract digest). See the hand-off notes for the image build. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1115.0-canary.17 |
||
|
|
b5447432bd |
fix(ui): keep the active company on Cases and Status navigation links (#21794)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI builds company-prefixed links, such as `/PAP/cases`,
from short paths in the sidebar
> - The Cases and Status links use `/cases` and `/status`, but these
roots were not in the board route list
> - The router read `cases` as a company prefix and left the link
unprefixed, so the link only worked through the unprefixed redirect
> - When a person clicked Cases before the company list loaded, the
redirect picked an arbitrary company and showed its empty case list
> - The same race made `tests/e2e/cases-read-only.spec.ts` fail in "ci /
e2e shard (2/8)" on unrelated PRs (#21683, #21715, #21765)
> - This pull request adds both roots to the board route list and makes
the e2e test force the race
> - The benefit is that Cases and Status always open in the company the
person is in, and the flaky e2e test becomes deterministic
## Linked Issues or Issue Description
Refs #21683
Refs #21715
Refs #21765
**What happened?**
The read-only Cases e2e test failed about once per day with `Expected:
3, Received: 0` links in the case list (job 132875164249 on #21765, job
132680942843 on #21715). The failure screenshot shows the sidebar of a
different company ("Passive artifacts …", created by the previous spec)
and "No cases yet". The server log shows that the page asked for the
cases of the other company and never for the test company.
**Expected behavior**
A click on the Cases or Status link opens that page for the company in
the URL, also when the click happens before the company list has loaded.
**Steps to reproduce**
1. Have two or more companies.
2. Open `/<PREFIX>/issues` on a slow connection (the e2e test holds `GET
/api/companies`).
3. Before the company list loads, open More and click Cases.
4. The URL becomes `/cases`. When the list loads, the app redirects to
`/<OTHER>/cases` and shows another company's empty case list.
**Paperclip version or commit**
`master` at
canary/v2026.1115.0-canary.16
|
||
|
|
df9914dbd9 |
fix(native): let the outgoing server checkpoint its idle warm session while the incoming follow-up waits for it (#21791)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. A warm agent session stays with the outgoing server that ran its last turn. > - Since #21774, a follow-up that the incoming server claims during the overlap waits (`native_controller_recovery_waiting_for_owner`) and checks again every 5 s. > - About 1 s after the traffic switch (#21627), the outgoing server tries to save and close its idle warm session. The checkpoint fence refuses that save with `native_runner_checkpoint_owner_lost`. The task's current run is the parked follow-up. That run has its own finalization row under another controller boot, and the fence does not accept that kind of successor. > - The close retry from #21782 fails the same way each time. So the follow-up waits until the outgoing server exits, or until the predecessor's finish time plus the idle window plus 2 min. In the CHA-5 case that was about 225 s, and the worst case is about 425 s. Speed is a requirement. > - This pull request records which owner a parked follow-up waits on, and lets the fence accept the outgoing owner's final save when the follow-up is parked on exactly that owner and has not started. > - The benefit is that the outgoing server saves and closes the session on its first attempt, and the follow-up continues at its next 5 s check. ## Linked Issues or Issue Description Refs #21774, #21782, #21697, #21627, #21279 **What happened?** During a release-swap overlap, a warm follow-up was claimed by the incoming server while the outgoing server still owned the idle warm session. The follow-up waited correctly. The outgoing server's idle close failed with `native_runner_checkpoint_owner_lost` on every attempt. The wait ended only when the outgoing server exited, about 225 s after the follow-up was created. **Expected behavior** The outgoing server saves and closes the session while the follow-up waits. The follow-up sees the saved session and continues within one 5 s check. **Steps to reproduce** 1. Run a warm native session on a remote sandbox and let a run finish on server A. 2. Start server B with an overlap and switch traffic to it. 3. Post a follow-up comment so that B claims the follow-up while A is still alive. 4. A's idle close fails with `native_runner_checkpoint_owner_lost`, and the follow-up waits until A exits. **Paperclip version or commit** master `d14a3b88b1`. ## What Changed - `NativeControllerRecoveryWaitingError` carries the owner that the follow-up waits on: the predecessor run and its controller boot. `waitForLiveWarmSessionOwner` and the live-owner wait in `inspectRemoteWarmSessionRecovery` set it. - `deferNativeControllerRecovery` stores that owner in the finalization `failureDetail.waitingForOwner`. The executor passes it through. - `validateNativeRunnerCheckpointOwner` accepts one more successor kind: a parked follow-up waiting on this exact owner. All of these must be true, or the fence still refuses: - The checkpoint holds an accepted result and has no new work (unchanged rule). - The follow-up finalization is `retryable_failure` with `native_controller_recovery_waiting_for_owner`, `attempt = 0`, no result, and no lease owner (no claim is running it). - `waitingForOwner` names this owner's run and controller boot. - The follow-up run is a running native run for the same company and task, on the same native session as the owner's run. - The follow-up has no process fields, no own process-start events, and no checkpoint rows. - The fence locks the follow-up's finalization row (`for share`), so a resumption claim and the save cannot interleave. - Tests in `native-controller-recovery.integration.test.ts` (real Postgres): - The outgoing server saves and closes its idle warm session while the incoming follow-up waits, and the follow-up resumes within one recheck. - Negative cases: the save is still refused when the follow-up has its own process, its own process-start event, its own checkpoint, or a live resumption claim, or when it waits on a different owner boot or run. ## Verification - Red on master: with the `native-runner-checkpoint.ts` change removed, the scenario test fails with `native_runner_checkpoint_owner_lost` at the outgoing server's first close attempt. The follow-up keeps waiting. Green with the change. - The negative cases pass with and without the change. - Measured in the test scenario (warm idle window 600 s): - Before: the follow-up waits until the outgoing server exits, or about 720 s (finish + 600 s idle window + 120 s margin). - After: the outgoing save and close succeed on the first attempt. The follow-up is claimable and its wait resolves at `retryAt`, 4998 ms after it parked. - `vitest run` of `native-controller-recovery.integration.test.ts`, `native-runner-restart-recovery.integration.test.ts` and `native-harness-backup-storage.integration.test.ts`: 349 passed, 14 skipped. - `vitest run` of `native-session-executor.test.ts`, `remote-warm-session-recovery.test.ts`, `native-runner-checkpoint-save.test.ts`, `native-runner-checkpoint-retirement.test.ts` and `native-runner-checkpoint-authority.integration.test.ts`: 1566 passed, 7 skipped. - `tsc --noEmit` for `server` passes. - `pnpm verify` exits 2 (opaque setup dependency). Required CI supplies the broad coverage. - Not run: a release-swap wave. The fix is not yet measured on a live overlap. ## Risks - The fence is wider. It accepts only a final save with an accepted result and no new work, and only while the current run is parked on this exact owner and has not started. A follow-up that has a claim, a process, process-start events or checkpoint rows still causes a refusal. - A follow-up that a server parked before this change has no `waitingForOwner`. The fence refuses its owner's save as before. At its next recheck, the follow-up parks again with the owner recorded. - While a resumption claim runs the follow-up's recheck (a short window every 5 s), the save is refused. The close retry from #21782 then tries again. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cec8c2b3f4 |
perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues on one Node main thread. The `stranded-assignments` and `assigned-work-invariant` queues check assigned tasks that have no live run. One stranded check costs about 19 database statements. > - #21552 scoped run hints to the run's task graph. #21729 made the 5-minute backstop re-check only changed tasks. A hint with only a company id still makes the queue re-check every open task of the company, 25 per pass. > - Under load on staging (7 native runs), the stranded check ran about 6.7 times per second, and its statement bundle was more than a third of all statements. An activity fixture shows the cause: ordinary run writers send a company-only hint once per run, and one company hint costs about 850 checks. At 1,000 agents this work never stops. > - This pull request makes those writers name the run (or the task) they already know. A run that has no task now resolves to no tasks instead of the whole company. The changed-tasks backstop no longer counts an agent's own run heartbeat as a change to every task of that agent. > - The benefit: with 7 runs a minute plus one taskless run, stranded checks drop from 18.4 to 1.0 per second and stranded-check statements from 344 to 7 per second. Real company-wide changes (pause lifted, budget, company status, tree holds) keep the full company pass. ## Linked Issues or Issue Description Refs #21729, #21552, #21676, #21724 **What existing behavior does this improve?** The issue recovery task queues (`stranded-assignments`, `assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`) and the writers that wake them. **Subsystem affected** Server: issue recovery background work and the run lifecycle writers that publish recovery hints. **Current behavior** Many run lifecycle writers know the run but publish a company-only hint, for example `notifyExecutionRunWork(tx, { companyId })` in the native finalizer and projection, `settleNativeTerminalWake`, the claimed-wake settlement, and an environment lease release of a run without a task. A queued run without a task publishes a company hint. A graph hint for a run without a task resolves to "whole company". A comment sends a company hint to `assigned-work-invariant` and `resolved-dependencies`. Each of these re-checks all open tasks of the company. Also, the changed-tasks backstop treats any change to the assignee agent's row as a change to all of that agent's tasks, and every run start and finish writes that row. **Proposed behavior** A writer that knows the run sends the run's graph hint. A writer that knows the task sends the task hint. A run without a task affects no task queue entry. An agent row change that is only the agent's own run heartbeat does not mark its other tasks changed. Company-wide facts keep the company hint. **Reason and benefit** The work is now proportional to activity, not to (activity x history). See Verification. ## What Changed - `server/src/services/issue-recovery-notifications.ts`: - `notifyIssueRecovery` with a task sends the task hint to all four task queues (before: only to `stranded-assignments` and `stale-issue-locks`; `assigned-work-invariant` and `resolved-dependencies` got the company hint). Only comments (`issue-comment-writes.ts`) call it with a task. - `resolveIssueGraph` returns `[]` when a run-only graph has no task. A missing task seed still returns `null` (whole company). - `server/src/services/recovery/work.ts`: a load for an empty graph returns no rows. The `assigned-work-invariant` queue emits the same `check` debug event as `stranded-assignments` (with `PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load reason. - `server/src/services/recovery/changed-tasks.ts`: the assignee-agent part of the changed-tasks condition skips an agent whose last write was its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or configuration write (no heartbeat stamp) still counts. - `server/src/services/heartbeat/queue.ts`: a queued run without a task gets its id before the insert and publishes a run hint, not a company hint. The daily-cap comment release publishes the run's graph hint. - `server/src/services/environments.ts`, `work-intent-outbox.ts`, `chat-inbound-wake-notice-notifications.ts`: a lease release passes its run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a run to that run's graph. `notifyChatInboundWakeNotices` accepts an optional `runId` for its settlement hint. - Native and recovery run writers now pass the run id they already have to `notifyExecutionRunWork`, `writeExecutionControl` or `notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3), `native-run-projection.ts` (2), `native-session-executor.ts` (3), `wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`, `native-safe-replacement.ts`, `native-restart-recovery.ts` (2), `native-failed-workspace-recovery.ts`, `native-recovery-preparation-failure.ts`, `execution-control-reconciliation.ts`, `execution-recovery-resolution.ts` (2). These are the native equivalents of the legacy run paths in `heartbeat/queue.ts`, which already pass the run id. - Tests: - `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new): staging-shaped company (780 in-review tasks with succeeded native runs, 40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real recovery, dispatch and retry-cancellation workers, a fake clock, and real process-adapter runs. Each run is woken by a user comment through `heartbeat.wakeup`, starts, calls back to post an agent comment and move the task to `in_review`, and finishes. One (or five) taskless runs a minute run beside them. It counts checks by queue and reason, task-queue signals by caller, and statements by owner. It asserts the workload (runs started and succeeded, agent comments, run events, zero agent write failures, at most 10% failed runs), then no company check and no company-only task-queue signal. In CI it runs the 7-runs-a-minute workload for 4 minutes (about 1 minute of real time). With `RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length, attributes statements by stack, and writes the numbers below to that file. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests for each narrowed writer (see Verification). ### Publishers of company-only (or `*`) hints to the recovery task queues "Per" is how often the writer fires on a busy instance. "Known" says whether the run or task is known at the call site. Narrowed in this pull request: | Call site | Event | Per | Known | Now | | --- | --- | --- | --- | --- | | `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task (timer, manual, chat without a task) | each taskless run | run (id generated) | run graph (no tasks) | | `environments.ts:1780` `releaseLease` -> `notifyChatInboundWakeNotices` | environment lease released | each run without a task | run | run graph | | `environments.ts:1792` `releaseLease` -> `promoteDeferredWorkIntents(lock_released)` | lease released | each run without a task | run | run graph | | `environments.ts:1934` `releaseLeasesForRun` | leases of a run released | each run without a task | run | run graph | | `issue-comment-writes.ts:12` (`assigned-work-invariant`, `resolved-dependencies` only) | comment | each comment | task | task | | `native-runtime/native-run-projection.ts:54` `settleNativeTerminalWake` | native run's wake claim settled | each native run finish | run | run graph | | `native-runtime/native-run-projection.ts:177` | native run status projected | each native run finish | run | run graph | | `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native finalization terminal / failure / projection | each native run finish | run | run graph | | `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284` | native execution control deadline, cancelled retry, pre-result settlement | each native run end | run | run graph | | `recovery/wake-request-claimed-settlement.ts:67` | claimed wake settled after the run | each run whose claim is reconciled | run | run graph | | `heartbeat/queue.ts:590` | daily-cap comment key release | each run cancelled at the daily cap | run | run graph | | `native-runtime/native-superseded-retry.ts:42`, `native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`, `:1935`, `native-failed-workspace-recovery.ts:164`, `native-recovery-preparation-failure.ts:30`, `execution-control-reconciliation.ts:270`, `execution-recovery-resolution.ts:830`, `:965` | native recovery terminal writes | each recovered run | run | run graph | | `resolveIssueGraph` for any run-only hint of a run without a task (for example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle | each taskless run event | run | no tasks | Left with the company hint (company-wide fact, or the graph is not known): | Call site | Event | Per | Why kept | | --- | --- | --- | --- | | `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy changed, scope paused or resumed | operator or budget event | eligibility of every task can change | | `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`, `scheduling_resumed`, `reconciler_backstop`, and `accounting_settled` with an active hard stop | hold release | operator event; `accounting_settled` once per run only with a hard-stop policy | can admit any task; guarded by "keeps the company pass for hold releases that can admit any task" | | `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company created, paused, resumed, archived | operator | company-wide | | `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`, `agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent paused, resumed, configured, deleted | operator | can change many tasks, and other agents' review paths | | `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`, `board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and authority changes | operator, first sign-in | company authority | | `chat-channels/endpoints.ts:461`, `identity-links.ts:323`, `resource-reconciliation.ts:290`, `slack/setup-services.ts:50`, `chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50` | chat authority changes | operator | company authority | | `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting changed | operator | instance-wide | | `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold created or released | operator | many tasks | | `company-deletion.ts:83`, `company-portability.ts:6560` | company delete, import | rare | many tasks | | `issues.ts:8015` (blockers replaced), `:11165` (task created), `:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph changes | per task create or blocker edit | graph moves; #21552 kept these | | `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete and restore | rare | #21552 kept these | | `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552 kept these | | `system-recovery.ts:17`, `:70`, `task-retry.ts:294`, `workspace-recovery-capture-policy.ts:59`, `workspace-recovery-stop-proof.ts:122` | system and workspace recovery | per recovery | not shown to stay in one graph | | `agent-conversations.ts:607`, `chat-channels/telegram/attachment-failure.ts:83`, `native-runtime/chat-attachment-reuse.ts:1754`, `native-runner-file-handoff.ts:1317` | conversation and attachment paths | per failure | not shown to stay in one graph | | `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`, `routes/issues.ts:14552`, `issues.ts:12070`, `:12171`, `heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`, `environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) | saved notice and wake changes | per manual wake or project change | company admission state; not reached by run activity in the fixture | | `heartbeat/run-state.ts:1746`, `native-harness-backup-expiration.ts:356`, `:526`, `explicit-native-continuation.ts:705` | session goal clear, backup expiry, explicit continuation | rare | no run id at the call site | | `fast-responses.ts:201`, `:559`, `:672`, `:1053`, `chat-github-response-comments.ts:62`, `:229`, `:440`, `chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a delivery only) | chat response availability | per chat response | no task at the call site; not measured here (chat paths are not in the fixture) | Per-run-event path: appending run events, saving native checkpoints, renewing leases and streaming deltas publish no recovery task-queue hint. `heartbeat-run-events.ts` reaches only live-run recovery queues (`native.process_*`, `turn.*` events), chat progress, response presentation, and `notifyNativeStatusReassessment` for control-plane evidence events only. Checkpoints and lease renewals reach `notifyLiveRunRecovery`, which does not include the task queues. A run's start and finish publish its graph hint from several writers (about 20 task-queue signals per legacy run in the fixture, for example `notifyChatRunMilestones` in `run-identity.ts` activation, `claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce into about 3 checks of the run's own task per run. That is not one hint, but it is bounded by the run's graph. This pull request does not change it. ## Verification - Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per second by queue and reason, and statements per second. Each workload minute is exactly 60 s of fake-clock time, and the rates use the elapsed fake-clock time. "Before" is this branch's test against the `origin/master` service files. Both versions did the same work (7 runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620 to 622 succeeded, about 590 agent comments, about 5,550 run events). The test now asserts this workload and zero agent write failures before it accepts the counts. | | master | this branch | | --- | --- | --- | | 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30 company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company | | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 | | 7 runs/min, all statements/s | 577.5 | 220.4 | | 7 runs/min, company-only task-queue signals | 710 | 0 | | 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86 company + 0.31 hint | 1.05 hint, 0 company | | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 | | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 | | 50 runs/min, company-only task-queue signals | 2,864 | 0 | "Statements in stranded checks" counts statements whose stack includes `reconcileStrandedAssignedIssues`. The other statements are the runs themselves. Recovery load queries (graph lookups and hint loads) are 12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions. `stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both versions. - Without the taskless runs, master already showed the backstop degrade under activity: with 7 runs a minute, 2.46 backstop checks/s, because every run writes its agent's row and marks all of that agent's tasks changed. With this branch: 0.27/s. - New tests in `issue-recovery-deadlines.test.ts`, red on `origin/master` service files and green here: - "re-checks a run's own tasks for its settlement, lease and dispatch writers, and nothing for a run without a task": for `notifyRunDispatch`, `notifyExecutionRunWork`, `notifyExternalConversationSettlement`, `notifyChatInboundWakeNotices` and `promoteDeferredWorkIntents(lock_released)` with a run id, the run's graph (task, parent, dependent) is checked within one pass and none of 300 backlog tasks is. For a run without a task, no task is checked. A comment checks only its task in `stranded-assignments` and `assigned-work-invariant`. - "counts an agent's own run heartbeat as no change to its other tasks, and a resume as a change". - "resolves a run without a task to no tasks and a missing task to the whole company". - Unchanged and passing: the #21729 pickup tests in `issue-recovery-idle-backstop.test.ts` (finished run, user comment, raised budget, resumed agent within a minute; missed hint within the backstop bound; outside change within the hourly pass), the #21552 tests in `issue-recovery-deadlines.test.ts` (including "keeps the company pass for hold releases that can admit any task" and "falls back to the company pass when a run affects more tasks than a hint names"). - Regression, embedded PostgreSQL: `issue-recovery-deadlines`, `issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`, `issue-recovery-work-lifecycle` (42 tests); `native-run-projection-work`, `native-recovery-notice-work`, `muse-terminal-settlement`, `heartbeat-held-comment-wake`, `superseded-workspace-export-recovery`, `native-recovery-preparation-failure`, `upgrade-readiness-post-result`, `chat-wake-conversation-batch`, `environment-service`, `retained-native-session-cleanup-work`, `slack-conversation-lifecycle`, `workspace-recovery-dependency-finalize`, `native-failed-run-owner-release`, `chat-run-milestone-work` (462 tests). - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - **Recovery time bounds. The owner must see this.** No hint-driven pickup becomes slower for a task in the run's graph. Two missed-hint bounds change: - An agent change that sends no hint and is followed by a run heartbeat of the same agent inside the 5-minute window (for example a resume whose hint was lost, after which the agent ran another task) is now found by the hourly full pass, not the 5-minute changed-tasks pass. A normal resume sends `pause_lifted` and is checked at once (tested). - A task outside the run's graph whose recovery depended on the end of a taskless run, or on the native finish writers above, is now found by its own hint, the 5-minute changed-tasks pass (its own rows), or the hourly full pass. Before, any run finish re-checked the whole company. I found no recovery decision that waits on another run of the agent: concurrency waits are queued runs that dispatch resumes. - A run-only hint for a run row that does not exist resolves to no tasks (it was the whole company). A run row is not deleted while its tasks are open, except by company deletion, which sends the company hint. - Native finish writers now send run-scoped hints to the live-run recovery queues too (`notifyLiveRunRecovery`, `notifyLiveResourceRecovery`, attention retention). The legacy run paths already send these with the run id. - The fixture uses process-adapter runs, not native runs. The native writers are narrowed by code reading. Their effect on the hosted staging instance is not verified. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
10ddb26dbb |
test(server): accept dash's exit code for a sandbox-denied shell write (#21792)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner workspace bridge runs agent commands in a `bwrap` sandbox that binds only `/usr`, `/bin`, `/lib` and the task workspace > - A server test checks that a sandboxed command cannot read or write a file outside the workspace > - The test only runs where `/usr/bin/bwrap` exists. CI jobs that the gate sends to GitHub's `ubuntu-latest` now get the `ubuntu-26.04` image, which has `bwrap`, so the test runs on CI for the first time > - There it fails on every run, because it expects the write attempt to exit with 1, but Ubuntu's `/bin/sh` (dash) exits with 2 when a redirection fails > - This pull request checks that the write fails and that the file is unchanged, without depending on the shell's exit code > - The benefit is that the "General tests (server (7/12))" job passes again on the new image, and the test still proves the sandbox denial ## Linked Issues or Issue Description **What happened?** `server/src/services/native-runtime/runner-bridge-tools.test.ts` › `Runner workspace bridge > runs useful commands but denies files outside the workspace and injected credentials` fails in "ci / General tests (server (7/12))" on unrelated branches, on the first run and the retry: - PR #21765, head `98197c87e9` - branch `feat/run-loop-detector`, run 43732047843, head `b8c6e27429`, job 132866487949, image `ubuntu-26.04` version 20261111.196.1 ``` AssertionError: expected { exitCode: 2, signal: null, …(3) } to match object { exitCode: 1 } ❯ src/services/native-runtime/runner-bridge-tools.test.ts:180:19 ``` On jobs that ran on the RunsOn `ubuntu24` image there is no `/usr/bin/bwrap`, so the test is skipped (7 skipped instead of 6). **Expected behavior** The test passes wherever the command sandbox is available, and it fails if the sandboxed command can read or change the outside file. **Steps to reproduce** 1. In a Debian or Ubuntu container, run `/bin/sh -c 'printf stolen > "$1"' sh /nonexistent/dir/private.txt`. 2. dash prints `cannot create /nonexistent/dir/private.txt: Directory nonexistent` and exits with 2. bash exits with 1. `cat` of the same path exits with 1. 3. In the sandbox, the outside temporary directory is not bound, so the path does not exist. This is the same case. **Root cause** A stale test expectation, exposed by a CI image change. No commit on `master` changed the bridge, the sandbox or this test today (the last change to the test is from 2026-11-08). The test expected the shell write to exit with 1. That is bash behavior. On Ubuntu, `/bin/sh` is dash, which exits with 2 for a failed redirection. Until today no CI image had `bwrap`, so the test never ran on CI. **Production impact** None. The sandbox denies the access as intended: the `cat` step on the same run exits with 1 and its output does not contain the secret, and the write fails with `Directory nonexistent` because the outside directory is not visible in the sandbox. This is not a sandbox regression. ## What Changed - The write step now checks that the outside file still contains `PRIVATE` first, then that the command was not stopped and exited with a non-zero code. It no longer requires exit code 1 from the shell. - The read step and the success step are unchanged. ## Verification - Linux container (`node:24-bookworm-slim`, `/bin/sh -> dash`): the write command exits with 2 under dash and 1 under bash, and both are failures, as the test now accepts. - `npx vitest run src/services/native-runtime/runner-bridge-tools.test.ts` in `server/` on macOS: 11 passed, 6 skipped (the sandbox tests need Linux with `bwrap`). - I cannot run the sandbox test on this Mac. It runs in this pull request's CI only when the job lands on an `ubuntu-26.04` runner. ## Risks - Low risk. Only one test assertion changes. The test still fails if the write succeeds, because the file content check runs first and the exit code must be non-zero. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI, Docker). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |