mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
88469bd2bc57add904ec8fc93a4265be47e8e8f4
8383
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
88469bd2bc |
fix(retry): wait for the failed run's sandbox to stop before repeating its chat request (#22423)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task from a connected chat (for example a GitHub pull request review) can fail. Retry then repeats the exact failed chat request in a new run > - Before that run starts the agent, it checks again that the failed run's executor is stopped. For a remote sandbox, the proof is a stop receipt on the sandbox lease > - Workspace recovery can restart the failed run's sandbox at the same time, to save its files. While it does this, the lease has no stop receipt > - The check then refused the repeated request. The run failed for good, although recovery stopped the sandbox again a few seconds later > - This pull request makes the repeated request wait while a stop of that sandbox is in progress, and then ask the retry authority again > - The benefit is that a correct Retry is not lost because of a short overlap with workspace recovery ## Linked Issues or Issue Description Refs #22414. That PR fixes the next step of the same incident: the system replacement run that copied the consumed retry selector. This PR fixes the first refusal. The two PRs change different files and are independent. **What happened?** On a hosted staging deploy, a GitHub review reply task failed (`native_continuation_requires_reconciliation`). Its Daytona sandbox went into workspace recovery. A Retry started a new run that repeats the exact chat request, in a fresh isolated workspace. The run passed admission and its first execution check. After about 14 seconds of workspace preparation, the `dispatching` recheck refused it with 409 `chat_failed_run_retry_not_authorized` ("This chat request can't be repeated exactly as it was sent."). No provider work started. The recovery reported "Saved work recovered" 12 seconds later. The GitHub reply request was not answered. **Expected behavior** The repeated request waits while the failed run's sandbox stops again, and then continues. If the stop does not settle, the request is refused as before. **Steps to reproduce** 1. Make a native chat task fail on a remote sandbox with a recorded stop receipt. 2. Start workspace recovery for that run. `prepareStoppedSandboxForRecovery` moves the lease to `pending_cleanup` and removes `remoteExecutionTermination` while the sandbox restarts. 3. Press Retry during that interval. `taskChatRetryExecutionStopped(source)` is false, so `assertFailedNativeRetryState` refuses the run at the `dispatching` recheck. The new test does steps 1 and 2 with the real lease transitions on an embedded Postgres database. **Paperclip version or commit** `201abacd6ee5` (the staging release in the incident). ## What Changed - Add `server/src/services/chat-channels/failed-chat-retry-source-stop-wait.ts`: - `failedChatRetrySourceStopPending()` is true when a non-local lease of the failed run is `pending_cleanup` with a valid stop-only cleanup intent (`readStopOnlyCleanup`). Workspace recovery holds such an intent while it restarts and stops the sandbox. - `awaitFailedChatRetrySourceStop()` runs the authorization again every 5 seconds, for up to 3 minutes, while that stop is pending. It reads the pending stop before and after each refused attempt. If a stop was pending before the attempt but is gone after it, the stop settled while the refused transaction held no locks, so it reads the authorization once more at once. If no stop is pending before or after, it returns the refusal after one attempt, exactly as before. It waits only for the reason-less containment refusal. It does not wait for a reasoned refusal, a mismatched retry selector (`FailedChatRunRetryAuthorizationError`), or any other error. It stops when the run is stopped, and it returns the last refusal at the deadline. - In `heartbeat.ts`, both execution-phase checks of an exact replay (the first check and the `dispatching` recheck) use this wait. The run log records one line when the wait starts. The wait grants no authority. Each attempt still runs the unchanged `authorizeFailedChatRunRetryWake`. The controller lease is checked on each attempt, as before. ## Verification - `cd server && npx vitest run src/services/chat-channels/failed-chat-retry-source-stop-wait.test.ts`: 10 tests pass. They use real lease rows and the real `prepareStoppedSandboxForRecovery`, `restoreSandboxStopAfterFailedRestart` and `taskChatRetryExecutionStopped`. - Red check: with the helper changed to make one attempt only (the old behavior), the wait test fails with the incident's refusal: `promise rejected "Error: This chat request can't be repeated…" { status: 409, details: { code: 'chat_failed_run_retry_not_authorized' } }`. The abort test also fails. Then I restored the helper. - Red check for the review finding: with the old "no pending stop, so refuse" line, the new test that commits the stop receipt between the refusal and the pending-stop read fails with the old refusal. It passes with the final read. - A first version of that fix read the authorization again after every refusal. CI showed this was wrong: `durable-chat-wakeup.test.ts` counts authorization calls, and a budget-hold case in `chat-channels.integration.test.ts` changed outcome. I reproduced both failures locally with that version. The current version rechecks only when a stop was pending, and those tests pass: `durable-chat-wakeup.test.ts` (51), the CI-failing chat integration cases by name (73), the retry-filtered chat integration run (261), and the focused suites (300). - Safety cases in the same file: refusal at the deadline when the sandbox stays held; immediate refusal when no stop is in progress (an active lease whose executor can still run); no wait for a reasoned refusal, a selector mismatch, or another error; no wait for a run that is not an exact replay; the wait stops when the run is stopped. - Focused existing suites pass: `authorization-run-read.test.ts`, `wake-queue/application/use-cases.test.ts`, `__tests__/durable-chat-wakeup.test.ts`, `task-retry.test.ts`, `budget-resume-continuation.test.ts` (299 tests in total, including the new file). - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify` selects almost the full suite because `heartbeat.ts` is a shared module. I did not run that locally. Required CI supplies that coverage. - I did not do a live staging reproduction. The incident timing cannot be produced on demand. The test reproduces the lease state that the refusal saw. ## Risks - A repeated request can now hold its own prepared sandbox for up to 3 more minutes before it fails. This happens only while a stop of the failed run's sandbox is in progress. - The wait can not let a second executor run. It changes only when the check runs, and every attempt uses the unchanged authority. - If workspace recovery takes longer than 3 minutes, the run is refused as before. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not needed: the code and tests describe the behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dce4d47f60 |
feat(plugins): carry Grok, Copilot and OpenCode credentials to plugin Runners (#22359)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment-driver plugins can run `paperclip_runner` tasks on hosts
that the plugin owns
> - Core transfers the selected managed AI credential to the plugin as a
typed task credential
> - The credential contract has only two types: a Codex auth file and a
direct Claude credential
> - Grok, GitHub Copilot and OpenCode agents in a task-capable plugin
environment therefore fail closed, although the Runner already reads
their credentials from its environment
> - This pull request adds one generic, allowlisted environment
credential for these three harnesses and accepts the Runner's Grok
launch settings
> - The benefit is that plugin-provided Runner execution can run Grok,
Copilot and OpenCode agents with their selected connection
## Linked Issues or Issue Description
**Subsystem affected**
Plugin environment tasks (`packages/plugins/sdk` task contract) and
plugin-provided Runner execution in `server`.
**Problem or motivation**
An agent with the Grok, Copilot or OpenCode harness cannot run in a
plugin environment that runs Runner tasks. Core stops at credential
preparation with "Plugin Runner does not support the selected credential
provider". A Grok agent fails before that point too. The Runner emits
`grokEffort` and `grokDisableWebSearch` in its launch requirements (Core
turns off Grok web search by default), but the strict `settings` schema
of `environmentRunnerLaunchRequirementsSchema` rejects both keys.
**Proposed solution**
Core transfers the selected connection's credential variables for these
harnesses through one typed `provider-env` credential. Each provider has
a fixed allowlist of complete variable sets. The plugin sets the
variables in the Runner process environment.
**Alternatives considered**
One credential type per harness (`grok-env`, `copilot-env`,
`opencode-env`). This adds a type for each new harness. One generic type
with an exported allowlist keeps the same strictness with less surface.
**Roadmap alignment**
The roadmap lists Grok and OpenCode as supported harnesses. This change
lets plugin environments run them. It adds no new product concept.
**Additional context**
Grok subscription login (an auth file with refresh and writeback) is out
of scope. It still fails closed, with an error that names the xAI API
key as the supported option.
## What Changed
- `packages/plugins/sdk`: Added the `provider-env` task credential: `{
type: "provider-env", provider: "grok" | "copilot" | "opencode",
environment }`. The environment must contain exactly one complete
variable set for the provider and no other variable. Values are
non-empty, NUL-free and at most 65,536 characters, as for the other
credential types.
- `grok`: `XAI_API_KEY`
- `copilot`: `COPILOT_GITHUB_TOKEN`
- `opencode`: `OPENROUTER_API_KEY`, or `PAPERCLIP_AI_PROVIDER_URL` with
`PAPERCLIP_AI_PROVIDER_KEY`
- `packages/plugins/sdk`: Exported the allowlist as
`ENVIRONMENT_TASK_PROVIDER_CREDENTIAL_KEY_SETS` and the
`PluginEnvironmentTaskCredentialProvider` type. `claude-env` and
`codex-auth-json` do not change.
- `packages/plugins/sdk`: Added `grokEffort` (`minimal` to `xhigh`) and
`grokDisableWebSearch` (boolean) to the launch requirements `settings`,
with the types that the Runner uses.
- `server`: `preparePluginRunnerCredentials` now transfers the selected
connection's variables for the `grok`, `copilot` and `opencode`
harnesses, with the same account identity semantics as the Claude
branch. It refuses a Grok subscription, a Grok base URL override, and an
incomplete or ambiguous set. The error never contains a credential
value.
- `server`: The submit check now matches a `provider-env` credential to
the task harness. Before, it treated every non-Codex credential as
Anthropic.
- Drift test:
`server/src/__tests__/plugin-runner-credential-names.test.ts` checks the
SDK key sets against the Runner. The SDK list stays written by hand, and
the Runner keeps its own lists of forwarded names. For `grok` and
`copilot`, each name must be in `ACPX_CREDENTIAL_NAMES`, survive sidecar
admission with the session binding that the Runner mints, and reach the
agent process. The binding requirement must match the Rust sidecar's
candidate agents. For `opencode`, each name must be in the OpenCode
proxy environment set. Each name must also be in the Rust sidecar
credential keys or the Rust OpenCode provider keys. The test reads those
from the Runner source, as the Codex security test does. A provider
without a Runner route fails the test.
`OPEN_CODE_RUNNER_ENVIRONMENT_KEYS` is now exported for this test.
- Docs: Added a "Typed credentials" section to
`doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. It also documents the ACPX
credential binding that the Runner requires with a Copilot token.
Updated `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`.
## Verification
- `npx vitest run tests/environment-task-release-resume.test.ts
tests/environment-tasks.test.ts` in `packages/plugins/sdk`: 96 passed.
The tests accept each valid shape. They reject mixed sets, two sets,
extra variables, another provider's variable, empty values, NUL bytes,
oversized values, unknown providers and extra fields, on `submit` and on
`resume`. They also check that every published allowlist set parses.
- `npx vitest run
src/services/native-runtime/plugin-runner-credentials.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 79 passed.
The new tests cover each provider's transfer, Grok subscription refusal,
Grok routing refusal, the fail-closed cases without value leaks, harness
matching at submit, and Runner-built Grok requirements
(`runnerProviderLaunchRequirements`) that parse through the task
contract and produce a Grok launch.
- `npx vitest run src/__tests__/plugin-runner-credential-names.test.ts`
in `server`: 4 passed. I broke each list locally to confirm that the
test fails: a renamed SDK key, an SDK provider without a Runner route,
and a dropped name in the Rust sidecar keys, the Rust candidate agents,
the Rust OpenCode keys and the TypeScript OpenCode keys. Each change
failed the test.
- `node scripts/typecheck-layers.mjs` in `server` (includes
`src/__tests__`): no errors.
- `npx vitest run
server/src/services/native-runtime/plugin-runner-probe.test.ts` (from
the root): 15 passed.
`server/src/services/native-runtime/plugin-runner-task.test.ts`: passed.
- `tsc --noEmit` for `packages/plugins/sdk` and `server`: no errors.
- I did not run the full suite locally. CI supplies that coverage.
## Risks
- Low risk for existing plugins. The change only adds a credential type
and two optional settings. A plugin that does not handle `provider-env`
gets a typed value that it does not recognize. Before this change, Core
failed these runs before it called the plugin.
- A plugin must set `PAPERCLIP_ACPX_CREDENTIAL_BINDING` for a Copilot
task. Otherwise the Runner refuses the token. The new doc section shows
the value.
- An OpenCode gateway connection without a key (`auth: none`) still
fails closed, because every value must be non-empty.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, through Claude Code
with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a0fe90f24b |
fix(retry): never offer Retry while a live run owns the task (#22428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each task shows an interruption banner. The banner offers Retry when an attempt failed and nothing will move the task forward > - The banner reads the task's newest run. A continuation that the system cancels before it starts can be the newest run, while a different run owns the task and does the work > - In a restart-chaos wave, the banner offered Retry for about 140 seconds while the owning run worked. The task then finished with no click > - A Retry that the user does not need breaks the rule that the needed-Retry rate is 0% > - This pull request makes the banner and the Retry endpoint read the same observed run. That run is the live run that owns the task, or else the newest real attempt > - The benefit is that Retry shows only when nothing will advance the task, and a shown Retry is accepted ## Linked Issues or Issue Description **What happened?** A restart-chaos wave (three release swaps) showed a Retry banner on one task for 141 to 144 seconds. The reason code was `issue_execution_lock_changed`. Nobody clicked Retry, and the task completed. Timeline (UTC): - 14:32:09. Run A starts. - 14:51:12. A user follow-up wakes run B. Run B is queued behind run A. - 14:52:46. Run A succeeds. The system schedules continuation run C (`retryOfRunId` = A, reason `native_agent_blocker_continuation`). Run B starts and takes the task execution lock. - 14:53:16. The dispatch gate cancels C before it starts with `issue_execution_lock_changed`, because run B owns the lock. This is correct de-duplication. - 14:53:17. The banner shows `failed` with Retry. Run C was created after run B, so C is the newest run. The projection reads only the newest run and ignores that run B is running and owns the lock. - 14:55:41. Run B succeeds and the task is done. The banner goes away. Retry was never needed. Run B owned the task for the full banner window. **Expected behavior** The banner does not offer Retry while a run owns the task and is queued or running. A run that the system cancelled before it started because another run took the lock is not a failed attempt. When Retry is shown, pressing it is accepted for the run that the banner shows. **Steps to reproduce** 1. Create a task. Run A succeeds and leaves the task in progress. 2. Queue run B for the same task (for example, a user comment). Create continuation run C after run B. 3. Start run B and set it as the task execution lock owner. 4. Cancel run C before it starts with error code `issue_execution_lock_changed`. 5. Read the task interruption. Before this change: `state: failed`, `canRetry: true`, observed run C. After this change: `state: working`, `canRetry: false`, observed run B. **Paperclip version or commit** `fe0c4aa95d` (master at the time of the wave). ## What Changed - `server/src/services/task-retry-eligibility.ts`: add two shared conditions. - `taskAttemptCondition()`: a run cancelled before it started with `recovery_no_longer_needed`, `issue_closed_before_start` or `issue_execution_lock_changed` is not an attempt. The first two codes were already excluded inline in the projection. `issue_execution_lock_changed` is new. - `liveExecutionOwnerCondition()`: the run is the task's execution lock owner and it is queued or running. - `server/src/services/task-interruption-presentation.ts`: the observed run is the live execution owner when there is one. Otherwise it is the newest attempt, with the same native, first-class and legacy precedence as before. The owner lookup is a primary-key read, so the per-task run scan does not grow. - `server/src/services/task-retry.ts`: Retry admission, the refused chat Retry continuation and the superseded-Retry check use the same attempt filter. Admission also accepts the live execution owner when the request names it. The newest attempt stays acceptable, so the automatic Retry hooks (restart, provider limit, budget resume) keep their behavior. - `heldTaskRetryRefusals` uses the same attempt filter. - New test `server/src/__tests__/task-retry-banner-execution-owner.test.ts`: one state table. Each row asserts the list and detail projection, and when Retry is shown, presses it through `requestTaskRetry`. ### State table (tested as one table) | Row | Runs and lock | Banner | Observed run | Retry press | | --- | --- | --- | --- | --- | | 1 | Continuation scheduled; follow-up running and owns the lock | working, no Retry | follow-up | n/a | | 2 | Continuation cancelled unstarted (`issue_execution_lock_changed`); follow-up running and owns the lock (t032) | working, no Retry | follow-up | n/a | | 3 | Same as row 2, lock pointer cleared mid-transfer | working, no Retry | follow-up | n/a | | 4 | Scheduled continuation overdue past the 2-minute dispatch grace; follow-up running and owns the lock | working, no Retry | follow-up | n/a | | 5 | Newer attempt failed; older queued run owns the lock | working, no Retry | queued owner | automatic Retry of the failed run is still accepted | | 6 | Continuation superseded; follow-up then succeeded | resolved, no Retry | follow-up | n/a | | 7 | Continuation superseded; follow-up then failed | failed, Retry | follow-up | accepted | | 8 | Continuation superseded; predecessor failed; nothing else runs | failed, Retry | predecessor | accepted | | 9 | Continuation started, then lost the lock | failed, Retry | continuation | accepted | | 10 | Lock pointer left on a finished failed run | failed, Retry | that run | accepted | | 11 | Row 2 with native runs (`nativeIssueId`) | working, no Retry | native follow-up | Retry naming the owner is accepted | | 12 | Queued owner past its 15-minute move-forward wait; newer failed attempt | working, Retry (`move_forward`) | queued owner | accepted | Invariants that the table checks: - Retry is never shown while something will advance the task (rows 1 to 6, 11). Row 12 offers only `move_forward` on the owner. - A Retry press that names the live owner is accepted (rows 1, 2, 3, 5, 11, 12). - Retry is always shown when nothing will advance the task (rows 7 to 10). - The shared attempt filter is tested in `heldTaskRetryRefusals`, `retireSupersededTaskRetry` and the refused chat Retry continuation (`person_with_superseded_continuation` in `chat-channels.integration.test.ts`). - When Retry is shown, pressing it is accepted for the observed run (rows 7 to 10). ## Verification - Red: with the service changes removed, rows 1 to 8 fail (8 of 10). Rows 9 and 10 pass on both sides. - Green: `cd server && npx vitest run src/__tests__/task-retry-banner-execution-owner.test.ts` passes 14 of 14. - Related suites pass: `task-interruption-query.test.ts` (includes the query-plan bound test), `task-interruption-automatic-path.test.ts`, `legacy-continuation-authority.test.ts`, `native-closed-session-retry.integration.test.ts`, `interrupted-preparation-restore-retry.test.ts`, and the other server tests that call `taskInterruptionPresentation`, `requestTaskRetry` or `heldTaskRetryRefusals`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The change is read-side only. It does not change scheduling, dispatch, the lock, or the run-dispatch gate. - A run cancelled before start with `issue_execution_lock_changed` no longer shows as a failed attempt. If the lock was released and nothing else ran, the banner shows the previous attempt. A failed previous attempt still offers Retry (row 8). - The projection query has one more union branch. It is a primary-key lookup through `issues.execution_run_id`. The existing query-plan bound test passes. - No runner, migration or API contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #22306 (the earlier fix for a due scheduled retry shown as failed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0e38c80323 |
fix(recovery): do not auto-replace a chat Retry under its consumed exact-request authority (#22414)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs that fail before they do provider work can continue automatically. The native safe-replacement sweep schedules a system successor run for them. > - The successor copies the failed run's context. The sweep removes consumed wake fields, but it keeps the `chatFailedRunRetry` selector and `taskRetryRequestId`. > - That selector is the authority of an exact chat Retry. It is bound to the failed run's own chat action and to the user's Retry intent. The successor gets a new system wakeup request, so admission can never authorize it. > - Thus every automatic successor of a chat Retry fails at setup with `FailedChatRunRetryAuthorizationError`. The sweep has already retired the task's recovery hold at that time, so the task can be left with no owner of the next step. > - This pull request stops the sweep from replacing a run that carries `chatFailedRunRetry`. > - The benefit is that the task keeps its hold. The user's next Retry or message continues it through the normal authorized path, and no run fails for a known reason. ## Linked Issues or Issue Description **What happened?** On staging, a user pressed Retry on a failed GitHub chat run. The Retry run failed before its first claim, after a workspace restore failure. Its own exact-request authority refused it (`chat_failed_run_retry_not_authorized`). The pre-claim continuation path then scheduled a `native_safe_replacement` successor. The successor kept the copied `chatFailedRunRetry` selector under a new wakeup request ID, and `authorizeFailedChatRunRetryWake` refused it at setup. **Expected behavior** An automatic system successor must not be scheduled under chat Retry authority that it cannot hold. The task must keep its recovery hold for the user's next Retry or message. **Steps to reproduce** 1. Make a native run whose context has `chatFailedRunRetry` (an exact chat Retry). 2. Fail it before its first coordinator claim, or with a provider terminal failure, so it qualifies for safe replacement. 3. Run `reconcileSafeNativeReplacements`. 4. See that a successor run is scheduled with the copied selector. At admission it always fails with `FailedChatRunRetryAuthorizationError`. Related: #22413 changes the same file for a different defect (the reviewed chat message batch on fresh-session retries). It does not touch the eligibility predicate or `chatFailedRunRetry`. The two changes are independent. ## What Changed - `nativeReplacementPending()` now excludes a failed run whose context carries `chatFailedRunRetry`. Because the shared eligibility predicate has the guard, the pre-claim, historical listener, and provider terminal-failure replacement paths all get it. - New regression in `native-preclaim-continuation.test.ts`: this is the staging shape (a pre-claim failure of an exact chat Retry after its saved work is recovered). It schedules no successor, the hold stays effective, and the task stays blocked. - New paired regression in `native-safe-replacement.test.ts` for the provider terminal-failure path. An ordinary run still gets exactly one successor. An exact chat Retry gets none. ## Verification - Red before the fix: the pre-claim chat Retry case failed with `expected length 0, got 1`. The terminal-failure chat Retry case also failed. Its ordinary-run control passed. - Green after the fix, rebased on `fe0c4aa95d`: - `npx vitest run src/services/native-preclaim-continuation.test.ts src/services/native-runtime/native-safe-replacement.test.ts`: 89 passed. - `npx vitest run src/services/native-runtime/native-safe-replacement.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/execution-control-work.test.ts src/__tests__/recovery-settled-records-census.test.ts src/__tests__/durable-chat-wakeup.test.ts`: 206 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Second-order check: `nativeInterruptionHold()` uses `not(nativeReplacementPending())`. Thus an excluded run can now reach the existing fresh-continuation fallback, the same as other pre-claim runs that do not qualify. That fallback records a continuation decision and lets normal admission build a fresh context. It does not copy the predecessor's selector. - `pnpm verify` selects 2,950 of 3,817 files for this widely imported module. I ran the focused suites above and rely on required CI for the broad coverage. ## Risks - Low risk. The change only removes candidates from automatic replacement. Before this change, a removed candidate always produced a successor that failed at setup. - Behavior shift: after a chat Retry run fails, the user (or a new chat message) continues the task. A system successor that cannot be authorized does not start. Ordinary chat-message runs and non-chat runs are not changed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1120.0-canary.838000 |
||
|
|
875738a8d4 |
fix(tool-access): keep provider-side OAuth registration failures out of Sentry (#22416)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People connect remote MCP apps through OAuth. Paperclip registers
its own OAuth client with the provider through dynamic client
registration (RFC 7591).
> - When the provider's registration endpoint answers with an error,
Paperclip returns a 502 to the person. The error handler also sends
every 5xx `HttpError` to Sentry.
> - A provider that refuses or fails registration (401, 403, 404, 429 or
5xx) is a provider-side failure. It is not a Paperclip server fault, so
it adds noise to Sentry.
> - MCP discovery already marks the same provider statuses as expected
remote failures, and the error handler does not report them.
> - This pull request applies the same marking to the registration
response. It also adds the provider status and the allowlisted error
label to registration failures that Paperclip still reports.
> - The benefit is that Sentry shows only registration failures that can
be Paperclip faults, and each one tells which status the provider sent.
## Linked Issues or Issue Description
**What happened?**
A hosted server reported `Error: OAuth dynamic client registration
failed` three times in two minutes. The stack is `startOAuth` ->
`registerOAuthClient` -> `requestOAuthClientRegistration`. The message
is the generic fallback, so the provider sent no recognized RFC 7591
`error` code. The event did not include the provider's HTTP status.
Thus, the event did not show if the provider refused the request or if
Paperclip sent bad metadata.
**Expected behavior**
The person gets the same 502 and code
`oauth_dynamic_client_registration_failed`. Sentry does not get a crash
report when the provider refuses or fails the request on its side (401,
403, 404, 429, 5xx). Sentry continues to get a report for other
statuses, such as a 400 that rejects Paperclip's metadata. That report
includes the status, so the cause is visible.
**Steps to reproduce**
1. Point an OAuth connection at an authorization server whose
registration endpoint returns 403 (closed or token-gated registration),
404, 429 or 503.
2. Start the OAuth connection.
3. See the 502 response and one Sentry crash report.
**Paperclip version or commit**
`master` at
|
||
|
|
425a30139f |
fix(ui): attribute a stackless unhandled fetch rejection to the code that started it (#22396)
Chromium creates `TypeError: Failed to fetch` with no stack frames, and the browser error monitor removes the SDK Breadcrumbs integration, so nothing wraps fetch to add a call site. An unhandled network rejection reached Sentry with no caller. Once monitoring starts, wrap window.fetch once. Only on rejection, classify the call-site stack as app, extension, other or unknown, keyed by the rejected value in a WeakMap. Tag unhandled object rejections with browser_fetch_initiator and browser_fetch_wrapper. No URL, path or stack text is sent. The returned promise rethrows the identical value, so a dropped promise is still reported. Diagnostics only; this does not fix the underlying rejection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1120.0-canary.837800 |
||
|
|
54972096ba |
fix(native): wait out a provider rate limit that refuses the session start (#22386)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts each agent run in a provider session on a remote sandbox > - A provider can refuse `session.open` because the account reached its rate limit or quota > - After one same-run retry, a repeat of that refusal ended the run as `native_provider_startup_rejected` > - That code blocks the task for a board Retry and sends the failure to Sentry, but the limit passes by itself > - This pull request records that refusal as a provider usage-limit hold, which already exists for limits on a turn > - The benefit is that Paperclip retries the task at the reset by itself, and Sentry gets no error for an expected wait ## Linked Issues or Issue Description Refs #20808, #19246, #18854. No public issue exists for this case. **What happened?** On staging, an account-wide Anthropic 429 refused `session.open` for all runs of one agent. Each run waited the one-hour default cooldown and retried on the same run. The retry met the same 429. The repeat check then ended each run as `native_provider_startup_rejected`. Sentry recorded 25 events in two bursts. Each task became blocked and waited for a person to click Retry. **What did you expect to happen?** A rate limit or quota at session start must wait for the reset, the same as a limit that the provider reports on a turn. The usage-limit hold retries the task at the reset by itself and does not report to Sentry. **Steps to reproduce** 1. Use a remote native runner with an ACPX provider whose account is rate limited. 2. Start a task run. The provider refuses `session.open` with `classification=provider_rate_limit` and a typed `providerFailure` (`phase: "startup"`, `category: "rate_limit"`). 3. Let the same-run retry run after the cooldown while the limit continues. 4. See that the run fails as `native_provider_startup_rejected`, the task becomes blocked for a board Retry, and Sentry gets a `run_failure` event. ## What Changed - `nativeRemoteStartupRejectionCode` returns `native_provider_usage_limit` instead of `native_provider_startup_rejected` when the typed startup failure is `rate_limit` or `quota_exhausted`. The first refusal with a proven runner exit still retries on the same run after the cooldown, as before. Other startup failures do not change. - `readNativeUsageLimitReset` also accepts a startup refusal when the run has no turn event. It reads the current failure (from the executor) or the run's saved `providerFailure`, and it requires `phase: "startup"`. A run with a turn event keeps the old rules. - `planProviderLimitResume` accepts the current startup failure, so the hold gets its reset before the run row records the failure. - The hold saves the startup refusal that created it (`providerLimitStartup`, with its observation time). When the hold resumes, it reads that refusal first. A same-run retry keeps the run row's first saved failure, so the run row can name an older reset. - Tests: a classifier unit test, an integration test for the hold and its automatic Retry, an executor-level test for two startup attempts with different resets, and a Sentry-report test for the startup shape. ## Verification - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "session.open the provider itself rejected"`: 9 tests pass. The new test fails on `master` with `Received: "native_provider_startup_rejected"`. - `npx vitest run src/services/native-runtime/native-provider-capacity.integration.test.ts src/services/__tests__/run-failure-report-native-usage-limit.test.ts`: 26 tests pass. The new integration test fails on `master` (the plan is not automatic without a turn event). - `npx vitest run` on `native-startup-rejected-retry`, `native-provider-capacity`, `native-startup-retry` and `native-provider-limit-cooldown` (integration), and `run-failure-report-native-usage-limit`: 76 tests pass. The new two-attempt test fails on the first head (`c3ee40f166`) with `reason: "reset_unknown"`. - The full `native-session-executor.test.ts`, `native-provider-limit-cooldown.integration.test.ts` and `run-failure-report*.test.ts` files: 1865 tests pass and 1 fails. The failure (`compares export process compatibility without changing conversation identity (changed: false)`) also fails on an unchanged `master` in the same sandbox. - I could not run the full server typecheck locally, because the sandbox has an 8 GiB memory limit and `tsc` stopped with an out-of-memory error. CI runs the typecheck. ## Risks - Low risk. The change applies only to a remote startup refusal with a typed `rate_limit` or `quota_exhausted` category that would otherwise end the run. - The task still shows as blocked during the hold, as for a limit on a turn. The hold now retries by itself at the reset, at most three times in a row without a successful run. - A quota without a reported reset gives the same "restore usage capacity" action as before, so a person still decides. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1120.0-canary.837700 |
||
|
|
c669cfca17 |
fix(native): name a full sandbox disk that fails a run during setup (#22383)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs execute in a managed sandbox with a fixed disk size > - After a server restart, Paperclip reconnects to the running agent. The reconnect opens a new process session in the sandbox through the sandbox provider plugin > - When the agent filled the sandbox disk, the provider refuses that call with "no space left on device" > - The setup failure path did not recognize this refusal. The run failed as the generic `setup_failed`, and the task showed no useful explanation > - #22239 names `sandbox_disk_full` when the runner reports a full disk during a turn. It does not cover a refusal that reaches the setup failure path > - This pull request names that refusal `sandbox_disk_full` in the setup failure path, and keeps continuation recovery from repeating such a run > - The benefit is that the task tells the user what to do in plain language, and Paperclip does not repeat work that fills the same disk again ## Linked Issues or Issue Description Refs #22239 (names a full sandbox disk during a turn). This PR covers the setup failure path. **What happened?** A native agent ran a large dependency installation in its Daytona sandbox. A server restart then happened. The reconnect could not open a process session. The Daytona plugin returned `JsonRpcCallError` (-32002): `internal server error: failed to create session config directory: mkdir /home/daytona/.daytona/sessions/...: no space left on device`. The run failed with `setup_failed` in the `setup` phase. The Sentry report grouped it with unrelated `setup_failed` plugin errors. **Expected behavior** The run failure names `sandbox_disk_full`. The task shows the existing plain-language action (free disk space through the agent, then press Retry). Automatic retry and generic continuation recovery do not repeat the work on a sandbox of the same size. A full disk on the Paperclip host keeps its own code. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Let the agent fill the sandbox disk. 3. Restart the server while the run is active. 4. See the reconnect fail with "no space left on device" and the run fail as `setup_failed`. ## What Changed - `native-sandbox-disk-full.ts`: add `isSandboxProviderDiskFullError`. It is true for the runner's named `sandbox_disk_full` failure, or for a plugin worker error (`JsonRpcCallError`, code `WORKER_ERROR`) whose message reports a full disk. It reads the error, its causes and the first entries of an `AggregateError`. - `run-completion.ts`: the setup failure ladder uses `sandbox_disk_full` for that error before the generic `setup_failed` fallback. Named setup codes and `storage_unavailable` keep their precedence. - `recovery/service.ts`: add `sandbox_disk_full` to the codes that generic continuation recovery does not repeat. `setup_failed` was already in this set, so the new code does not open a new automatic path. This also applies to the turn failures that #22239 names. - Tests for the helper, the setup ladder (with a real database) and the continuation classification. ## Verification - In `server/`: `pnpm vitest run src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-sandbox-disk-full.test.ts src/services/recovery/provider-failure-classification.test.ts src/__tests__/failed-continuation-continues-again.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts`: 5 files, 312 tests pass. - Red check: without the `run-completion.ts` change, the new setup test fails with `errorCode: "setup_failed"`, the same code as the observed failure. Without the helper and recovery change, the helper and continuation tests fail. - The local layered typecheck could not finish: the sandbox memory limit stopped its largest layer. CI runs the typecheck. ## Risks - Low to medium. The change only renames a setup failure that was already terminal and not retried. It uses an existing code that already has retry and UI handling. - A plugin worker error that reports a full disk on the Paperclip host (not in the sandbox) gets `sandbox_disk_full`. This is unlikely: host storage failures use `storage_unavailable`, which keeps precedence. - Generic continuation recovery no longer repeats a turn failure named `sandbox_disk_full`. This matches the intent of #22239. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
92271da899 |
fix(native): keep a chat turn's message batch when a fresh-session retry continues it (#22413)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - External chat turns (GitHub, Slack, Discord) wake an agent on a task
with an admitted message batch.
> - When a task is in review, dispatch checks that the run is bound to
that reviewed chat batch. A run without a batch fails with
`reviewed_chat_execution_binding_not_authorized` (`context_mismatch`).
> - Automatic fresh-session retries (tool-receipt limit, provider
capacity, environment blocker, agent blocker, and safe replacement)
remove all consumed wake input from the failed run's context. This also
removes the chat message IDs, but the context keeps `source:
chat:<provider>`.
> - Thus every automatic retry of a reviewed chat turn fails at setup.
The one-shot review-path recovery that follows copies the same empty
context and fails the same way.
> - This pull request keeps the admitted chat message batch on those
retries and removes the failed run's dispatch proofs, so dispatch checks
ownership and current access again.
> - The benefit is that a reviewed chat turn can continue after a
transient failure, and access checks still apply.
## Linked Issues or Issue Description
Refs #13818 (the same approach for review-path recovery). Related:
#20689, #20697, #20842 (other causes of the same denial).
**What happened?**
A GitHub PR review turn on a task in review stopped on a transient
failure (tool-receipt limit or sandbox environment blocker). Paperclip
scheduled an automatic fresh-session retry. The retry failed at setup in
less than one second with
`reviewed_chat_execution_binding_not_authorized` and denial reason
`context_mismatch`. The review-path recovery wake then failed with the
same error. On staging this occurred 5 times on 3 tasks after #20689,
#20697 and #20842 were deployed.
**Expected behavior**
An automatic retry of a chat turn keeps the admitted message batch.
Dispatch then checks the retry against current task ownership and
current endpoint, conversation and principal access. If access is still
valid, the retry runs. If access was revoked, dispatch refuses the
retry.
**Steps to reproduce**
1. Put a task in review with a run-bound external chat turn (for example
`source: chat:github` with `wakeCommentIds`).
2. Fail that native run with a fresh-session retry lane (for example
`native_environment_blocker`).
3. Let dispatch claim the scheduled retry. The retry context has
`source: chat:github` but no `wakeCommentIds`.
4. Dispatch refuses the retry with `context_mismatch`.
**Paperclip version or commit**
`master` at
canary/v2026.1120.0-canary.837500
|
||
|
|
5db2ccdae6 |
fix(sentry): keep a continued native tool-receipt-limit stop out of Sentry (#22399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends unexpected run failures to Sentry as run-failure events. It keeps known, recovered failures local > - The runner stops a native turn when the turn holds too many tool results (`semantic_tool_turn_receipt_limit`). The finalizer then continues the task in a fresh session > - The runner's stop has no recorded cancellation request. So the cancelled run keeps `unknown`, unexpected cancellation provenance, and `reportRunFailure` sends it to Sentry as "Native session cancelled" > - The continuation owns recovery, so this event is noise. The native capacity retry had the same problem, and #19051 and #19320 fixed it with the successor run as the proof > - This pull request uses the same successor-run proof for the tool-receipt-limit continuation > - The benefit is that a continued receipt-limit stop does not make Sentry noise. Exhausted continuations still report ## Linked Issues or Issue Description Refs #19051, #19320 **What happened?** On staging, a native Codex run stopped at its tool-result receipt limit. The finalizer committed the `semantic_tool_turn_receipt_limit` decision and scheduled a fresh-session continuation run (`retryOfRunId` = the stopped run, `scheduledRetryReason` = `semantic_tool_turn_receipt_limit`). The continuation succeeded and the task finished. Sentry still got a run-failure event: `Error: Native session cancelled`, `run_status=cancelled`, cancellation `source: unknown`, `expected: false`. **Expected behavior** A receipt-limit stop that has a scheduled continuation stays local. The continuation reports itself if it fails. A stop with no continuation (exhausted, or the user stopped it) stays reportable as before. **Steps to reproduce** 1. Run a native turn that reaches the semantic-tool receipt limit. 2. Let the finalizer schedule the fresh-session continuation. 3. With `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`, see that the cancelled run is sent to Sentry. ## What Changed - `server/src/services/run-failure-report.ts`: `isScheduledNativeCapacityRetry` becomes `isScheduledNativeRetry`. It reads the successor run for the lane that can own the run: `native_provider_overloaded` for a failed capacity run (unchanged), and `semantic_tool_turn_receipt_limit` for any other failed or cancelled native run. The query, task binding, setup-phase, signal, and read-failure behavior do not change. - `server/src/services/__tests__/run-failure-report.test.ts`: regression tests with the persisted shape of the staging run, plus negative cases (no successor, other retry lane, other task, no native task, legacy, signal, timed out, setup phase, `all` mode, failed read). - The shared `buildRun` fixture in `run-failure-report.test.ts` now has `createdAt`, like a persisted run row. Without it, the wider scheduled-retry read failed on in-memory test runs and logged a spurious warning. Real report inputs are full database rows, so production code does not change for this. - `server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`: the real native executor and finalizer path now also asserts the Sentry outcome. A continued stop sends no report. A cancelled turn without a continuation still sends one. The fixture run now has `startedAt`, as a real started run does. Without it, no cancelled-run report can occur, and the new assertion would prove nothing. ## Verification - Red before the fix: the two new unit cases failed (`captureRunFailure` was called), and the integration assertion failed (`expected [...] to have a length of +0 but got 1`). - Green after the fix: - `npx vitest run src/services/__tests__/run-failure-report.test.ts src/__tests__/run-failure-sentry.test.ts src/__tests__/run-failure-sentry-real-sdk.test.ts src/services/native-runtime/native-provider-overload.integration.test.ts src/services/native-runtime/native-tool-receipt-limit.integration.test.ts src/services/native-runtime/status-arbiter.test.ts` → 383 passed, 13 skipped. - `node scripts/typecheck-layers.mjs` in `server` → no errors. - `pnpm verify` did not finish in the local sandbox in the time limit. The only failures were 3 cases in `server/src/__tests__/archived-nested-repository.test.ts`. They also fail on unchanged `master` in the same sandbox, and they do not use this code. CI runs the full suite. ## Risks - Low risk. The change only removes Sentry events. It does not change run status, task status, retries, or the user's view. - A failed or cancelled native run that is not a capacity failure now does one more indexed read before its report. That read is bounded by company, task, and agent, and it happens only when a report is about to be sent. - A successor with this retry reason exists only when the status arbiter proved the receipt-limit stop from a persisted runner diagnostic. A user Stop refuses the continuation, and an exhausted lane creates none, so these runs still report. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1120.0-canary.837400 |
||
|
|
a097949864 |
fix(server): stop issue reference syncs from deadlocking on mutual mentions (#22389)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server keeps a table of issue reference mentions. It rebuilds the rows for an issue when its title or description changes. > - `syncIssue` locks the source issue row `FOR UPDATE` before it rebuilds the rows (added in #13898). > - Each mention insert does a foreign-key check. That check takes a `FOR KEY SHARE` lock on the target issue. `FOR KEY SHARE` conflicts with `FOR UPDATE`. > - When issue A mentions B and issue B mentions A, two concurrent syncs each hold one `FOR UPDATE` lock and wait for the other. Postgres reports `deadlock detected`, and one update fails. We saw this on staging. > - This pull request changes the lock to `FOR NO KEY UPDATE`. > - The benefit is that syncs of one issue stay serialized, and foreign-key checks from other issues do not wait on that lock. ## Linked Issues or Issue Description Refs #13898 **What happened?** A staging server reported `Failed query: insert into "issue_reference_mentions" ...` caused by `PostgresError: deadlock detected`. The insert came from `replaceSourceMentions` inside the `syncIssue` transaction. The source issue description mentioned the target issue. **Steps to reproduce** 1. Create issue A with a description that mentions issue B. 2. Create issue B with a description that mentions issue A. 3. Run `syncIssue(A)` and `syncIssue(B)` at the same time, several times. 4. One sync fails with `deadlock detected`. **Expected behavior** Both syncs complete, and each issue has its mention rows. ## What Changed - `server/src/services/issue-references.ts`: `syncIssue` now locks the source issue with `FOR NO KEY UPDATE`, not `FOR UPDATE`. `FOR NO KEY UPDATE` still conflicts with itself, so concurrent syncs of one issue stay serialized (the #13898 behavior). It does not conflict with the `FOR KEY SHARE` lock of a foreign-key check, so the lock cycle cannot form. - `server/src/__tests__/issue-references-service.test.ts`: two regression tests. - One test holds a sync lock in an open transaction. It then syncs a second issue that mentions the locked issue, and expects that sync to complete without a wait. - One test runs 20 concurrent syncs of two issues that mention each other, and expects all four mention rows. ## Verification - `cd server && npx vitest run src/__tests__/issue-references-service.test.ts` - Before the fix (old lock): the two new tests fail. One test shows `expected 'blocked' to be 'completed'`. The other test fails with `PostgresError: deadlock detected` on `select ... from "issues" ... for update`. This is the same error as the staging event. - After the fix: 5/5 tests pass. I ran the file three times and all runs passed. The existing test "serializes concurrent syncs for a description with repeated references" still passes. ## Risks - Low risk. There is no schema change or migration. The lock mode changes only for this read. `UPDATE` statements that do not change key columns already take `FOR NO KEY UPDATE`, so this lock blocks the same issue writers as before. Only foreign-key `KEY SHARE` checks can now continue at the same time. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in a Claude Code agent session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
97737c4f7f |
fix(workspaces): give a failed saved-workspace restore a stable error code (#22419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a run starts, Paperclip can reopen a task's archived execution workspace and restore its saved Git commit > - When the saved restoration record no longer matches its workspace, or a restore step fails, the code throws `GitCacheRestorationFailure` > - That error records which step failed and what kind of failure it was, but it has no `code` > - Run-failure reporting groups failures by error code, never by message. A setup failure without a code falls back to its class name, and this class reports its name as plain `Error`. So these failures share one Sentry group with every other uncoded setup failure > - On staging this one failure makes up almost all of that catch-all group: hundreds of events from automatic restarts of tasks whose saved workspace record can no longer be restored > - This pull request gives the error a stable code > - The benefit is a separate, correctly named Sentry group for restore failures, and run records whose `failureCause` names the restore failure instead of `Error` ## Linked Issues or Issue Description **What happened?** On staging, runs keep failing during setup with "The saved workspace commit could not be restored. Check repository access and retry." The stack is `selectHeartbeatEnvironment` → `readGitCacheExpiration` → `readGitCacheRestoration`. The stored restoration record of an archived workspace fails validation, and the run fails as `setup_failed`. The error has no code, so the Sentry fingerprint uses `exception:Error` and puts it in the same group as unrelated setup failures. `resultJson.failureCause` for these runs is the generic `Error`. **Expected behavior** A failed saved-workspace restore has a stable error code. Sentry groups it on its own, and the run's failure cause names it. **Related** Refs #19736, which added the safe `step` and `kind` restore diagnostics this change keeps. #22093 took the same approach for the execution-target Git context probe. I found no open PR for this error. **Steps to reproduce** Construct `new GitCacheRestorationFailure()` (or call `readGitCacheRestoration` with a record that does not match its workspace). On current master the error has no `code`, `failureCauseFromError` returns `Error`, and the run-failure diagnostics record no root code. ## What Changed - `server/src/services/git-workspace-restoration.ts`: add `GIT_CACHE_RESTORATION_FAILED_CODE` (`git_cache_restoration_failed`) and set it as `code` on every `GitCacheRestorationFailure`. - The message, `step` and `kind` do not change. `classifyWorkspaceRestorationError` still checks the class first, so restore classification does not change. - `server/src/__tests__/git-workspace-restoration-diagnostic.test.ts`: add a test that the error carries the code, with and without a step diagnostic, that `failureCauseFromError` returns it (also when the error is a `cause`), and that `collectRunFailureDiagnostics` records it as the root exception code. That root code is what the Sentry fingerprint uses. ## Verification - Red/green: the new test fails on unmodified master and passes with the fix. - `vitest run` on `git-workspace-restoration-diagnostic`, `git-expiration-auth-fallback` and `run-failure-sentry`: 63 tests pass. - I also ran the other suites that use this error or its readers locally (archived nested repository, fetch head, reproducible managed checkouts, base storage, repository object cache, remote workspace migration, Git workspace expiration, heartbeat environment selection, closed-workspace routes). In this sandbox many `remote-workspace-migration` tests stopped at about 10 seconds each. The migration code doesn't read this error's `code` (it only compares a database lock code and a fixed supersession code), so I expect the slow sandbox to be the cause. CI's full run is the authoritative result. - I couldn't run the full server typecheck locally because the sandbox ran out of memory. The change only adds a new `code` field, which `Error` does not define. CI's Typecheck job covers it. ## Risks - Low risk. Only an added `code` property on one error class. No control flow changes. - `resultJson.failureCause` for these runs becomes `git_cache_restoration_failed` instead of `Error`. The only code that compares `failureCause` checks a different prefix (`remote_workspace_sandbox_disk_full:`). - Every setup-failure classifier in `run-completion` matches a specific error class or HTTP error, not any coded error, so the run's `errorCode` stays `setup_failed` and retry behavior does not change. - Not in scope: automatic fresh continuations still retry this deterministic failure. Stopping that needs a recovery-policy change in `fresh-continuation-fallback`, which decides on `run.errorCode`. That's a separate change for the recovery owners. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, as the Paperclip Sentry triage agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; see Verification for the slow migration suite) - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documentation describes this error's fields) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a792409d07 |
fix(tools): retry a named MCP gateway create that loses a deadlock (#22408)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs prepare a run-scoped MCP gateway for their agent before the provider starts. `createNamedGateway` writes the `tool_mcp_gateways` row and its profile binding in one transaction. > - That transaction locks the agent row first (`FOR NO KEY UPDATE`). Then its gateway insert takes a key-share lock on the company and profile rows for the foreign keys. > - Another writer can lock the company row first and then the agent row. The two transactions then wait for each other. Postgres detects a deadlock and aborts the gateway create (`40P01`). > - The run then fails before the provider starts. The task stops on `native_continuation_requires_reconciliation`, so a person must reconcile a run that did no work. > - This pull request runs the gateway create transaction through the shared bounded retry that #22329 and #20971 use for the same type of lock cycle. > - The benefit is that a short lock conflict no longer fails the run preparation. ## Linked Issues or Issue Description **What happened?** A native run failed while it prepared its MCP gateway. The error was `Failed query: insert into "tool_mcp_gateways" (...)`. The Postgres cause was `deadlock detected` (`40P01`). The task then stopped and needed reconciliation. **Expected behavior** A transient lock conflict does not fail the run preparation. The gateway create writes the gateway and its profile binding, and the run continues. **Steps to reproduce** 1. Transaction A: lock the company row `FOR UPDATE`. 2. Call `createNamedGateway` for an agent in that company. It locks the agent row and then waits for the company key-share lock. 3. Transaction A: lock the agent row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the gateway create is the victim and `createNamedGateway` rejects with the database error. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. Transaction A gets a longer `deadlock_timeout`, so the gateway create is always the victim, as in the observed failure. The failing statement in the test is the same insert as in the observed error. Refs #22329 Refs #20971 ## What Changed - `server/src/services/tool-gateway.ts`: `createNamedGateway` runs its transaction through `retryTransientLockContention`. The helper retries only lock timeout, deadlock and serialization failure (`55P03`, `40P01`, `40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail as before. - `server/src/__tests__/tool-gateway-named-gateway-deadlock.test.ts`: new test with a real two-transaction deadlock between a company-then-agent locker and a real `createNamedGateway` call. The test checks that exactly one gateway row and one profile binding exist after the retry. ## Verification - Failing before the fix: `cd server && npx vitest run src/__tests__/tool-gateway-named-gateway-deadlock.test.ts` failed. `createNamedGateway` rejected with `Failed query: insert into "tool_mcp_gateways" ...`, cause `deadlock detected`, code `40P01`. - Passing after the fix: the same test passes. I ran it 3 times. - `heartbeat-runtime-mcp-servers.test.ts`, `agent-runtime-assignments.test.ts`, `agent-interaction-permissions.test.ts` and `tool-gateway-service.test.ts` pass (4 files, 238 tests). - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low risk. Only the gateway create transaction retries, and only on the three contention codes. - Postgres rolls back the complete aborted transaction. A retry therefore cannot write a duplicate gateway or profile binding. - The interaction-permission check runs again in each attempt. A retry cannot skip it. - If a concurrent create with the same slug commits first, the retry gets a unique violation. That error is not retried. The existing native caller then reads the committed gateway by slug, as before. - The worst added delay is about 1 s of backoff across 5 attempts, and only when the lock conflict continues. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution on embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fa96716bb5 |
fix(storage): back off a throttled small upload that has no source file (#22404)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server saves many objects to S3-compatible storage: skill packages, attachments, recovery archives and manifests > - Object storage answers a burst with `503 SlowDown`. Most large and file-backed uploads already back off (#20165, #20313, #20772, #21010) > - A small upload with no source file still had almost no retry. The S3 SDK never resends a stream body, so a stream got one attempt. A Buffer got the SDK's three closely spaced attempts > - Hosted production and staging reported `SlowDown` on `put_object` from request handlers, for example right after a skill import fetched its files. The error continues on the current production release > - This pull request holds a small stream's bytes within the existing buffered-payload budget, and backs off a throttled small upload like a file upload > - The benefit is that a short request-rate limit no longer fails a small save ## Linked Issues or Issue Description Refs #20165, #20313, #20772, #21010 (the same backoff for other upload paths). **What happened?** A request handler saved a small object with `storage.putFile` and a stream body (for example a skill package archive), or with a small Buffer. Object storage answered `503 SlowDown` ("Please reduce your request rate."). The stream was not sent again. The Buffer was sent three times within about a second. The request failed with a 500. **Expected behavior** A request-rate limit slows the upload down. It does not fail the save. **Steps to reproduce** 1. Point the S3 provider at an endpoint that answers the first `PUT` with `503 SlowDown`. 2. Call `putFile` with a small `Readable` body and no source file. 3. On `master` it throws `SlowDown: Please reduce your request rate.` after one `PUT`. **Paperclip version or commit** `master` at `201abacd6e`. ## What Changed - `server/src/storage/s3-multipart.ts`: `withBufferedUploadBody` holds a small stream body within the shared buffered-payload budget that multipart already uses. It checks the declared length. `sendThrottledRequestWithRetry` uses the same jittered exponential backoff as file uploads. It resends only request-rate limits (SlowDown, 503, 429). - `server/src/storage/s3-provider.ts`: the small `putObject` path without a source file sends Buffer bodies and buffered stream bodies through that retry. When the budget is full, the stream is sent once, as before. - `server/src/__tests__/storage-s3-small-upload-throttle.test.ts` (new): a real S3 client against a local endpoint. ## Verification - `cd server && npx vitest run src/__tests__/storage-s3-small-upload-throttle.test.ts`: with the `master` source files, 3 of 4 cases fail (the stream and Buffer cases reject with `SlowDown`). With this change, 4 of 4 pass. - `cd server && npx vitest run src/__tests__/storage-* src/storage`: 17 files, 135 tests pass on top of `201abacd6e`, including `does not repeat a write that timed out`. - I did not run the full server typecheck locally. The sandbox ran out of memory. CI must cover it. ## Risks - Low. A small stream is now held in memory, at most 16 MiB, inside the existing 32 MiB process budget. If the budget is full, behavior is unchanged. - Only request-rate limits are resent. The store did not apply those writes. Other write failures keep their SDK handling, so a timed-out write is still not repeated. - A stream that ends early or runs long now fails before any request is sent. - Conditional publication puts (`putObjectConditional`) are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only Sentry reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: this is internal storage behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> |
||
|
|
7ef731ec7e |
fix(workspaces): name incompatible project storage when a volume-free server refuses a run (#22398)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free server admits a run only when its task files have
durable storage.
> - A project can keep its task files in plugin storage, such as an
agent-computers plugin. Only environments of that plugin can use that
storage.
> - A task keeps its selected source when it moves to another project.
So the source project's storage still applies after the move.
> - When an agent in a Daytona sandbox got such a task, the volume-free
check refused the run with a generic message. The message said "native
Paperclip Runner with remote workspace storage". The agent already used
native Paperclip Runner.
> - The run ended `setup_failed`. The task offered Retry, but Retry
always failed the same way. Sentry received the refusal as an unexpected
run failure.
> - This pull request finds the real cause with the existing
project-storage compatibility rule. It records the refusal as a
configuration problem that an owner must repair.
> - The benefit is a message that names the storage and the environment,
a task notice with the correct owner action, and no Sentry noise for
this known configuration state.
## Linked Issues or Issue Description
Related: #17552 (a task project and its selected workspace are separate
settings, so a moved task keeps its source).
**What happened?**
On a volume-free server, a task's source project used plugin storage.
The assigned agent used a sandbox environment that cannot use that
storage. Workspace preparation refused the run with "Volume-free servers
require native Paperclip Runner execution with remote workspace
storage." The run failed with `setup_failed`. The task notice told the
user to press Retry. Each retry failed the same way. Sentry received
each refusal as an unexpected run failure.
**Expected behavior**
The refusal names the real cause: the project's workspace storage does
not support the agent's environment. The run fails
`configuration_incomplete`. The task notice tells the owner to use an
agent in a compatible environment, or a source from a project for this
environment. Sentry does not receive this known configuration state.
**Steps to reproduce**
1. Start the server with `PAPERCLIP_EPHEMERAL_SERVER=true`.
2. Create a project whose storage is a plugin workspace provider.
3. Assign a task in that project to a Paperclip Runner agent in a
Daytona sandbox environment.
4. Wake the agent. The run fails `setup_failed` with the generic
volume-free message.
5. Move the task to a project without plugin storage and comment "try
again". The task keeps its source, and the run fails the same way.
**Paperclip version or commit**
Seen on a hosted volume-free staging server. The new regression test
fails on `master` at
canary/v2026.1120.0-canary.836900
|
||
|
|
c3260b1c03 |
fix(native): name the missing precondition when a bootstrap restart refuses an unopened provider (#22406)
Adds a closed diagnostic suffix to the provider_identity_incomplete refusal raised when a same-run bootstrap restart meets an unopened provider: bootstrap_restart=<same_run|other_run>,provider=<unopened|opened>,proof=<held|not_recorded|stop outcome>. Recovery behaviour is unchanged. Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bbb6b4fd5c |
fix(workspaces): stop a cancelled single-pass capture from leaking a stream AbortError (#22397)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip saves remote agent workspaces as portable archives in a
bounded worker thread
> - When a save is cancelled during the single-pass archive write, the
worker leaks an uncaught `AbortError`
> - The parent already rejects with the cancellation reason, so the save
behaves correctly, but the worker `error` event is reported to Sentry as
an unhandled error
> - This pull request checks the cancellation signal immediately before
the single-pass read stream is created
> - The benefit is that a normal cancellation stays quiet and the
archive worker reports only real failures
## Linked Issues or Issue Description
**What happened?**
A staging server reported `AbortError: The operation was aborted` from
the workspace archive worker thread (`workspaceArchive` →
`readStableFile` → `FileHandle.createReadStream`). The error was
unhandled in the worker.
**Expected behavior**
A cancelled workspace capture rejects with the cancellation reason. It
does not emit an uncaught error in the archive worker.
**Steps to reproduce**
1. Start `captureRemoteWorkspaceDirectory({ singlePass: true, signal })`
on a workspace with many files.
2. Abort the signal while the archive body is written.
3. The worker emits an `error` event with an `AbortError`.
**Root cause**
The unverified path of `readStableFile` awaits `open`, `stat` and
`realpath`, then calls `handle.createReadStream({ autoClose: false,
signal })` without a new signal check. If the cancellation arrives in
that gap, Node creates the stream with an already-aborted signal. The
`for await` loop rejects as expected, but the stream also emits a
second, uncaught `AbortError`. A standalone Node script shows the same
behavior. The verified path is not affected because it calls `check()`
synchronously before it creates the stream.
## What Changed
- Call `check()` immediately before the single-pass `FileHandle` read
stream is created in `readStableFile`.
- Add a regression test that cancels a single-pass capture when the
archive write opens its first source. The test asserts that the capture
rejects with the cancellation reason, that no uncaught exception occurs,
and that the partial archive is removed.
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-snapshot-archive.test.ts -t "leaking a
stream abort"`: fails before the fix (rejects with the stream
`AbortError` instead of the reason) and passes after the fix.
- A local worker reproduction (single-pass capture of 400 files, abort
after 1–500 ms, count worker `error` events) reported one worker
`AbortError` before the fix and none after the fix.
- The full `remote-workspace-snapshot-archive.test.ts` file has the same
results with and without this change in the local sandbox, apart from
the new passing test. The other local failures come from this sandbox
(umask `077` and Git differences) and also fail on `master`. CI is the
authority for that file.
## Risks
- Low risk. The change adds one signal check on a path that already
checks the signal on every chunk. A cancelled capture now rejects with
the caller's reason instead of the stream's `AbortError`, which matches
the verified path.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
16dfa200ac |
fix(environments): clean up an unconfirmed setup sandbox and answer a provider outage with 503 (#22377)
A provider outage during an interactive custom-image setup start left a provisional sandbox with no cleanup owner, and the raw RPC error reached the HTTP error handler as a 500 crash report. - Keep the creation-cleanup envelope and create status for environmentStartInteractiveSetup errors at the worker RPC boundary. - Record an in-scope unconfirmed setup sandbox as a pending-cleanup lease that the existing sweep destroys. - Pin failed-creation cleanup (probe and teardown) to the plugin that created the sandbox. - Answer a transient provider refusal with 503 environment_provider_unavailable instead of a crash report. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
49adbbe2f6 |
fix(environments): retry recording a started sandbox lease that loses a deadlock (#22392)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A sandbox run creates its sandbox with the provider first. Then `attachSandboxCreation` records the started sandbox as the run's `environment_leases` row in one transaction. > - That transaction locks the environment row, the creation operation row and the run row. Then the lease insert takes a foreign-key key-share lock on the parent issue (and execution workspace). > - Other writers lock the issue row first and then the run row. When such a writer holds the issue and asks for the run, a lock cycle occurs. PostgreSQL aborts the lease insert with `deadlock detected` (`40P01`). > - The transaction rolls back whole. But the run failed during setup with `Failed to acquire lease ... Failed query: insert into "environment_leases" ...`, and the sandbox that the provider had already started was discarded. The bootstrap retry then had to create a new sandbox. > - This pull request runs the `attachSandboxCreation` transaction through the shared bounded `retryTransientLockContention` helper. #22329, #21703 and #20971 use the same helper for the same kind of lock cycle. > - The benefit is that a short lock conflict no longer fails the run setup or discards a started sandbox. ## Linked Issues or Issue Description Refs #22329 Refs #21703 Refs #20971 **What happened?** On staging, a sandbox run failed during setup with `EnvironmentRunError: Failed to acquire lease for environment "..." (sandbox): Failed query: insert into "environment_leases" ...`. The cause chain was `EnvironmentProviderUnavailableError` ("a database query failed while recording the lease") and then `PostgresError: deadlock detected`. The provider had already started the sandbox (`sandboxState: started`). **What did you expect?** A lost lock race on the lease record must not fail the run. The run must keep the sandbox that the provider started and continue to its first turn. **Steps to reproduce** 1. Start a sandbox creation operation for a running heartbeat run on an issue. 2. In another transaction, lock the issue row `FOR UPDATE`. 3. Record the created sandbox lease (`environmentService(db).acquireLease({ creationOperationId, ... })`). It locks the run row and then waits for the issue key-share lock. 4. In the other transaction, lock the run row `FOR UPDATE`. 5. PostgreSQL aborts the lease insert with `40P01`. ## What Changed - `server/src/services/environments.ts`: `attachSandboxCreation` now runs its transaction through `retryTransientLockContention` (5 attempts, 100 ms × attempt backoff, only for `40P01`, `40001` and `55P03`). A fresh attempt re-reads the pending creation operation and the run row, and re-proves every check. A doc comment records the lock order and the rule that `attach` writes only through `tx`. - `server/src/services/environment-lease-attach-deadlock.integration.test.ts`: new embedded-Postgres regression that creates the real issue-then-run lock cycle, and a negative case that a deterministic failure is not retried and leaves the operation `pending`. ## Verification - Red before the fix: `npx vitest run src/services/environment-lease-attach-deadlock.integration.test.ts` failed with `PostgresError: deadlock detected` (`routine: DeadLockReport`) on the same `insert into "environment_leases"` statement as the staging event. - Green after the fix: the same file passes (2/2). The lease row is recorded once, `active`, with the provider lease id, and the creation operation is `attached` to it. - Related suites pass: `src/__tests__/sandbox-creation-operations.test.ts`, `src/__tests__/environment-task-authority.integration.test.ts`, `src/__tests__/plugin-warm-tasks.integration.test.ts`, `src/services/environment-provider-unavailable-error.test.ts` (158 tests), and `src/__tests__/environment-runtime.test.ts` (214 tests). - Not run locally: the server `tsc` typecheck. The sandbox memory limit killed it (exit 137). CI typecheck covers this revision. ## Risks - Low risk. Only a transaction that PostgreSQL already rolled back whole is retried, and the number of attempts is bounded. - Both callers of `attachSandboxCreation` pass the root database handle, and both `attach` callbacks write only through `tx`. If a future caller passes an open transaction, a retry runs inside it as a savepoint. That retry stays bounded and then throws the original error. - No change to the error classification of other lease acquisition failures. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, embedded-Postgres test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
48025caaca |
fix(ui): mark the connectivity probe's timeout abort as intentional (#22390)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The browser UI shows a connection banner. It checks the server with a health probe (`probeServerHealth` in `ui/src/lib/connectivity.ts`) > - The probe stops its own `fetch` after 10 seconds with a reasonless `controller.abort()`. The app catches that rejection > - A `fetch` wrapper outside the app, such as a browser extension, can drop the rejection. The browser then reports `AbortError: signal is aborted without reason` as an unhandled rejection, and Sentry records an error that has no user effect > - #22090 added a marked abort reason for the shared GET and a Sentry filter that drops only that marked object. The probe timeout did not use it > - This pull request makes the probe timeout abort with the same marked reason > - The benefit is that the probe's own timeout no longer shows as a production error, and every other `AbortError` is still reported ## Linked Issues or Issue Description No public issue exists. This is the problem, in the bug template fields: **What happened?** A hosted browser reported one unhandled rejection: `AbortError: signal is aborted without reason` (DOMException code 20, mechanism `onunhandledrejection`, native trusted promise). The stack is a `setTimeout` callback in `probeServerHealth` that calls `controller.abort()` on the `/api/health` probe. The tab was hidden and had just resumed. **Expected behavior** The probe's own timeout does not appear in error monitoring. A real abort failure still appears. **Steps to reproduce** 1. Replace `window.fetch` with a wrapper that calls the real `fetch` and attaches only `.then(() => {})` to its promise. 2. Make `/api/health` hang for more than 10 seconds. 3. The probe timeout aborts the request. The app's `try/catch` handles its own promise, but the wrapper's derived promise rejects with no handler. **Paperclip version or commit** Reported on release `651749bdb4`. `probeServerHealth` is unchanged on master up to this branch's base (`201abacd6e`). **Deployment mode** Hosted, browser UI (Chrome-family message text). Refs #22090 (same filter, shared GET call site) and #22380 (same approach, announcement card). I searched open PRs for `probeServerHealth` and `intentionalAbortReason` and found no duplicate for this call site. ## What Changed - `ui/src/lib/connectivity.ts`: the probe timeout calls `controller.abort(intentionalAbortReason())` instead of `controller.abort()`. - `ui/src/lib/connectivity.test.ts`: a new test uses a `fetch` that settles only when its signal aborts. It checks that the probe returns `{ reachable: false }` after 10 seconds and that the signal reason is the marked intentional reason. ## Verification - `cd ui && npx vitest run src/lib/connectivity.test.ts src/lib/sentry.test.ts src/api/client.test.ts` → 3 files, 80 tests pass. - Red check: without the `connectivity.ts` change, the new test fails (`isIntentionalAbortReason(signal.reason)` is `false`). - `cd ui && npx vitest related src/lib/connectivity.ts --run` → 346 of 347 files pass. The 2 failures in `src/components/Layout.viewport.test.tsx` also fail on unmodified `master` in this sandbox, so this change does not cause them. - `tsc --noEmit` for `ui` reports no errors in the changed files. - The existing real-SDK tests in `ui/src/lib/sentry.test.ts` (from #22090) prove that `beforeSend` drops a marked rejection and still sends an unmarked `AbortError`. ## Risks - Low risk. The filter matches by object identity, and only `intentionalAbortReason()` makes these objects. An `AbortError` from any other source is still reported. - The probe's abort message changes from the browser's text to "The operation was aborted.". The probe catches every rejection and does not read the message. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
434e574ec6 |
fix(ui): stop reporting posthog-js request timeouts as unhandled rejections (#22381)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted Paperclip Cloud loads PostHog's browser script (`posthog-js`)
through a same-origin proxy for product analytics, and reports browser
errors to Sentry
> - posthog-js stops its own requests at a client-side timeout. For
`/flags` this timeout is 3000 ms by default. It makes `AbortError:
PostHog request timed out after 3000ms`, logs it as a warning, and tries
again
> - On a slow visitor network, this abort reason can escape as an
unhandled promise rejection inside PostHog's script. Sentry's global
handler then reports it as a board error. The board has no handler to
attach there, and the visitor sees no effect, because analytics never
gates the UI
> - This pull request makes the browser Sentry `beforeSend` drop exactly
that one error, and only when it comes through the unhandled-rejection
handler
> - The benefit is that Sentry shows board failures, not third-party
analytics timeouts
## Linked Issues or Issue Description
No public issue exists. This is the problem, in the bug template fields:
**What happened?**
A hosted browser reported one unhandled rejection: `AbortError: PostHog
request timed out after 3000ms` (mechanism `onunhandledrejection`,
native trusted promise). All stack frames are in `/_ph/static/array.js`
(posthog-js), below the SDK's `setTimeout` wrapper. posthog-js 1.453.0
makes this error in its fetch transport timeout (`abort(new
Error("PostHog request timed out after <n>ms"))` with `name =
"AbortError"`). `feature_flag_request_timeout_ms` is 3000 by default, so
this is the `/flags` request. posthog-js logs this as a warning and
tries again. The board has no handler that it can attach inside
PostHog's script.
**Expected behavior**
A timeout of an analytics request that posthog-js handles and retries
does not appear in error monitoring. Analytics never gates the board, so
the visitor sees no effect. Any other `AbortError` still appears.
**Steps to reproduce**
1. Open a Cloud board with browser analytics on, on a slow network.
2. Let a posthog-js `/flags` request take more than 3000 ms.
3. posthog-js aborts the request with its own `AbortError`. The reason
escapes as an unhandled rejection, and Sentry reports it.
**Paperclip version or commit**
Reported on release `651749bdb463aa09d1eb2ae6e61ac3da26161a74`. This
branch is based on `201abacd6e`.
**Deployment mode**
Hosted Paperclip Cloud, browser UI.
Related: #22090 (drops the shared GET's own cancellation through the
same `beforeSend` check). This change adds a second exact match to that
check. The PostHog proxy (`server/src/routes/posthog-proxy.ts`) refuses
at once at its limits and has a 30 s upstream timeout, so it does not
cause a 3 s client stall. I searched open and closed PRs for this error
message and found no duplicate.
## What Changed
- `ui/src/lib/sentry.ts`: add `isPostHogRequestTimeout`. It matches only
an `Error` with `name === "AbortError"` and the exact posthog-js message
`PostHog request timed out` (with an optional ` after <n>ms`).
- `ui/src/lib/sentry.ts`: the existing unhandled-rejection drop in
`beforeSend` (from #22090) now also matches `isPostHogRequestTimeout`.
As for the existing match, it consumes the rejection-event snapshot. All
other events keep the same enrichment and are sent.
- `ui/src/lib/sentry.test.ts`: add real `@sentry/browser` SDK tests that
count what reaches the transport.
## Verification
- `cd ui && npx vitest run src/lib/sentry.test.ts
src/api/client.test.ts`: 2 files, 56 tests pass.
- Red proof: with master's `sentry.ts`, the new test "drops posthog-js's
own timeout when it escapes as an unhandled rejection" fails (1 failed,
36 passed).
- The tests prove that these are still reported: another `AbortError`,
the same text on a plain `Error`, a string reason with the same text,
and the same error when Paperclip code captures it with
`captureException`.
- `npx tsc --noEmit -p ui/tsconfig.json` shows no errors in the changed
files. Other errors come from the unbuilt `@paperclipai/plugin-sdk` in
the local checkout.
## Risks
- Low risk. The filter matches one exact third-party error name and
message, and only on the unhandled-rejection path.
- If posthog-js changes the message, the filter stops matching, and
Sentry reports the error again. No board failure can be hidden.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9874608d0a |
fix(native): retry a workspace failure write that loses a deadlock (#21507)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes each agent run after the provider turn
ends. It copies the workspace back and records the result.
> - When the workspace copy-back fails, `recordRetryableFailure` records
the failure and a bounded export retry in one transaction.
> - That transaction locks the coordinator, issue and run rows. Then the
recovery-action insert takes `FOR KEY SHARE` on the agent row for its
foreign keys.
> - A different transaction that holds the agent row and waits on the
run row deadlocks with it. Postgres aborts the failure write with 40P01.
> - The run then fails with `Database query failed (40P01)`, and the
export retry is never recorded.
> - This pull request runs the whole failure write through the existing
`retryTransientLockContention` helper.
> - The benefit is that a lost deadlock no longer fails the run or drops
its workspace export retry.
## Linked Issues or Issue Description
Refs #21494 (same mechanism, but for the provider-turn failure write in
`native-session-executor.ts`. This PR covers `native-run-finalizer.ts`.)
Refs #21506 (a different agent-row deadlock in run setup. It does not
change this write.)
**What happened?**
On a hosted staging server, two runs finished their provider turn, and
then the workspace copy-back failed. The server tried to record that
failure and lost a Postgres deadlock on `insert into
issue_recovery_actions`. The stack was `recordNativeFinalizationFailure`
-> `recordRetryableFailure` -> `upsertSourceScoped`. Each run failed
with `Database query failed (40P01)`.
**Expected behavior**
A lost deadlock in the failure write is retried. The run records
`retryable_failure` and an active `active_run_watchdog` recovery action
for the bounded workspace export retry.
**Steps to reproduce**
1. Hold `FOR UPDATE` on the agent row in one transaction.
2. Start `recordRetryableFailure` for a run of that agent. It locks the
run and issue rows, then waits on the agent row.
3. In the first transaction, request a lock on the run row. Postgres
detects the deadlock and aborts one side with 40P01.
The new integration test does these steps with embedded Postgres.
**Paperclip version or commit**
`master` at
canary/v2026.1120.0-canary.836200
|
||
|
|
b5c53a996b |
fix(native): stop a repeated startup rejection when a run resumes its predecessor's provider session (#22393)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start a Runner, and the Runner opens
the provider session with `session.open`
> - When the provider refuses `session.open` at startup, no provider
work began. The executor records a startup failure signature, so the
next attempt can stop a repeat of the same refusal (#20874)
> - Only bootstrap retries recorded that signature. A run that continues
its predecessor's provider session (for example a provider-limit Retry)
recovers as an exact checkpoint resume, so it never recorded one
> - Such a run retried every 30 seconds against the same provider rate
limit, spent all 3 attempts, and ended as
`provider_session_open_rejected`. That outcome is reported as a run
failure
> - This pull request records the signature for every remote startup
rejection
> - The benefit is that the second identical refusal ends the run at
once. With #22386, a repeated rate-limit refusal then becomes the
usage-limit hold that retries the task at the reset
## Linked Issues or Issue Description
Refs #22386
Refs #20874
**What happened?**
A staging run resumed its predecessor's provider session. The provider
refused `session.open` with HTTP 429 ("would exceed your account's rate
limit") on each attempt. The recovery mode was
`exact_checkpoint_resume`, so no startup failure signature was saved.
Each attempt looked new. The run made 3 attempts in about 3 minutes and
ended as `provider_session_open_rejected` (retry budget spent).
**Expected behavior**
The second identical startup refusal stops the retries, as it does for a
bootstrap retry. A rate-limit refusal waits for the reset (with #22386)
and does not use the full retry budget.
**Steps to reproduce**
1. Give a remote native run a session checkpoint with an established
provider session.
2. Make the provider refuse `session.open` at startup with the same
error on two attempts, with the runner exit proven.
3. See that the second attempt is retried again, and the run ends only
when the attempt budget is spent.
**Paperclip version or commit**
`master` at
|
||
|
|
7485e01161 |
fix(heartbeat): keep a credential denial an owner action when the lease lapses during AI selection (#22379)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat prepares each run before the agent starts. One
preparation step selects the AI account for the run.
> - AI selection can reject an account for a reason that only its owner
can fix. An example is a credential that is not shared with the
responsible user.
> - Run setup gives a lost controller lease priority over every other
failure code. When the lease lapsed during the same setup, the run got
`controller_lease_lost` instead of `configuration_incomplete`.
> - `controller_lease_lost` is retryable. So the run retried and failed
again with the same rejection. The code also bypassed the Sentry filter
for owned AI selection blockers, so the server sent a spurious Sentry
event.
> - This pull request keeps `configuration_incomplete` when AI selection
itself rejected the account, even when the lease is lost at the same
time.
> - The benefit is that the owner gets the correct action at once, with
no wasted retry and no Sentry noise.
## Linked Issues or Issue Description
Refs #15714 (this PR adds the Sentry filter for owned AI selection
blockers, and this change reuses that check).
**What happened?**
A legacy run was in setup. AI selection correctly rejected the account
(`ai_connection_credential_not_shared`). At the same time, a lease
renewal stalled for about 29 seconds and the controller lease lapsed.
Setup recorded the run as `controller_lease_lost` with the configuration
message. The run was retried. The retry failed again as
`configuration_incomplete`. The first failure also went to Sentry,
because the filter only matches `configuration_incomplete`.
**Expected behavior**
When AI selection rejects the account for an owned reason, the run fails
as `configuration_incomplete`. The run is not retried. The board gets
the configuration action. No Sentry event is sent. The run result keeps
the lease-loss evidence.
**Steps to reproduce**
1. Start a legacy run whose AI account selection fails with an owned
rejection (for example, a credential that is not shared with the
responsible user).
2. Make the controller lease lapse while setup selects the AI account.
3. Look at the run `errorCode`, the retry schedule, and the Sentry
report.
**Paperclip version or commit**
`master` at
|
||
|
|
812e30d423 |
fix(ui): drop Firefox for iOS injected-script errors from browser Sentry (#22378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The browser UI sends unhandled errors to Sentry through `ui/src/lib/sentry.ts` > - Firefox for iOS injects its own Reader-mode and helper scripts into each page. These scripts read a `__firefox__` global > - When one of these scripts runs before its setup is complete, the page's `window.onerror` reports an error. Sentry then records it as a Paperclip error > - No Paperclip code defines or reads `__firefox__`. These reports are browser noise and hide real errors > - This pull request drops errors whose message contains `__firefox__` with the built-in Sentry `ignoreErrors` option > - The benefit is fewer false error reports. All other errors from the same browser still go to Sentry ## Linked Issues or Issue Description **What happened?** Hosted browser Sentry recorded two errors from one page load on Firefox for iOS: `TypeError: undefined is not an object (evaluating 'window.__firefox__.reader')` and `ReferenceError: Can't find variable: __firefox__`. Both events have the same trace and the same timestamp. The only stack frame is the page document (line 1, column 19), not a Paperclip bundle file. **Steps to reproduce** 1. Open a Paperclip page in Firefox for iOS. 2. Let the browser's Reader-mode script run before its `__firefox__` setup is complete. 3. See the `__firefox__` error in the browser Sentry project. **Expected behavior** Sentry does not record errors that come from scripts the browser injects. **Actual behavior** Sentry records them as unhandled Paperclip errors. No duplicate or related public PRs were found (searched for `__firefox__` and "firefox ios sentry"). ## What Changed - `ui/src/lib/sentry.ts`: add `ignoreErrors: ["__firefox__"]` to the browser Sentry init options. A module comment tells why. - `ui/src/lib/sentry.test.ts`: add tests that use the real Sentry SDK. The tests send both `__firefox__` messages and make sure that Sentry drops them. Each test also sends a normal app error and makes sure that Sentry keeps it. - `doc/observability.md`: describe the filter. Correct an old line that said that the browser adds no `beforeSend` hook. ## Verification - `cd ui && pnpm exec vitest run src/lib/sentry.test.ts src/lib/browser-error-context.test.ts`: 55 passed. - Without the fix (tests kept), the 3 new tests fail. - `pnpm verify` was not run locally because of a partial dependency install. CI runs the full suite. ## Risks - Low risk. The filter drops only messages that contain the string `__firefox__`. Paperclip code does not use that name. All other errors are not changed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4d954646eb |
fix(native-runtime): refresh a runtime asset lease once for concurrent turns (#22384)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a turn starts, the server copies each assigned skill into a
content-addressed runtime asset bundle. Retention garbage collection can
delete old bundles.
> - `withRuntimeAssetUse` keeps a bundle safe from deletion with an
hourly access lease. It records the lease under the same directory
publication lock that deletion uses. The lease is held in memory, so it
is cold after a server restart and again each hour.
> - When the lease is cold, each concurrent turn that uses the same
bundle takes the exclusive lock in sequence and verifies the full bundle
again. One large skill has 1,350 files (24 MB). On a busy server, the
turns at the end of that queue wait more than the 30 s lock budget. The
run then fails with "Timed out waiting for workspace restore lock". The
lock diagnostics show that the same process holds the lock
(`knownLocalHolder: true`).
> - This pull request lets concurrent uses of one digest share one
locked lease refresh.
> - The benefit is that a restart or an hourly lease expiry no longer
makes the preparation of parallel turns fail.
## Linked Issues or Issue Description
No public issue exists. This is the bug description.
**What happened?**
Agent runs on a hosted staging server failed during preparation with
`ERR_WORKSPACE_RESTORE_LOCK_TIMEOUT` ("Timed out waiting for workspace
restore lock at <instance>/locks/directory-merge/<sha256>.lock"). The
stack is `materializeSelectedSkills` (Promise.all) ->
`materializeSkillAsset` -> `withCanonicalDirectoryMergeLock` ->
`acquireDirectoryMergeLock` -> `waitForLock`. The lock diagnostics were
`knownLocalHolder: true` and `waitMs: 30052`, and the owner sidecar read
timed out. The server process had started 48 minutes before the failure.
Eight events occurred in two days. One agent ran many turns in parallel
with the same large skill.
**Expected behavior**
Concurrent turns that use the same runtime asset bundle start without
waiting for each other to verify the bundle again.
**Steps to reproduce**
1. Start a server with an empty in-memory lease map (a new process).
2. Start many turns at the same time. Assign the same large skill to all
of them.
3. Each turn takes the publication lock for that bundle in sequence. The
last turns time out after 30 s.
The new unit test shows the cause directly: 8 concurrent cold uses of
one digest take the lock 8 times on `master`. With this change, they
take it once.
**Paperclip version or commit**
Release `24132dac169fafc616615b4f3144e921aae8ad54`. The code is
unchanged on `master`.
**Deployment mode**
Hosted (staging).
**Agent adapter(s) involved**
Not adapter-specific (core bug).
## What Changed
- `withRuntimeAssetUse`
(`server/src/services/native-runtime/runtime-asset-retention.ts`): the
first non-mutation use of a digest with a cold lease runs the locked
refresh. It registers the refresh in a per-process map before its first
`await`. Concurrent non-mutation uses of the same digest wait for that
refresh. When the refresh records the lease, they use the asset without
the lock, as warm turns already do. When it records no lease (for
example, the refresh failed or no manifest exists), each waiter takes
the lock as before.
- Mutation uses (`captureSkills`, rebuilds) always take the lock. Their
behavior does not change.
- The locked body moved to a `useLocked` helper without changes.
- Two regression tests in `runtime-asset-retention.test.ts`: one refresh
serves 8 concurrent cold uses, and waiters fall back to the lock when
the shared refresh records no lease.
## Verification
- Red: on `master` the new test fails with `expected [ true, true, true,
true, true, …(3) ] to have a length of 1 but got 8`.
- Green: `npx vitest run
src/services/native-runtime/runtime-asset-retention.test.ts` passes 4/4
with this change.
- Related suites pass: `runtime-context.test.ts`,
`runtime-skill-preservation.test.ts`, `managed-skill-content.test.ts`,
`ephemeral-server-runtime-skill-startup.test.ts` (with the retention
suite: 103 of 104 tests pass). The single failure is
`remote-launch-assets.test.ts` "delivers the selected Codex skills…"
(`expected 256 to be 292`). It fails the same way on unmodified `master`
in this sandbox because the sandbox umask is 077.
- `node server/scripts/typecheck-layers.mjs`: all layers pass (below,
cycle, above, tests, test-tree-0..2), exit 0.
## Risks
- Low risk. The change only removes repeated lock acquisitions while one
refresh for the same digest runs in the same process. The lease
semantics and the deletion lock do not change.
- A waiter uses the asset without the lock only after the shared refresh
recorded a fresh lease. This is the same condition that warm turns use
today.
- If a refresh never settles, its waiters wait with it. Before this
change they waited on the lock and failed after 30 s. The lock
acquisition in the refresh keeps its own 30 s budget, so a refresh
cannot wait for the lock without limit.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1196a3402d |
fix(native): do not report a provider usage-limit run that a restart ended (#22385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs report an unexpected terminal failure to Sentry through the run-failure reporter > - The reporter keeps an expected provider usage-limit failure (`native_provider_usage_limit`) out of Sentry, because the task takes its provider-limit wait and runs again after the reset > - A server restart can end such a run after its provider turn failed on the limit, but before its executor saves `native_provider_usage_limit`. Restart recovery then blocks the run as `provider_checkpoint_permanently_failed`, and the reporter sends it to Sentry > - On staging, one deploy restart sent seven such reports at once. Each task then took its provider-limit wait and finished after the reset, so no operator action was necessary > - This pull request re-derives the usage-limit fact from the run's last committed runner turn terminal and keeps that expected failure out of Sentry > - The benefit is that a deploy during a provider limit does not create false error reports, and real checkpoint failures are still reported ## Linked Issues or Issue Description **What happened?** A deploy restarted the server while several native runs were settling a provider turn that failed on a weekly allowance (`quota_exhausted`). Restart recovery blocked each run as `native_restart_recovery_blocked` / `provider_checkpoint_permanently_failed`. The run-failure reporter sent one Sentry event per run. **Expected behavior** A run that ended on a provider usage limit is an expected failure. The reporter does not send it to Sentry when the executor settles it normally. A restart during settlement must not change that. **Steps to reproduce** 1. Start a native run whose provider turn fails with a `quota_exhausted` or `rate_limit` provider failure. 2. Restart the server before the executor saves `native_provider_usage_limit`. 3. Restart recovery blocks the run as `provider_checkpoint_permanently_failed`, and the reporter sends a Sentry event. Refs #20149 ## What Changed - `native-provider-failure.ts`: move `nativeProviderUsageLimitFromEvent` here from `native-session-executor.ts` so that the evidence reader can use it. The executor re-exports it, so its callers do not change. - `native-provider-failure-evidence.ts`: add `readPersistedNativeTerminalUsageLimit`. It reads the run's last committed turn lifecycle event. It returns true only when that event is a `turn.failed` from the run's own runner instance, the PRP event validates against the stored row (run, session, source, sequence and payload hash), and the failure is a provider usage limit. - `run-failure-report.ts`: for an unexpected `native_restart_recovery_blocked` / `provider_checkpoint_permanently_failed` failure, skip the report when that reader returns true. A read error logs a warning and keeps the report. - Add a real-process restart-recovery test with four cases: usage limit and rate limit are not reported; an unknown failure and a later turn after the limit are still reported. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "provider usage limit|pending retry|unreported"`: 5 passed. - Red proof: with only `run-failure-report.ts` reverted, the `usage-limit` and `rate-limit` cases fail (`expected ... to have a length of +0 but got 1`). The two report cases still pass. - `cd server && PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm run typecheck`: passed. - `npx vitest run src/services/run-failure-report src/services/native-runtime/native-provider-failure src/services/native-runtime/native-session-executor.test.ts`: 1619 passed, 1 failed. The failure (`compares export process compatibility without changing conversation identity (changed: false)`, `db.transaction is not a function`) also fails on unchanged `master` (`ce8006de3c`), so this change does not cause it. ## Risks - Low risk. The change only removes a Sentry report. It does not change run status, task status, recovery, or the provider-limit wait. - The filter needs a validated, run-bound runner terminal. A missing, tampered, foreign or later event keeps the report. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d60cecf625 |
fix(server): show a retry message when managed app sign-in is rate-limited (#22309)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People connect apps such as GitHub through the Paperclip Cloud connector > - The connector `session` call can return 429 with the reason `RATE_LIMITED` > - The OAuth start path does not catch this error, so the server answers 500 > - A 500 tells the user that Paperclip is broken. It does not tell the user to try again later. The error also goes to error reporting as a server crash > - This pull request changes a connector rate limit into a 429 with a clear retry message > - The benefit is that users see "try again in a moment" and the server does not report a false crash ## Linked Issues or Issue Description **What happened?** A user on a Cloud beta workspace started GitHub connect two times. The Paperclip Cloud connector rate-limited the `session` call (status 429, reason `RATE_LIMITED`). Both times, the UI showed an internal server error. **Steps to reproduce** 1. Use a Cloud connector where the `session` call returns 429 `RATE_LIMITED`. 2. POST `/api/companies/:companyId/tools/apps/connect` with `galleryKey: "github"` and `connectionMethodKey: "managed"`. 3. Look at the response status. **Expected behavior** The server answers 429 with a readable message and retry guidance. **Actual behavior** The server answers 500 with "Internal server error". No public GitHub issue exists for this. ## What Changed - `server/src/services/tool-access/oauth/authorization-start.ts`: the managed connector branch catches errors from `startAuthorization`. A `PaperclipCloudConnectorError` with status 429 or broker reason `RATE_LIMITED` becomes an `HttpError` 429. The message is "`<App>` sign-in is busy right now. Try again in a moment.", the code is `paperclip_cloud_connector_rate_limited`, and `details.retryAfterSeconds` comes from the broker `Retry-After` value when the broker sends one. - All other errors pass through with no change. - `server/src/__tests__/tool-access-service.test.ts`: a new route test for a rate-limited managed GitHub connect. It checks the 429 status, the message, the code, `retryAfterSeconds`, and that the raw broker reason is not in the response. ## Verification - Reproduced first: before the fix, the new test failed with `expected 500 to be 429`. - `pnpm exec vitest run src/__tests__/tool-access-service.test.ts` in `server/`: 514 tests pass. - `pnpm typecheck` in `server/`: no errors in `server/src`. The local run reported errors only in sibling packages that the limited local install did not include. CI runs the full check. - The UI uses the `error` string of the response body as the message (`ui/src/api/errors.ts` `apiErrorMessage`), so the user sees the new text. ## Risks - Low risk. The change applies only to the managed connector sign-in start, and only when the connector reports a rate limit. - The response does not set a `Retry-After` header. The retry delay is in `details.retryAfterSeconds`, the same as other 429 responses that the service layer makes. - This change does not change the broker rate limit. A separate investigation must find out why the broker rate-limits the `session` call. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and shell code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: paperclipbugbot <bots+paperclipbugbot@paperclip.ing> |
||
|
|
62bc0519d9 |
fix(native): bound per-turn harness backups by inactivity, not total time (#22382)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a remote native turn is accepted, the executor saves the provider's harness state to durable storage, so another web server can continue the same conversation. > - That save runs under a fixed 300 s total deadline (`AbortSignal.timeout(300_000)`). The deadline covers archive capture, digest, encryption, the upload, any storage-outage wait and a full verifying read-back. > - On staging, five accepted runs failed this save 337–388 s after it began. The run then reported `native_workspace_sync_out_failed` with "The operation was aborted" or tar-stream's "already finalized or destroyed", and workspace copy-back went to retry. > - #19906 fixed the same fixed-deadline rule on the restore side. The write side still uses it. Since #20772/#20871 the save also waits out storage throttling and outages for up to five minutes, which cannot finish inside a five-minute total. > - This pull request bounds the upload and the read-back by inactivity instead of total time. A slow transfer continues while the store makes progress. A transfer that stops fails with a named code. > - The benefit is that an accepted turn no longer loses its provider backup only because the backup is large or the store is slow. ## Linked Issues or Issue Description Refs #19906 (same idle-deadline rule for failover-backup restores), #20772 and #20871 (file-source resends and storage-outage waits during the save), #19108 (stops tar-stream from hiding the real abort), #18932 (durable harness backups). **What happened?** On staging, five native per-turn runs failed within one hour, all after the provider turn was accepted. In each run, `harness_state.backup.persist` failed after 359,588 / 365,422 / 387,471 / 383,801 / 337,335 ms, and `session.checkpoint.persist` failed with it. The only fixed limit on that path is `AbortSignal.timeout(300_000)` around `persistNativeHarnessBackup` in `native-session-executor.ts`. The durations sit just above 300 s, plus a variable sandbox sync-out step before the save. Errors were reported as "The operation was aborted" and, before #19108, as "already finalized or destroyed". **Expected behavior** An accepted turn saves its provider backup while the store keeps making progress, even when the total time is more than five minutes. A write or read-back that stalls fails with a named, diagnosable code. A failed save publishes no backup. **Steps to reproduce** 1. Persist a native harness backup through a store that takes upload chunks slowly, so the full save takes longer than the deadline. 2. With the executor's fixed total deadline, the save fails with "The operation was aborted" and publishes nothing. 3. A store that stops taking bytes, with no caller deadline, waits indefinitely. ## What Changed - `native-harness-backup-storage.ts`: generalize `withDownloadProgressDeadline` into `withTransferProgressDeadline`. It still wraps `getObject` exactly as before, and it now also wraps `putFile` with a five-minute idle timer: - A provider that reads the stream body (local disk) renews it with each chunk it takes. - A provider that sends the file source itself (S3 with `sourceFile`, which never reads the stream) renews it with each request the store answers, each multipart part it takes, and each resend or outage wait that starts. - After the last of them the timer stays armed until the store answers, so a write that never completes is named too. - `persistNativeHarnessBackup` gains `transferIdleTimeoutMs` and `persistMaxMs` (default 30 minutes). The outer bound wraps the whole save, including the storage-outage wait. With an idle limit, a stalled upload throws `native_harness_storage_upload_stalled`, a stalled read-back throws `native_harness_storage_download_stalled`, and the whole save throws `native_harness_storage_persist_deadline_exceeded` at the outer bound. Caller cancellation keeps its own reason, and a storage outage that outlasts its own budget still surfaces as the storage error. Callers that pass no idle limit (controller archive, offline migration, preservation) are unchanged. - `storage/request-stats.ts`: add `observeStorageProgress(onProgress, work)`, an `AsyncLocalStorage` observer called from the existing `recordStorageRequest`, `recordStoragePart` and `recordStorageBackoff` hooks. It is scoped to the wrapped `putFile` call only, and a throwing observer is ignored. Stats collection is unchanged. - `native-session-executor.ts`: the per-turn backup passes `NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS` (five minutes) instead of `AbortSignal.timeout(300_000)`. - Restore behavior and error codes are unchanged: `withDownloadProgressDeadline` now delegates to the shared helper with the same defaults. - `doc/database-backups.md`: document the write-side idle limit, what counts as progress, and the new codes. Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Backup rows, references, receipts and checkpoints are unchanged. A deploy that lands during a save stops the process, and the pending row is never published, as before. ## Verification Run from `server/` (provider env vars unset): - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-upload-deadline.integration.test.ts`: 6 tests passed. They use embedded PostgreSQL, local-disk storage and an incompressible 1 MB acpx backup. - "keeps saving a backup whose upload outlasts the idle window while the store keeps taking bytes". Under the former fixed-total policy the same slow upload fails with "The operation was aborted" and publishes nothing. With the idle limit it completes after more than twice the window and restores. - "names an upload that stops taking bytes and publishes no backup", then saves the same source once the store responds. - "keeps a file-source upload going while the store answers its requests, and names one that stops". The fake store never reads the stream and records each request answer the way the S3 provider does. - "names a read-back that stops sending bytes and publishes no backup". - "stops at the outer bound while an upload still progresses". - "lets Stop end a progressing upload with its own reason". - RED, part 1: with `native-harness-backup-storage.ts` and `request-stats.ts` reverted to `master`, 4 tests fail. Three stall tests hang until the 10 s test timeout, and the outer-bound test resolves instead of rejecting. - RED, part 2: with only the progress observer disabled, the file-source test fails with `native_harness_storage_upload_stalled` on an upload that is still progressing. - `native-harness-backup-*` suites, `native-harness-online-preservation`, `src/storage/`, `storage-request-stats` and `storage-s3-multipart-backoff`: 200 of 201 passed. The one failure, `native-harness-backup-cleanup.test.ts` ("removes retired backups containing immutable skills…", expected mode 256 to be 292), fails the same way on unmodified `master` in this sandbox. - `native-session-executor.test.ts`: 1541 of 1542 passed. The new assertion checks that the per-turn backup passes `transferIdleTimeoutMs` and no fixed signal, and flipping it fails the `opencode-new-durable*` cases. The one failure, "compares export process compatibility without changing conversation identity (changed: false)" (`db.transaction is not a function`), fails the same way on unmodified `master`. - `node scripts/typecheck-layers.mjs`: no errors in `server/`. The local filtered install lacked some CLI/UI/demo packages, which reported unrelated errors. CI covers the full typecheck. Not verified: a live staging run with this build. The staging evidence shows the save failing just after 300 s on several runs. It does not show which step was slow (upload, read-back or local capture), so the fix removes the fixed total for every step on that path. ## Risks - A save that progresses very slowly can now take longer than five minutes. Before, it failed. The settle step waits for it, up to the 30-minute outer bound. - A local step that hangs (for example a stalled disk during capture) is now cut off at 30 minutes instead of 5 minutes, with `native_harness_storage_persist_deadline_exceeded`. - For S3 file-source uploads, progress is measured per request, not per byte. A single request that takes more than five minutes counts as stalled. For a 16 MiB single PUT, that is a store slower than about 55 KB/s. Multipart parts are 8 MiB, eight in flight. - If a successor takes over during a long save, the existing fence checks before and after the upload reject it with `owner_superseded`, and nothing is published. This is unchanged behavior. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3f6c3d37dc |
fix(mcp): keep remote HTTP 405 connection checks out of Sentry (#22376)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Connected apps can be remote MCP servers that Paperclip reaches over
Streamable HTTP
> - A connection health check sends a `tools/list` POST to the saved URL
> - `markMcpHttpResponseFailure` marks 401, 403, 404, 429 and 5xx
answers as known external failures, so the error handler does not send
them to Sentry
> - A remote app answered the health check with HTTP 405, and this
status was not marked, so Sentry recorded an app error
> - A Streamable HTTP MCP endpoint always accepts POST, so a 405 means
that the saved URL is not an MCP endpoint (for example a web page or a
legacy SSE-only path)
> - This pull request marks a 405 answer as `endpoint_not_found`, the
same as a 404
> - The benefit is that a bad remote URL no longer creates a Sentry
error, and the connection still shows the failed check
## Linked Issues or Issue Description
No public issue exists.
**What happened?**
On a hosted staging server, a connection health check for a gallery app
failed with `Error: Remote app returned HTTP 405`. The stack is
`connectGalleryApp` -> `checkConnectionHealth` -> remote MCP
`tools/list` POST. The error handler sent this error to Sentry as an
unexpected app error.
**Expected behavior**
A 405 answer from a remote app is an external failure. The connection
shows the failed check with the `endpoint_not_found` reason. Sentry does
not record it.
**Steps to reproduce**
1. Save a remote MCP app with a URL that answers POST with HTTP 405.
2. Run the connection health check.
3. On `master`, the failure has no known reason, so it goes to Sentry.
The new test in
`server/src/services/__tests__/mcp-connection-failure.test.ts` does
these steps.
**Paperclip version or commit**
`master` at
canary/v2026.1120.0-canary.835400
|
||
|
|
fe0c4aa95d |
fix(recovery): settle a late workspace hold when a remote-epoch continuation already took over (#22387)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery must continue a task after a server is killed. A person must never need to press Retry. > - When Paperclip cannot resume a cut-off run exactly, it records a "fresh continuation". For a remote workspace, the continuation abandons the old run's workspace epoch and restores the last save into a new root. It does not apply the old run's unsaved files. > - Workspace recovery can try to save the old run's files after that. The epoch is already abandoned, so the source is "unavailable". Recovery then wrote an active hold that asked a person to press Retry to continue in a new workspace, which the continuation had already arranged. > - That hold made the dispatch gate withhold every queued run of the task, and the task rested on a Retry banner. > - This pull request settles that hold into the recorded continuation at the moment workspace recovery would leave it exhausted. Dispatch and continuation delivery are unchanged. > - The benefit is that the queued continuation starts automatically after a hard kill, with no Retry banner. ## Linked Issues or Issue Description **What happened?** A restart-chaos release-swap wave hard-killed the old server. One task (of 6) then rested on a Retry banner for about 5 minutes. The other 5 tasks passed. The sequence for the failed task: 1. A run started on the incoming server. The wave's injected database delay held its process-identity write for 480 s, so its runner attach timed out. The run saved a same-run resume. 2. Restart recovery found a newer queued run for the task (a child-completion wake). It treated the same-run resume as superseded, and the run ended `failed` with `native_restart_recovery_blocked`. 3. The dispatch gate withheld the queued run (`execution_reconciliation_required`). In the same transaction, it recorded a fresh continuation with `remote_epoch` isolation, which abandoned the old run's remote workspace epoch. 4. 4 s later, workspace preservation for the old run found the abandoned epoch (`source_unavailable`, `use_alternative_strategy`). It left an active hold: "Paperclip cannot recover the unsaved files from the previous run ... Retry continues the task in a new workspace". The retained-source fallback also refused it later, with `workspace_authority_changed`, because the continuation owned the epoch. 5. The gate withheld both bounded retries because of that hold. The retries were spent, and the task showed a Retry banner. 6. A person pressed Retry. Retry started a run in a new workspace, which the recorded continuation had already decided to do. **Expected behavior** The queued continuation starts automatically. No Retry banner shows. **Steps to reproduce** 1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old server is hard-killed 120 s after SIGTERM. 2. Use an OpenCode task on a remote sandbox whose run starts just before the second swap and that gets a child-completion wake while it runs. 3. Watch the task after the kill. Workspace recovery writes the unsaved-files hold, the gate cancels each queued run with `execution_reconciliation_required`, and the task rests on a Retry banner. `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts` reproduces the database state at step 5. **Paperclip version or commit** `master` at |
||
|
|
414abc2dbf |
fix(execution-target): give a failed Git context probe a stable error code (#22093)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent run starts on a remote execution target (sandbox or SSH), Paperclip runs a small probe that reads the Git context of the remote workspace > - When that probe fails, it throws a plain `Error` with one fixed message and no code > - Run-failure reporting groups failures by error code, never by message. So these failures share one Sentry group with every other setup failure that has no code > - The thrown error also drops the exit facts of the probe. Nobody can see from the run why the probe failed > - This pull request gives the probe failure a stable code and a bounded reason > - The benefit is a separate, correctly named Sentry group for this failure, and run records that show why the probe failed ## Linked Issues or Issue Description **What happened?** On staging, many runs failed during setup with "Could not read execution-target Git context". The burst occurred during a sandbox provider outage. These failures had no error code. The Sentry fingerprint for run failures uses the error code, the adapter, the phase and the root error code or class name. Because the message is deliberately not in the fingerprint, these failures went into the same group as unrelated setup failures. The probe also discarded its exit code, timeout and signal, so the cause of each failure was not recorded. **Expected behavior** A failed Git context probe has a stable error code, so it gets its own Sentry group. The error message names a bounded reason (exit code, timeout, signal, or incomplete or invalid output), so the run record shows why the probe failed. The probe output is never included, because it can contain credentials and paths. **Steps to reproduce** Run `prepareGitHubExecutionEnvironment` with a remote target whose command runner returns a non-zero exit code, a timeout, a signal, or output without the probe frame. On current master the error has no `code` and no reason. ## What Changed - `packages/adapter-utils/src/execution-target.ts`: add `EXECUTION_TARGET_GIT_CONTEXT_UNAVAILABLE_CODE` (`execution_target_git_context_unavailable`). The three probe failure paths now throw an `Error` with this code. The message keeps the old prefix and adds a bounded reason: exit code, timeout, signal, or incomplete or invalid output. - The probe output is never put in the message. - An error that the command runner throws itself (for example a provider HTTP error) is not changed. It keeps its own code and status. - The success condition is not changed. - `packages/adapter-utils/src/github-launcher-environment.test.ts`: add tests for each failure reason, a test that the probe output does not leak into the message, and a test that a runner error passes through unchanged. ## Verification - `vitest run packages/adapter-utils/src/github-launcher-environment.test.ts packages/adapter-utils/src/execution-target*`: 273 tests pass, including the 6 new tests. - Red/green: on the unfixed source, 5 of the 6 new tests fail. The runner-error test passes on both, as intended, because it checks behavior that must not change. - `tsc --noEmit -p packages/adapter-utils/tsconfig.json` passes (sources and tests). - One existing test, "preserves local host credential helpers and validates worktree metadata", fails in my local sandbox on unmodified master too. It depends on the local Git configuration. CI covers it. ## Risks - Low risk. Only the error that the probe throws changes. The success path and the conditions for failure do not change. - `resultJson.failureCause` for these runs becomes `execution_target_git_context_unavailable` instead of `Error`. No code compares `failureCause` with `Error`. - New Sentry events for this failure go to a new group. The old shared group stops receiving them. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used as an agent with tool use and code execution (shell, Git, Vitest). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
34a553a4b6 |
fix(ui): mark the announcement card's own cancellations as intentional aborts (#22380)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The browser UI shows an optional announcement card. The `useAnnouncement` hook fetches it and cancels that fetch when the tab is hidden, when the user dismisses the card, or when the hook unmounts > - The hook aborts with no reason. The app handles that rejection. But a `fetch` wrapper outside the app, such as a browser extension, can drop it. The browser then reports `AbortError: signal is aborted without reason` as an unhandled rejection, and Sentry records an error that has no user effect > - #22090 adds a marked abort reason, `intentionalAbortReason()`, and a Sentry `beforeSend` rule that drops an unhandled rejection only when its value is that exact marked object > - This pull request makes the announcement hook use that marked reason for its own cancellations > - The benefit is that hiding a tab no longer appears as a production error, and every other `AbortError` is still reported ## Linked Issues or Issue Description Refs #22090, which is merged. This change uses its `intentionalAbortReason()` marker and its Sentry `beforeSend` rule. The diff against `master` contains only the `useAnnouncement` change. **What happened?** A hosted browser reported one unhandled rejection: `AbortError: signal is aborted without reason` (DOMException code 20, mechanism `onunhandledrejection`, native trusted promise, tab visibility `hidden` less than one second after the change). The stack is: `document` `visibilitychange` → `useAnnouncement` `onVisibility` → `suspend` → `fetchController.abort()`. **Expected behavior** An intentional cancellation of the announcement fetch does not appear in error monitoring. A real abort failure still appears. **Steps to reproduce** 1. Replace `window.fetch` with a wrapper that calls the real `fetch` and attaches only `.then(() => {})` to its promise. Some extensions and injected scripts do this. 2. Sign in, so that the announcement hook starts its fetch. 3. Hide the tab while that fetch is in flight. 4. The hook aborts the fetch. The wrapper's derived promise rejects with no handler. The app's own chain ends in `.catch()`, so the app does not cause the rejection. **Paperclip version or commit** Reported on release `651749bdb4`. `ui/src/hooks/useAnnouncement.ts` and `ui/src/api/announcements.ts` are unchanged on `master` up to this branch's base. **Deployment mode** Hosted, browser UI (Chrome-family message text). I searched open and closed PRs for this error message and for `useAnnouncement` abort changes. I found only #22090, which this change reuses. ## What Changed - `ui/src/hooks/useAnnouncement.ts`: the hide, dismiss, and unmount aborts now use `intentionalAbortReason()`. This includes the abort of pending dismissal writes on unmount. - The 10-second dismissal-write timeout still aborts with no reason. A timeout is a real failure, not a cancellation, so it stays reportable. - `ui/src/hooks/useAnnouncement.test.tsx`: a new test checks that each of those aborts carries the marked reason. ## Verification - Red first: with only the new test, `cd ui && npx vitest run src/hooks/useAnnouncement.test.tsx` failed (`isIntentionalAbortReason(hidden.reason)` was `false`). - Green (also again after the merge of `master` with #22090, head `afc31cb0ea`): `cd ui && npx vitest run src/hooks/useAnnouncement.test.tsx src/components/AnnouncementWell.test.tsx src/components/AnnouncementWell.onboarding.test.tsx src/lib/sentry.test.ts src/api/client.test.ts` → 5 files, 72 tests pass. - `cd ui && npx tsc --noEmit -p .` shows no errors in the changed files. The 14 errors it reports are in `storybook/support/feature-map` and are the same without this change. - The `beforeSend` filter itself is tested in #22090 with the real Sentry SDK. ## Risks - Low risk. The hook's behavior does not change: the same requests are cancelled at the same times. Only the abort reason object changes, and no code in the hook reads that reason. - The marked reason has the message "The operation was aborted." instead of the browser default. No code depends on the message. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
201abacd6e |
fix(ui): stop reporting the shared GET's own cancellation as an unhandled rejection (#22090)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The browser UI reads API data through a client that shares one
`fetch` between identical in-flight GETs (`ui/src/api/client.ts`)
> - When every caller of a shared GET leaves, the client aborts that
`fetch`. A live-update refetch that cancels an older read does this many
times in a busy tab
> - The app handles that rejection. But a `fetch` wrapper outside the
app, such as a browser extension, can drop it. The browser then reports
`AbortError: signal is aborted without reason` as an unhandled
rejection, and Sentry records an error that has no user effect
> - This pull request marks the abort reason that the client uses, and
makes the Sentry `beforeSend` hook drop an unhandled rejection only when
its value is that exact marked object
> - The benefit is that an intentional cancellation no longer appears as
a production error, and every other `AbortError` is still reported
## Linked Issues or Issue Description
No public issue exists. This is the problem, in the bug template fields:
**What happened?**
A hosted browser reported one unhandled rejection: `AbortError: signal
is aborted without reason` (DOMException code 20, mechanism
`onunhandledrejection`, native trusted promise). The stack is:
query-invalidation-batcher `flush` → `invalidateQueries` →
`refetchQueries` with `cancelRefetch` → React Query `Query.cancel` →
`onCancel` aborts the query signal → `coalescedGet` `onAbort` →
`releaseRef` → `controller.abort()` on the shared GET.
**Expected behavior**
An intentional cancellation of a shared GET does not appear in error
monitoring. A real abort failure still appears.
**Steps to reproduce**
1. Create a `QueryClient` and an observer whose `queryFn` calls
`api.get(path, { signal })`.
2. Replace `window.fetch` with a wrapper that calls the real `fetch` and
attaches only `.then(() => {})` to its promise. Some extensions and
injected scripts do this.
3. Start a refetch. While it is in flight, call `invalidateQueries` for
the same key through the batcher.
4. React Query cancels the first fetch, the shared GET is aborted, and
the wrapper's derived promise rejects with no handler. The app's own
promise chains do not cause the rejection: without the wrapper, the same
steps give zero unhandled rejections.
**Paperclip version or commit**
Reported on release `3a2f71db31`. `ui/src/api/client.ts` and
`ui/src/lib/sentry.ts` are unchanged on master up to this branch's base.
**Deployment mode**
Hosted, browser UI (Chrome-family message text).
Related: #21475 (groups failed-query Sentry reports) and #15940
(attributes stackless unhandled rejections). I searched open and closed
PRs for `releaseRef`, `inflightGets`, and this error message, and found
no duplicate.
## What Changed
- Added `ui/src/lib/intentional-abort.ts`. `intentionalAbortReason()`
makes an `AbortError` and records it in a `WeakSet`.
`isIntentionalAbortReason()` checks that identity without reading the
value.
- `ui/src/api/client.ts`: when the last caller leaves (abort or
timeout), the shared GET aborts with `intentionalAbortReason()` instead
of a reasonless `abort()`.
- `ui/src/lib/sentry.ts`: `beforeSend` returns `null` for an
unhandled-rejection event only when `hint.originalException` is a marked
reason. It also consumes the rejection-event snapshot, so the snapshot
cannot attach to a later report. The unhandled-rejection check is
computed once and reused by the existing diagnostics.
- `ui/src/lib/sentry.test.ts`: the real-SDK test harness now treats a
`null` return from `beforeSend` as a dropped event. Before, `?? event`
sent the event on.
- Tests: the shared signal reason is marked after an abort and after a
timeout, and a caller's own reason is not marked. Against the real
`@sentry/browser` SDK, the marked rejection is dropped and an unmarked
`AbortError` is still sent.
## Verification
- `cd ui && npx vitest run src/api/client.test.ts src/lib/sentry.test.ts
src/lib/browser-error-context.test.ts
src/lib/query-invalidation-batcher.test.ts` → 4 files, 100 tests pass.
- Red check: with the original `client.ts`, the new client test fails
(the reason is not marked). With the original `sentry.ts`, the new
Sentry test fails (the marked rejection is sent).
- A throwaway reproduction (not committed) used a real `QueryClient`,
the invalidation batcher, `api.get`, and a `fetch` stub that rejects
with `signal.reason`. Without a wrapper: 0 unhandled rejections. With
the `.then(() => {})` wrapper: exactly 1 unhandled `AbortError`, which
matches the report.
- `tsc --noEmit` for `ui` reports no errors in the changed files.
## Risks
- Low risk. The filter matches by object identity, and only the client
makes these objects. An `AbortError` from any other source is still
reported.
- The shared GET's rejection message changes from the browser's text to
"The operation was aborted.". Callers already receive their own
`AbortError`, and nothing reads the shared reason.
- If app code drops this rejection in the future, the filter also hides
that. The effect is the same: the request was cancelled on purpose.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1120.0-canary.835000
nightly/v2026.1120.0-nightly.0
|
||
|
|
ce8006de3c |
docs(release): stable notes for the 2026.1120.0-beta.0 soak (v2026.1123.0) (#22372)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release process promotes a soaked beta to stable. The stable GitHub Release body comes from `releases/beta/v<beta-version>.md` on `master` > - Beta `2026.1120.0-beta.0` published in workflow run 44647487455. Its `draft_stable_notes` job pushed a commit-log skeleton to this branch > - The skeleton is a raw commit list. It is not usable as release notes > - This pull request replaces the skeleton with curated stable notes. Each claim traces to a pull request in `v2026.1117.0..28043f8b` > - The benefit is that the stable promotion for this beta (v2026.1123.0) finds finished notes on `master` ## Linked Issues or Issue Description **Issue type** Missing content **Where is the issue?** `releases/beta/v2026.1120.0-beta.0.md` is the stable-notes file for the `2026.1120.0-beta.0` soak. At the branch head before this commit, it holds only the auto-generated commit-log skeleton. **What's wrong?** The stable promotion reads this file from `master` and publishes it as the GitHub Release body. A raw commit list is not suitable as a release body. **Suggested fix** Merge the curated notes. Correct the H1 and the `> Released:` line in a follow-up if the promotion date moves. ## What Changed - Replaced the skeleton in `releases/beta/v2026.1120.0-beta.0.md` with curated stable notes for v2026.1123.0, in the same layout as `releases/v2026.1117.0.md`: overview, Breaking Changes, Highlights, Improvements, Fixes, Upgrade Guide, Contributors - The notes are delta-framed against `releases/v2026.1117.0.md`. Features that the previous stable already describes are not repeated - The Upgrade Guide lists migrations `0452` through `0499` with one-phrase effects, the new environment variables with defaults, and the bundled harness pins - Release and CI internals, smoke specs, canary tooling, Paperclip Cloud work, dependency bumps, and test-only changes are left out ## Verification - Beta publish is complete: npm dist-tag `beta` is `2026.1120.0-beta.0`, tag `beta/v2026.1120.0-beta.0` points at `28043f8b93fa208ac83f96946953d05e7ba1dfd3`, and `publish_beta`, both `smoke_beta` jobs and `draft_stable_notes` in run 44647487455 succeeded. The beta Docker build (run 44679279012) succeeded - `git rev-list --count v2026.1117.0..28043f8b93fa208ac83f96946953d05e7ba1dfd3` returns 1222, which matches the overview and Contributors section - Contributors are the 16 unique GitHub logins of the commit authors in the range, read from the GitHub API with `[bot]` accounts excluded. External handles are thanked by name - Every `#NNNN` link in the file traces to a commit in the range (294 cited pull requests) - Migration numbers, setting keys, env var names, and defaults were checked in the code at the source commit, not in commit subjects - The file contains no internal ticket ids or instance-local links - The diff changes one file under `releases/` and no source files ## Risks - Low risk: a single markdown file, no source changes. If the promotion date moves past 2026-11-23, the H1 and `> Released:` line need a one-line update before the stable dispatch - The `draft_stable_notes` job force-pushes this branch name. A rerun of that job replaces this commit with the skeleton. Do not rerun it for this beta ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5` (Claude Opus 5.5), tool use via Claude Code ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1120.0-canary.834900 |
||
|
|
c41145e72a |
fix: wait on pending questions and quiet provider holds (#22363)
## Thinking Path > - Paperclip manages agents and the tasks they work on. > - Recovery should continue unfinished work when it is eligible. > - An active unanswered question is a persisted reason to wait for a person. > - Treating a later Retry as an answer makes the recovery sweep repeatedly request continuations that admission refuses. > - Provider-limit holds also produce one cancellation toast per task, even though provider work never started. > - This PR aligns recovery with persisted interaction status and presents these waits accurately. ## Linked Issues or Issue Description Refs: #20466. This addresses the question blocker presentation and automatic recovery loop; it preserves the existing interaction-resolution policy. Related work: #19420 schedules provider quota retries, and #21570 handles notices and reply ownership in agent chats. This change concerns assigned-task reconciliation, task/inbox status, and per-run cancellation toasts. **What happened?** The assigned-work sweep ignored pending interactions whenever a newer run carried `explicitUserContinuation`. Continuation admission still refused the pending question, so repeated sweeps recorded unsuccessful continuation attempts. The task could meanwhile show an older provider error as its current blocker. Separately, each pre-dispatch provider-limit hold generated a cancellation toast, and the one-hour fallback was described as a known reset and promised start time. **Expected behavior** Wait quietly while the question remains active under the shared interaction eligibility rules, then react to the existing resolution notification. Dismissed or historical ordinary questions remain answerable without blocking work; governed requests keep their gates. Show the current question on task and inbox surfaces. Keep provider failure history, suppress cancellation toasts for expected provider holds, and describe scheduled availability checks without promising the provider will accept work. **Steps to reproduce** 1. Leave an `ask_user_questions` interaction pending with `supersedeOnUserComment: false`. 2. Create a later failed/interrupted explicit Retry and run assigned-work reconciliation repeatedly. 3. Observe that the question remains pending while the sweep attempts fresh continuations. 4. Separately, let an agent with recent provider rate-limit evidence receive automatic task wakes without a reported reset time. ## What Changed - Remove inferred question supersession from both assigned-work fact loading and candidate selection. Use the existing active-interaction predicate consistently in those reads, fresh-continuation admission, and the task projection. Cover dismissed/historical ordinary questions, retained governed gates, repeated checks, idle workers, and the existing resolution notification. - Add the `question` interruption action, load active pending question IDs alongside approvals in one company-scoped query, and show that blocker on tasks and inbox rows. Preserve a supported manual Retry after a failed attempt, suppress duplicate actions for saved input, and keep other current user gates authoritative. This preserves the existing issue-detail SQL statement budget. Historical run failures remain intact. Refresh the shared source SDK input pin. - Suppress cancellation toasts specifically for cancelled `provider_limit_wait` runs. Label provider retry deadlines as availability rechecks, retain relative-time formatting, and record successful scheduling as an informational lifecycle event. - Document the wait contract in the continuation architecture guide. ## Verification - Demonstrated failing regressions before the implementation: continuation loop, task projection, task notice, cancellation toast, provider deadline wording, and deadline formatting. - Initial focused Vitest qualification: 642 tests passed across 14 files, using: ```sh pnpm exec vitest run \ server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts \ server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts \ server/src/services/recovery/assigned-work-invariant.test.ts \ ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx} ``` - `pnpm --filter @paperclipai/shared typecheck` — passed. - Review follow-up: 157 tests passed in `assigned-work-invariant`, `task-interruption-query`, and `fresh-continuation-fallback`; 575 passed in `task-interruption-query`, `assigned-work-recheck-idle`, `task-interruption-presentation`, `issue-question-context`, `issue-thread-interactions`, and `explicit-native-continuation` (the query file overlaps). The strengthened final dismissed/historical/governed projection regression also passed separately. - Final query-consolidation verification: 157 tests passed across five files, including the unchanged 64-statement issue-detail census and mixed approval/question batches: ```sh pnpm exec vitest run \ server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts \ server/src/services/task-interruption-presentation.test.ts ``` - `pnpm --filter @paperclipai/server typecheck` — passed, including test layers. - Manual-recovery follow-up: 204 tests passed across seven projection/recovery/notice suites, and all 104 Inbox tests passed. The lost-runner integration still starts exactly one run from a manual Retry and leaves the question unanswered. The saved-decision census verifies one decision batch query for a 41-task list, with at most three statements referencing interactions; the overall 64-statement issue-detail limit remains unchanged. - `pnpm --filter @paperclipai/ui typecheck` — passed. - `pnpm --filter @paperclipai/ui build` — passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` — passed. - `pnpm verify` — exit 2: broader CI is required for changed global inputs. Focused tests and package checks supply local evidence; broad CI remains required. - `git diff --check` and a local secret/instance-data scan — passed. - Final CI on `1c16fe7c0d8`: [trusted CI workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842) passed, including all server shards, native-server verification, build, canary, e2e, release registry, and serialized suites. All PR checks are complete, with no failures; the branch is conflict-free. - Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no actionable findings or unresolved review threads. The focused tests cover genuine provider failures, live work and terminal tasks, unchanged task locks/retry delivery, preservation of historical failures, and question resolution waking the affected task while other active pending questions remain quiet. Review regressions also cover dismissed/historical ordinary questions retaining Retry and governed requests retaining their gates. No live task was retried or answered for validation. ## Risks - An active pending question now consistently blocks automatic assigned-work recovery. Shared interaction eligibility determines when ordinary conversation input becomes deferred or historical; governed requests retain their authority. Automatic recovery no longer infers that a newer Retry resolved an active question. - Manual Retry keeps its existing admission policy and recovery checks. The question notice preserves that supported action after a failed attempt without changing approval or credential authority; pending saved input prevents duplicate actions. - The shared interruption action gains `question`; the task and inbox consumers are updated together. The pending-question query returns only task IDs and uses the existing company/task/status index. - Provider cooldown policy, quotas, credentials, routing, cancellation history, and durable retry timing are unchanged. The actual provider can still return a rate limit at the recheck. - No migration is required. Existing large owner modules receive small localized changes; no code file crosses 1,000 lines. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh reasoning effort, with code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1120.0-canary.834800 |
||
|
|
25b5cad7f9 |
fix(native-runtime): start first OpenCode turns from any saved launch-file copy instead of waiting on a stalled one (#22367)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents in remote sandboxes must have a saved, verified copy of their launch files before their first turn (#19622, #21287) > - The first runs of a company share one copy of these files per server. Other runs wait for it > - In a restart chaos wave, the shared copy out of one sandbox stalled for 330 s. Seven other runs waited silently for it. Two runs then failed before their first turn with the cause `unknown` > - The copy speed out of a sandbox differs from one sandbox to the next. When the first copy failed, the other sandboxes copied the same 314 MB in 15 to 24 s > - This pull request lets every waiting run use the first copy that any run saves, starts a second copy when the first one is slow, and names the cause of a stalled copy > - The benefit is a shorter cold start for the first runs of a company, no failed run when another run already saved the files, and a named cause instead of `unknown` ## Linked Issues or Issue Description Refs #19622 Refs #21287 **What happened?** A restart chaos wave started 15 OpenCode runs on new Daytona sandboxes (1 vCPU) in a new company. The first run (A) started the shared copy of the 314 MB launch files out of its sandbox. The copy did not finish. The plugin call timed out after 330 s, and run A failed before its first turn. The other runs waited for that copy without any run event, so the task looked dead for 284 to 343 s. When run A failed, every waiting run started its own copy at the same time. Five copies finished in 15 to 24 s. One copy (run B) stalled again and failed after 330 s, although a saved copy existed after 15 s. One copy (run C) took 5 min and started its turn 10 min after the run started, although it could have used the saved copy 5 min earlier. Both failures had the technical cause `unknown`. Each failure also logged "native result could not be marked workspace_failed; the reconciliation loop will retry persisted results", but the run had no result, so there was nothing to retry. **Expected behavior** A run uses the first saved copy as soon as one exists. A slow copy does not hold all first runs. A stalled copy out of the sandbox has a named cause. A run with no result does not log a result-finalization failure. **Steps to reproduce** 1. Use a company with remote OpenCode agents on Daytona sandboxes with durable storage and no saved launch-file copy. 2. Start 8 or more runs together on new sandboxes. 3. Make the copy out of the first sandbox stall (for example, a slow sandbox link). 4. See that all other runs wait for the full plugin timeout, and see the failures with the cause `unknown`. **Agent adapter(s) involved** OpenCode (native runtime, remote sandbox). ## What Changed - `shareNativeProviderArtifactCapture` (`native-provider-artifact-capture.ts`): - The first capture that saves a ready copy ends the wait for every run. A run whose own capture is still copying uses the saved copy and aborts its own copy. - A run whose own capture fails uses a saved copy when one exists, and fails only when none exists. - When the running capture is slower than 90 s, one waiting run starts one more capture from its own sandbox. At most two captures run at once. A capture keeps its slot until its copy out of the sandbox ends. - The start decision is owned by one `SharedArtifactCapture` coordinator (`tryStart()` returns a release lease). - When the shared capture fails, one waiting run starts the next capture. The other runs keep waiting for it. Before, every waiting run started its own copy at the same time. - The capture callback now gets its own abort signal, so a superseded copy stops before upload. - A failed copy out of the sandbox is now `NativeProviderArtifactExportError` with the code `native_provider_artifacts_export_timeout` (plugin call timed out) or `native_provider_artifacts_export_failed`. The run's technical cause shows this code instead of `unknown`. A timeout shows "Setting up the computer took too long ...". - `runtime-execution.ts`: do not try to mark the workspace as failed on a native result when the run has no result. Before, this always threw `native_finalization_missing` and logged a false "will retry" warning. - `doc/opencode-cold-recovery.md`: describe the new sharing, the second capture and the new cause codes. ## Verification - Red/green: with the old `native-provider-artifact-capture.ts`, these 5 new tests fail. With this change, they pass: - `native-provider-artifact-gate.test.ts`: "starts a run's turn from another run's saved artifact while its own capture is still copying", "uses a saved artifact when the run's own capture fails after another run saved one", "starts one more capture when the shared capture runs past the hedge delay, not one per waiting run", "does not start a capture per waiting run when the shared capture fails", "names a stalled copy out of the sandbox instead of an unknown cause". - Red/green: `heartbeat-native-predispatch-retry.test.ts` "continues on a fresh ... sandbox when the provider artifacts fail to save before any turn" now also asserts that no "could not be marked workspace_failed" warning is logged. It fails before the `runtime-execution.ts` change and passes after it. - `cd server && npx vitest run src/services/native-runtime/native-provider-artifact-gate.test.ts src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts src/services/native-runtime/native-provider-artifact-capture.test.ts src/services/heartbeat/runtime-execution.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/heartbeat-native-predispatch-retry.test.ts`: all pass. - `npx tsc --noEmit -p server`: pass. ## Risks - A slow first capture can now cause one more 314 MB copy after 90 s. At most two copies run at once for one company and manifest. A run whose saved copy cannot be reused (for example, an ownership fence mismatch) starts its own copy at once. Before, a failed burst started one copy per waiting run at once, which was more. - A superseded copy never uploads, but the provider call that copies files out of the sandbox continues until it ends or times out. Its local temporary directory is removed when it ends. - Sharing is still per server. Two servers can still copy the same files. - No change to the rule from #19622: a verified saved copy still exists before the first turn. - No runner or image change. No migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, test runs). Default context window; extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1120.0-canary.834700 |
||
|
|
19a7381398 |
feat: restore personal archiving in task views (#22358)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use Tasks to see shared work and manage the work that needs their attention. > - The old inbox already let each person archive a task and undo that action. > - The combined Tasks surface needs the same controls in Unread and in task table rows. > - The existing inbox archive rules determine when new activity brings a task back. > - This pull request adds an Archived view so people can find and restore their archived tasks. ## Linked Issues or Issue Description **What existing behavior does this improve?** Personal task archiving in Tasks. Related work includes the combined Inbox and Tasks surface (#15100), the Coordinator workspace (#17983), moving Mine into Tasks (#19363), and the existing inbox archive rules (#9931). **Subsystem affected** Cross-cutting. The UI and task list API reuse the existing inbox archive state. **Current behavior** Mine has an archive action. Unread and the task table have no equivalent action. The Tasks menu has no dedicated view for restoring archived tasks. **Proposed behavior** Add Archive and Undo to the task views. Let people restore tasks from Archived. Keep archived tasks available in All tasks. **Reason and benefit** People can clear their personal lists and still find the original task. **Breaking changes** The task list API adds an optional archive filter and viewer archive field. Existing callers use the same response and filters unless they opt in. ## What Changed - Reuse the current inbox archive and restore actions, activity rules, and authorization. - Add Archive and Restore to task table rows, Archive and Undo to Unread, and a personal Archived view. - Give each archived task its own Undo control, including when people archive tasks quickly. - Add tests for the archive list, account isolation, new activity, row actions, and Undo. - Document the archive flow and API filter. Refresh the generated SDK source input registry. ## Verification - Passed 287 focused tests in 12 Vitest files. These cover the task views, row actions, archive cache, and PostgreSQL routes. - Passed local server and UI typechecks. The source input registry check also passed. - Reproduced a missing Undo action when archiving two tasks in a row. After the fix, the 158 task and inbox tests and the 2 toast tests passed. - On macOS with Chrome, archived a task from Mine and restored it using Undo. Archived another task from All tasks, reloaded Archived, restored it, and verified its Todo status and return to Mine. - A human tester also verified Archive, Undo, and Restore in the local browser. - On commit `c5526fd9b1`, GitHub's [verification](https://github.com/paperclipai/paperclip/actions/runs/44630156078/job/135950130662) and [browser tests](https://github.com/paperclipai/paperclip/actions/runs/44630156078/job/135946228919) passed. A targeted retry passed after the first attempt hit a watchdog self-test failure and two cancelled jobs. - [Greptile scored 5/5](https://github.com/paperclipai/paperclip/pull/22358#issuecomment-7024083610) on the same commit. Both review threads are resolved. ## Risks - An incorrect visibility rule could hide a task from a personal list. The archived filter uses the same activity predicate as Mine. The PostgreSQL test covers isolation between people and resurfacing after a reply. - The archive state is already stored per person and company. No database migration is required. ## Model Used - OpenAI Codex, GPT-6, with shell, code editing, and browser verification tools. The serving model ID and context size were unavailable in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <1504818+scotttong@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
24132dac16 |
fix(tasks): verify PR merges with the managed run identity (#22357)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task completion independently verifies that declared PR deliverables are merged. > - Runs can already use an authorized managed GitHub connection to merge private PRs. > - Completion used the background credential resolver, which cannot select that run's managed identity. > - This pull request reuses operation-time identity and grant checks for completion verification. > - Agents can finish delivered work while unavailable grants and unverified merges remain blocked. ## Linked Issues or Issue Description Related: #22244 introduced the native PR merge gate. No duplicate fix was found in the open PR search. The roadmap's personal GitHub identity capability already covers the intended identity model. **What happened?** A native agent with a managed GitHub connection merges a private PR, then completion rejects it as not verified merged. The independent server-side check only tries legacy company token names or an instance token; it does not use the agent run's managed connection. **Expected behavior** Verify the merge with the run's current accepted GitHub identity, retaining current grant, membership, trust and company boundaries. GitHub must confirm the merge regardless of saved work-product metadata. **Steps to reproduce** 1. Configure a managed GitHub identity with access to a private repository and no legacy company GitHub token. 2. Use a native task run to merge and register a PR deliverable. 3. Finish the task. Before this change, the merge verifier cannot authenticate with the managed identity and rejects completion. ## What Changed - Bind completion credential lookup to the persisted company, native task and run. - Add a server-side completion credential path that shares managed grant, membership and trust checks with command credentials. Both active and succeeded runs capture identity under the existing task/run locks and reconcile pending steering acknowledgements before credential lookup. The command credential API still requires a running run. - Resolve credentials once for all PRs in a verification attempt; do no credential work for tasks without PR deliverables. - Preserve legacy fallback only for runs without a managed identity, and preserve the public-only restriction on server-environment credentials. - Add 23 PostgreSQL-backed authorization regression cases and document completion's identity behavior. ## Verification - Local checks cover revision `da2a3608b0375ead646bbbad35af853191488f07`. [Greptile review](https://github.com/paperclipai/paperclip/pull/22357#issuecomment-7023243798) is 5/5 on that revision with both prior threads resolved. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/44625662403) passed on attempt 2 for the same revision, including typecheck, build, canary, server/runner suites and all eight e2e shards. - Red/green: the private-PR and identity-selection cases failed before the initial fix. Review exposed a succeeded-run reassessment failure and then an unsettled-steering gap in that path; each has a regression that failed on the reviewed commit and passes after its correction. Unresolved steering performs no token lookup, and durable acknowledgement selects the new person. The full worker test completes the task after automatic-review withdrawal while command credentials remain unavailable to the finished run. - `pnpm exec vitest run server/src/services/native-runtime/native-pull-request-completion.test.ts server/src/services/native-runtime/native-completion-feedback.test.ts server/src/__tests__/github-operation-credentials.test.ts server/src/__tests__/native-status-reassessment-work.test.ts server/src/__tests__/run-identity.test.ts server/src/__tests__/run-identity-projections.test.ts` — 222 tests pass across six files, on macOS arm64 with Node 25.6.1 and embedded PostgreSQL. - `pnpm --filter @paperclipai/server typecheck` — passes. - `pnpm verify --explain` — exits 2 because the subsystem documentation selects broad CI coverage. Required CI supplies the broader test/build coverage; this exit is not a full-suite pass. - The credential selection, steering, grant and company checks use real PostgreSQL state. Only the secret vault and GitHub HTTP transport are mocked; test requests never reach GitHub. - `git diff --check` and the staged secret/PII scan pass. No runtime artifacts are committed. - Earlier CI revision `4e42f64`: the [Build job](https://github.com/paperclipai/paperclip/actions/runs/44624163145/job/135914203042) ended after the runner received a shutdown signal and the workflow was cancelled. Its log does not show a compiler error before interruption; this interrupted attempt is not build verification. The current revision subsequently passed its own CI build on attempt 2. - Current revision canary: the [Canary Dry Run job](https://github.com/paperclipai/paperclip/actions/runs/44625662403/job/135919504493) was interrupted by runner shutdown before completion. The repository runner-loss classifier returns `lost: true` for its actual annotation and log. A targeted retry was refused while the workflow remained active. The same revision subsequently passed Canary Dry Run and the full workflow on attempt 2; the earlier interrupted attempt remains recorded here separately. - Live acceptance remains pending deployment; the change has not been deployed. ## Risks Completion now reads managed credentials through a server-side path shared with the existing authorization checks. Successful finished runs can be reassessed without granting them command execution credentials. Denied or unavailable identities must remain blocked without falling through to a different identity; regression tests cover revocation and company/task isolation. Credential lookup is shared within one bounded PR verification attempt, not cached across attempts. The existing HTTP deadline and merge proof requirements remain in place. No migrations or API contract changes. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with repository inspection, code editing and test execution. Context-window size is not exposed by the harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1119.0-canary.834500 |
||
|
|
3c32b8d4dc |
fix(ci): waive lockfile bake time for pinned harnesses (#22349)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner ships pinned harnesses that run those agents. > - Some new models require a new harness release before they can run. > - The lockfile bot waits three days before it can merge new dependency versions automatically. > - That wait held the Claude and Codex refresh in #22231. > - This PR waives the wait for a fixed set of harness packages so model updates can ship as soon as npm confirms the releases. ## Linked Issues or Issue Description Refs #22231, #22278, #14942, #15572, and #18238. **What happened?** The refresh in #22231 held 16 new Claude Agent SDK and Codex versions even though the new model support needed those harness updates. **Expected behavior** Verified npm releases of named harness packages can pass the gate immediately. The gate applies the normal minimum age to other packages. **Steps to reproduce** Compare the parent and head of #22231 across the three lockfiles managed by the bot. Run the gate with the publish times in its hold comment and a clock within three days of the releases. ## What Changed - Exempt 28 exact package names from the minimum age. These cover the Claude Agent SDK, Codex CLI, their provider-specific ACP bridges, OpenCode, and the Copilot binaries pinned by Runner. - Take package identities from the Runner manifest and active provider profiles, then check the managed lockfiles for platform binaries. Codex platform aliases use the same registry name with a platform suffix on each version. Pi and Kimi use separate npm locks. Gemini, Grok, the local Cursor CLI, and Antigravity use separate install paths. The Cursor Cloud SDK gets model availability from the service. - Keep the npm publish-time and resolution checks for every entry. Update the bot report and setup documentation. ## Verification - Replayed the exact #22231 lockfile diff using the bot's recorded publish times and a fixed clock. The previous gate held all 16 versions. The new gate verified all 16 and held zero. - Ran the workflow CLI against the same three lockfile pairs and the live npm registry. It wrote `eligible=true` and reported 16 passing versions. - Reconciled the 28 exact names with the Runner manifest, provider profiles, all three bot-managed pnpm lockfiles, and the npm manifests for Claude Agent SDK 0.3.328, Codex 0.170.0, and OpenCode 1.18.34. - Ran `node --test '.github/scripts/tests/*.test.mjs'` locally. All 810 tests passed. The new cases cover mixed refreshes, similar names, registry errors, missing publish times, and non-registry resolutions. - Ran `git diff --check`. - GitHub Actions passed the required typecheck, build, test, and e2e checks on `9c8bccac51`. The Canary runner shut down on the first attempt, and browser shard 7 had a test-company cleanup error. Both passed on the failed-jobs retry. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/44610325309/attempts/2) - Standard Greptile scored the same revision 5/5. [Review](https://github.com/paperclipai/paperclip/pull/22349#issuecomment-7021573041) ## Risks An npm release of an allowlisted harness can reach master during its first three days through the existing automatic merge path. A compromised release can use that same path. Every added version still needs an npm publish time and a registry resolution. Review new platform package names before adding them to the allowlist. Superagent requests security review for this policy change. [Scanner finding](https://github.com/paperclipai/paperclip/pull/22349#discussion_r4634801827) ## Model Used OpenAI GPT-6 in Codex, with shell and code execution. The exact model snapshot and context window were not reported. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
d8ae55198d |
Scope runtime catalog reads to standing-instruction sources (#22356)
## Thinking Path > - Paperclip uses standing instructions from authorized app connections when it prepares a run. > - The gateway already knows which connections have instructions enabled. > - The catalog read still loads tool schemas for every eligible connection in the company. > - This work delays prompt preparation and consumes database and application resources. > - The gateway can restrict the existing catalog query while preserving the names used for authorization. ## Linked Issues or Issue Description **Subsystem affected** — Tool gateway and run preparation. **Problem or motivation** — The instruction resolver filters tool descriptors after the database has returned unrelated catalog entries. **Proposed solution** — Scope the persisted catalog read to possible instruction sources. Include peers that share a candidate’s short connection ID when assigning tool names. **Alternatives considered** — An exact connection ID filter could change a tool name and its policy decision. A separate instruction authorization query would duplicate gateway policy. **Roadmap alignment** — This improves the existing task-start path. Related work includes #22347 and #14864. **Additional context** — The change uses the existing read-only catalog snapshot. It adds no schema or cache. ## What Changed - Pass the instruction candidates to the shared gateway catalog reader. - Read eligible naming peers in the same snapshot. Return only candidate descriptors after assigning their names. - Retain the existing run identity, guest restrictions, model visibility, provider tools, policy decisions and grants. ## Verification - Ran the connection instruction and gateway listing-memory integration suites against embedded PostgreSQL. All 23 tests passed. - Confirmed the bounded-read assertion fails on the baseline and passes on the change. A full listing reads 20 catalog entries from two connections. The instruction lookup reads 10 entries from one. - Covered exact-name approval and block policies, peer quarantine, unchanged instruction text and digest, and visibility of unrelated tools in normal discovery. - Ran integration tests for gateway discovery, gateway service behavior, and assigned MCP tools. All 256 tests passed. - Compared complete instruction resolution with embedded PostgreSQL on macOS, using 16 alternating measured pairs after five warm-ups. The one-tool fixture took 10.05 ms median on the baseline and 10.32 ms on the change. With 1,994 synthetic entries across 21 connections and four candidate entries, the medians were 50.44 ms and 24.47 ms. The median paired saving was 26.03 ms. Final instruction text and digest matched for every pair. - The comparison measures the local instruction resolver. Remote submission-to-turn latency was outside this verification. ## Risks The candidate read uses the first UUID component because it participates in gateway naming. Matching peers contribute to name collisions even when an agent cannot use the peer. The regression checks that a blocked candidate cannot gain access when the catalog is narrowed. Companies that enable instructions on most connections can expect smaller savings. ## Model Used OpenAI Codex directed and qualified the change. GPT-6.2 Sol implemented the change and focused regression tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
28043f8b93 |
Measure database and projection costs in runtime tool discovery (#22347)
## Thinking Path > - Paperclip starts agent turns with instructions for the connections the agent can use. > - The gateway reads the connected-tool catalog during prompt preparation. > - A single discovery span currently covers both database waits and descriptor construction. > - Maintainers need to distinguish these costs before changing catalog reads. > - This change records the transaction, both queries, availability checks, and descriptor construction. ## Linked Issues or Issue Description **Subsystem affected** — Runtime tool discovery and run preparation. **Problem or motivation** — The existing catalog span cannot explain time spent before a remote turn starts. **Proposed solution** — Measure the existing read in place. Record the number of entries and connections returned. **Alternatives considered** — A local microbenchmark cannot reproduce a hosted database or a deployed catalog's sizes. A change to the query would confound the measurement. **Roadmap alignment** — This adds diagnostic detail to an existing operation. Related work includes #14864, which bounded catalog memory use. **Additional context** — The PostgreSQL reads share one read-only snapshot. ## What Changed - Add a transaction span with children for the entry read and the matched-connection read. - Attach the two returned row counts to that transaction. - Time availability checks and descriptor construction under the existing catalog discovery span. - Document which preparation timings can also reach the native run event table. ## Verification - Ran the gateway listing-memory, discovery HTTP, and execution timing context suites with local PostgreSQL. All 30 tests passed. - Checked the parentage and row counts with a real OpenTelemetry exporter in memory. - Ran the standing-instruction integration suite and measured the catalog from two staging application containers with read-only database sessions. ## Risks Each recorded catalog operation adds up to five spans. The existing per-phase bound applies. Only the OTel transaction span includes the aggregate row counts. With or without OTel, the existing native preparation timing path can also persist a run event for each new phase that takes at least 50 ms, subject to its normal caps. The executor defers those writes. The database query, tool eligibility, and authorization contracts stay the same. ## Model Used OpenAI Codex directed and qualified the work. GPT-6.2 Sol implemented the instrumentation and regression test. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting mergenightly/v2026.1119.0-nightly.2 canary/v2026.1119.0-canary.834200 beta/v2026.1120.0-beta.0 |
||
|
|
a640b4dac8 |
chore(lockfile): refresh pnpm lockfiles (#22231)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
|
6be53f0831 |
fix(issues): keep a held task blocked when its agent tries to resume it (#22245)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops with an unknown outcome, recovery can record an
execution hold for the task. Wake admission refuses every wake for the
task until the system continues it from saved work or a person uses
Retry. Recovery sets the task to "Blocked", so a person can see it.
> - An agent can move that task out of "Blocked". It can change the
status, send `reopen` or `resume`, or check the task out. None of these
routes checks the execution hold, so the hold stays.
> - The task then looks workable, but its agent cannot start. Its
scheduled follow-up check (monitor) comes due, and the scheduler tries
it on every tick. Wake admission refuses each try. Agents re-arm the
check, and the check stays overdue for days.
> - #22215 makes these tasks show as "Blocked" in lists. It does not
stop agents from making the state.
> - This pull request refuses an agent request that moves a held task
out of "Blocked". The response is 409 and gives the next action of the
hold.
> - The benefit is that a held task stays "Blocked" until its hold is
released. Its follow-up checks wait and do not retry on every tick, and
the agent gets the correct reason.
## Linked Issues or Issue Description
(B) No issue exists. Bug report:
**What happened?**
Scheduled follow-up checks on agent tasks stayed overdue for many days.
Each due check was skipped with `execution_reconciliation_required`. One
skip record merged more than 25,000 attempts. The tasks were in
`in_progress` or `in_review`, but an execution hold from a failed run
refused every wake. The agent re-armed the checks, and the checks never
started. An agent can move a task out of `blocked` while such a hold
exists. The resume guard and checkout do not check the hold.
**What did you expect to happen?**
A task that wake admission holds stays `blocked` until the hold is
released. An agent that tries to move it gets an error that tells it
what must happen first.
**Steps to reproduce**
1. Make a `blocked` task that is assigned to an agent.
2. Add a recovery record for the task with status `resolved`, cause
`provider_failure_meaning_unverified` and evidence
`automaticRecovery.replay = "blocked"`. `getExecutionBlocker` returns
this hold.
3. As the assigned agent, send `PATCH /api/issues/:id` with `{ "status":
"in_progress" }`. You can also send `reopen` or `resume`, or `POST
/api/issues/:id/checkout` with `expectedStatuses: ["blocked"]`.
4. See that the request succeeds and the task leaves `blocked`. The hold
stays, so wake admission still refuses the task.
5. Give the task an execution policy with a review stage. As the agent,
send `PATCH /api/issues/:id` with `{ "status": "done" }`. The execution
policy changes the status to `in_review` and assigns the reviewer. The
task leaves `blocked`, and the hold stays.
**Paperclip version or commit**
`master` at
|
||
|
|
cf89f0436c |
fix(native-runtime): name the closest assigned tools when a tool name is mistyped (#22327)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runners give agents their assigned app tools through two
wrapper tools: `paperclip_search_assigned_tools` and
`paperclip_call_assigned_tool`.
> - Search returns each tool with a hashed alias
(`app_<readable>_<digest>`). #22213 also accepts an unambiguous gateway
name or display name.
> - Some models copy the alias with an error: they drop or change one
digest character, change the characters of the bot id, or attach the
digest of a different tool.
> - The wrapper correctly rejects these names, because a guess could
select a write tool when the agent meant a read tool.
> - But the error only said "name must be an exact name". The agent
searched again and often repeated the same mistake.
> - This pull request adds the closest names that the agent can use to
the error. It does not resolve any name by approximation.
> - The benefit is a faster recovery after a mistyped name, with no
change to authorization.
## Linked Issues or Issue Description
Refs #22213
**What happened?**
An agent searched its assigned tools and then called
`paperclip_call_assigned_tool` with a name that is almost the returned
alias. For example, one digest character was missing, or the agent used
the digest of the read tool with the name of the submit tool. The call
failed with `assigned_mcp_tool_unknown: name must be an exact name
returned by paperclip_search_assigned_tools. Search assigned tools again
and use the returned name.` In observed runs, the agent made more failed
calls with other wrong names.
**Expected behavior**
The call is still rejected. The error also shows the closest assigned
names that the agent can use, so that the agent can correct the name
directly.
**Paperclip version or commit**
`master` at
canary/v2026.1119.0-canary.833900
|
||
|
|
806440d12c |
feat(runner): relaunch a released task as the next run after a restart (#22343)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An environment plugin can keep a warm task for an agent session. After each turn, Core releases the task's Runner, and the next run of the session resumes it > - #22288 added `resumeIdentityVersion: 1`: for a provider that stops a released Runner, Core rotates the session to the next run and the provider starts the Runner again directly as that run > - Core takes that path only while it still holds the released session in memory. After a Core restart, it recovers the released task through the older hand-off: it resumes the task as the previous run, then switches the Runner's authority to the new run > - So a provider that advertises the capability still has to keep the hand-off for every follow-up after a Core restart, and cannot drop it > - This pull request makes restart recovery take the same path when the provider advertises the capability > - The benefit is that a provider with the capability gets one start shape for a released task, before and after a Core restart, and can retire the hand-off ## Linked Issues or Issue Description Follows #22288. **What existing behavior does this improve?** The first run of a warm plugin session after a Core restart, when its task was released and its provider advertises `resumeIdentityVersion: 1`. **Subsystem affected** The native session executor and the plugin Runner task owner. **Current behavior** After a restart, Core finds the released source on disk (`inspectRemoteWarmSessionRecovery`), creates a backend that adopts the task, and calls `recoverSession`. The task owner calls `resume` without `runner`, the provider starts the Runner as the previous run, and Core hands it to the new run. **Proposed behavior** When the provider advertises the capability, Core recorded the build the task was released on, the process configuration is unchanged, and the session is not moving to another task, Core treats the recovery like a cached released session. It revalidates the recovery, loads the warm checkpoint, and opens a new backend on the same task with `relaunchReleased`. A new task owner takes over the submitted task without adopting a Runner process, prepares the claimant run with the released source as `priorIdentity`, and calls `resume` with `runner`. Every other recovery keeps the hand-off. **Reason and benefit** A provider that advertises the capability no longer needs the hand-off for its own tasks, after a restart or not. ## What Changed - Executor: a remote warm recovery with a released source, a recorded build, a reusable process configuration and no session move, on a provider that advertises the capability, revalidates the recovery and becomes a relaunch: `relaunchReleased` with the released source and build, the warm checkpoint, and no `remoteWarmRecovery` for the backend. The startup context ends as for a cached relaunch. - Executor: the relaunch's task owner can now come from the new backend (`onPluginTaskOwner`), so the unavailable-reuse fallback and the rotation revert work without a retained owner. Without an owner, no relaunch can have started, so the revert applies. - Backend: a relaunch neither stops a resumed task that has no durable binding nor resolves a `resumeFrom` task. - Task owner: an owner created with `relaunch` and the capability takes over an already submitted task, as an adopting owner does, but adopts no process: its prepare sends the released source as `priorIdentity`, and its launcher calls `resume` with `runner`. - `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`: Core prepares the claimant run before the resume, also after a restart, and a provider must accept that prepare without state kept in memory. ## Verification - `server`: `npx vitest run src/services/native-runtime/plugin-runner-release-resume.integration.test.ts`: new `adopted restart relaunch` cases for Codex and Claude, with real runnerd processes. Core restarts while the task is released; the next run relaunches the task as itself, with no claim, one inspection and no second submit. The existing `adopted relaunch` case now relaunches both follow-ups, since the idle close also recovers through this path. - `server`: `plugin-runner-task.test.ts`: a new owner relaunches without submitting or adopting, and prepares with the released source; without the capability it prepares a new task. - `server`: `plugin-runner-release-resume.integration.test.ts` and `plugin-runner-eviction.integration.test.ts`: 61 passed. `native-session-executor.test.ts`: 1535 passed, 7 skipped. `plugin-runner-task.test.ts`: 118 passed. - Typecheck: `server` passes. - Red-green: with the executor lane turned off, both `adopted restart relaunch` cases fail. Without the task owner change, the new-owner relaunch test fails. ## Risks - Only providers that advertise `resumeIdentityVersion: 1` see a change. Recovery without a recorded build, with a changed process configuration, or for a session moving to another task keeps the hand-off. - The provider sees a `prepare` for the claimant run on a released task right after a Core restart. A provider that tracked unclaimed released tasks only in memory must handle that prepare (the doc now says so). - A Core restart in the middle of this path is still not qualified, as for the cached relaunch. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (code search, editing, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1119.0-canary.833800 |
||
|
|
46dde54f52 |
fix(cloud): migrate run evidence during online preparation (#22337)
Drive bounded run-log and provider trace migration pages during online durable-data preparation. Share the page owner and settled-native archive rules with background migration, preserve active-source and storage-integrity guards, and report fresh page results. Keep canceled transfers owned and tracked until the underlying operation settles. Retain page cursors across the preparation budget. Verified 52 focused tests, including embedded PostgreSQL and object storage, server typechecking, and all 58 successful PR checks. Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
264beefeff |
fix(tools): retry a tool audit write that loses a deadlock instead of failing the call (#22329)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents call connector tools through the tool gateway. The gateway writes a `tool_call_events` audit row for each step of a call. > - The audit insert has foreign keys to `companies`, `agents`, `heartbeat_runs` and `issues`. Postgres takes a key-share lock on each parent row, in trigger order: company, agent, run, issue. > - Other writers lock the issue row and then the run row. When such a transaction holds the issue and then asks for the run, while the audit insert holds the run and waits for the issue, Postgres detects a deadlock and aborts the audit insert (`40P01`). > - The gateway then returns the raw `Failed query: insert into "tool_call_events" ...` error to the agent. The agent does not get the tool result. The same call works when the agent sends it again. > - This pull request runs the audit insert through the shared bounded retry that #20971 added for the same lock cycle on `workspace_operations`. > - The benefit is that a short lock conflict no longer fails a connector call or loses its audit row. ## Linked Issues or Issue Description **What happened?** Some connector calls fail with `Failed query: insert into "tool_call_events" ...`. This happens most often when an agent sends several tool calls in parallel in one run. The failed call waits for some seconds, then fails. The other parallel calls succeed. The same call succeeds when the agent sends it again. The Postgres cause is `deadlock detected` (`40P01`). **Expected behavior** A transient lock conflict does not fail the tool call. The gateway writes the audit row and returns the tool result. **Steps to reproduce** 1. Transaction A: lock the issue row `FOR UPDATE`. 2. Call a connector tool for a run on that issue. Its `tool_call_events` insert takes key-share on the company, agent and run rows, then waits for the issue. 3. Transaction A: lock the run row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the audit insert is the victim, and the tool call fails with the database error. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. Transaction A gets a longer `deadlock_timeout`, so the audit insert is always the victim, as in production. The failing statement in the test has the same column list and the same placeholder shape as the production error. Refs #20971 Refs #21703 Open PRs #15830 and #16596 make the audit insert non-fatal. That approach drops the audit row. This change keeps the row. ## What Changed - `server/src/services/tool-gateway.ts`: `writeToolCallEvent` runs its `tool_call_events` insert through `retryTransientLockContention`. The helper retries only lock timeout, deadlock and serialization failure (`55P03`, `40P01`, `40001`): 5 attempts, with 100 ms × attempt backoff. Other errors fail as before. - `server/src/__tests__/tool-gateway-connector-telemetry.test.ts`: new test with a real two-transaction deadlock between an issue-then-run locker and the audit insert of a real `executeTool` call. The fixture now also returns the issue. ## Verification - Failing before the fix: `cd server && npx vitest run src/__tests__/tool-gateway-connector-telemetry.test.ts -t "loses a deadlock"` failed. `executeTool` rejected with `Failed query: insert into "tool_call_events" ...`, cause `deadlock detected`, code `40P01`. - Passing after the fix: the same test passes. I ran it 3 times. - The full `tool-gateway-connector-telemetry.test.ts` file passes (9 tests). - `tool-gateway-service.test.ts` passes (194 tests). - Not run: `pnpm verify` and the server typecheck. The local sandbox has not enough memory and disk for them. CI runs both. ## Risks - Low risk. Only the `tool_call_events` insert retries, and only on the three contention codes. - Each attempt is a new single-statement write. A deadlock victim's statement rolls back completely, so a retry cannot write a duplicate row. - The worst added delay is about 1 s of backoff across 5 attempts, and only when the lock conflict continues. - Authorization, screening and approval checks do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution on embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
87ef7097b6 |
refactor(wake-queue): own blocked admission deferral (#22332)
## Thinking Path > - Paperclip runs agents in response to work on a task. > - A person can leave a comment while execution recovery holds that task. > - Wake-queue checks the recovery gate under the task lock. Heartbeat used to choose how to save the blocked input. > - The deferred-work worker later retries the saved receipt by ID. > - This change gives wake-queue responsibility for saving the input before it confirms deferral. ## Linked Issues or Issue Description **Current behavior** The gate and the write that makes a blocked wake recoverable have different owners. Heartbeat also reports an admitted deferral when an update to an existing receipt matches no row. **Proposed behavior** Let the recovery admission operation save the first receipt or update the existing one in the same transaction. Confirm admission only when the write succeeds. Keep replaceable automatic signals on their existing diagnostic path. **Reason and benefit** Changes to recovery admission can now be checked against one rule. A blocked comment must have a durable receipt and remain recoverable across delivery attempts. Related changes include [#22292](https://github.com/paperclipai/paperclip/pull/22292) and [#22290](https://github.com/paperclipai/paperclip/pull/22290). ## What Changed - Added a transaction-bound deferral operation to wake-queue recovery. It handles existing receipts, saved input, and replaceable system signals. - Scoped retry updates to the company, issue, agent, deferred status, and receipt ID. A stale receipt cannot confirm admission or cause a new receipt to be inserted. - Kept case refusals and issue state callbacks in heartbeat. A saved comment uses the existing delivery worker and post-commit effects. ## Verification - Retried a saved comment by receipt ID through the real heartbeat and run-control entry points. While the hold remained, no run started. After the hold cleared, repeated retry delivered the comment to exactly one run. - Forced a rollback after the wake-queue Postgres adapter saved a receipt. The test observed the receipt within the transaction and confirmed its absence after rollback. Tested missing receipts and receipts for another issue or agent. The admission application test covers error propagation and post-commit effects when the transaction fails. - The test-only revision `9f443f9e6b` passed 9 focused PostgreSQL and recovery checks and all 15 issue-wake admission application tests. - The preceding production revision `eb37328472` passed the server TypeScript check, the module boundary check, and 498 tests across five queue and recovery suites. Twelve pairs of ordinary comments issued the same SQL before and after that production change (42 statements for a fresh wake and 35 for a join). - Both updated test files passed in full (249 tests). [CI attempt 3](https://github.com/paperclipai/paperclip/actions/runs/44572369812/attempts/3) passed on `9f443f9e6b` after retrying interrupted jobs. [Standard Greptile review](https://github.com/paperclipai/paperclip/pull/22332#issuecomment-7014835241) scored the same commit 5/5 with no open findings. ## Risks The affected path handles blocked admissions that may race a retry or a reassignment. Existing recovery gates, saved comment IDs, receipt identity, and notifications must survive that race. This change has no migration or new worker. The healthy admission path is expected to issue the same SQL. ## Model Used OpenAI Codex with GPT-6.2 Sol for implementation and verification. Codex directed and reviewed the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
417802ca87 |
fix(plugins): refresh unchanged-version bundle descriptions (#22295)
Refresh known display descriptions in same-version bundled plugin registrations only when every behavioral field matches. Compare manifests in their persisted JSON form and await bundle reconciliation before final volume-free package qualification. Verified 77 focused tests, server typechecking, and all 59 successful PR checks. Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |