mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
62033b9994db20ac2fd4f9253052cf1752abb05f
936
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
afca556728 |
fix(github): clarify configured review task completion (#21748)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub bots publish reviews and report the result of their assigned task. > - A GitHub check and a Paperclip task have separate completion rules. > - The current prompt explains how to wait, but gives little completion guidance. > - A bot can keep a complete review in review while it waits for CI or merge that its assignment does not require. > - This pull request makes the configured completion rules clear in GitHub prompts. > - The benefit is a finite review task with an accurate status and publication evidence. ## Linked Issues or Issue Description Related change: Refs #17318 keeps advisory review waits separate from human attention. This change adds completion guidance in the same GitHub prompt owner. **What happened?** A bot can publish a complete current-head 5/5 review and still wait for CI or merge. Manual request prompts also lack the task completion guidance used by automatic reviews. **Expected behavior** When the configured assignment allows completion at 5/5, a complete published review can finish Done. CI and merge are completion conditions only when the assignment requires them. Explicit tracking rules still apply. **Steps to reproduce** 1. Configure an agent to finish after a complete published current-head 5/5 review. 2. Start a GitHub review with CI results unavailable. 3. Inspect the generated prompt. It explains a response wait but does not explain this completion case or publication safeguards. 4. With this change, automatic, mention, and comment prompts carry the same explicit guidance. **Paperclip version or commit** Base commit: `83541e89ad3b90740ffc61a32c396dddb5834526`. **Deployment mode** Built from source. The prompt change applies to GitHub bot tasks. ## What Changed - Add shared guidance within the GitHub prompt module for configured completion at 5/5 and finite discussion replies. - Require a complete assessment, confirmed publication, and the current head in the 5/5 guidance. - Preserve explicit CI, merge, and below-5 tracking rules. Keep the shared status arbiter and check thresholds unchanged. - Add six prompt cases. State that CI is unavailable in the existing lifecycle completion case. - Document the configuration example and its limits. ## Verification - Six new prompt cases failed before the change. - `pnpm exec vitest run server/src/services/chat-github-review-policy.test.ts`: 40 passed. - `pnpm exec vitest run server/src/services/native-runtime/external-chat-wait.integration.test.ts -t 'settles a published'`: 9 passed. - `pnpm exec vitest run server/src/services/native-runtime/external-chat-wait.integration.test.ts -t 'inherits.*review|published.*guard|unknown publication|pending publication|changed head|inherited.*review'`: 5 passed. - `git diff --check` passed. - `pnpm verify` returned exit 2. The connector document requires broader CI coverage. - Local preflight installed dependencies and built shared and plugin SDK packages. It stopped because Cargo is unavailable. Full local typecheck, build, and test suites were not run. - [Greptile reviewed this commit](https://github.com/paperclipai/paperclip/pull/21748#issuecomment-6894002401) at 5/5 with no actionable findings. Its check passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. - [CI](https://github.com/paperclipai/paperclip/actions/runs/43711597984) passed for `36dd146cdc00b0f3d7f6fc2971f24166ef0f6bf5`. Typecheck, build, Rust/Runner, all server tests, and all eight E2E test groups passed. The final [CI gate](https://github.com/paperclipai/paperclip/actions/runs/43711597984/job/132808715297) passed at 12:17:58 UTC on 2026-11-15. - At 12:23 UTC, all 61 check records were complete: 57 passed, 1 was neutral, and 3 were skipped. Snyk also passed. No checks failed or remained pending. ## Risks This change gives prompt guidance. It does not enforce a score-to-status rule. Custom assignments can still require CI, merge, or later review. The focused tests check prompt content and existing lifecycle behavior; they do not prove every live agent or webhook journey. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. This session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the bug report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My public branch name describes the change and contains no internal ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
819f61f1fd |
fix(evals): repair four cohort-15 chat cases and prefix board links in completion feedback (#21682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 15 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `36101ebcb9`) failed four cases where the agent did the right thing or where our own server text misled it. > - One failure was a harness race introduced by #21667. Two failures came from graders that read free-form text with keyword regexes. One failure came from an unprefixed approval link in the server's completion feedback, plus a grader that required a literal prefix. > - This pull request fixes the harness step, replaces the free-form graders with saved-state and clicked-link checks, and makes the server emit company-prefixed board links. > - The benefit is that these cases measure the feature again. The retained correct outcomes pass, and the relevant broken outcomes still fail. ## Linked Issues or Issue Description Refs #21667 **What happened?** In chat-eval cohort 15, four cases failed: - `projects.share-project`: the harness followed the agent's ID-based project link. The app then rewrote the URL to the project slug and redrew the page. The owner step timed out because Riley's option in the people picker was "detached from the DOM". #21667 introduced this path. - `skills.author-skill`: the saved skill was a correct, richer checklist. Per-sentence keyword regexes rejected it. For example, "ignore the mouse" tripped the forbidden word "ignore". - `inbox-search.show-decisions`: the reply linked the right approval and tasks and said the question was "addressed to a different teammate". A regex required the word "Riley". - `agents.hire-needs-approval`: both replies linked `/approvals/<id>` without the company prefix. The agent copied that link from `server/src/services/native-runtime/native-completion-feedback.ts`. The grader required a literal `/<prefix>/approvals/<id>`, although the board redirects the unprefixed route. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not free-form wording. The server gives agents canonical company-prefixed board links. **Steps to reproduce** 1. Use the retained cohort-15 evidence for the four cases. 2. Run the old graders on the retained skill markdown and decisions reply. Both fail. 3. For share-project, open a project configuration page by ID while the app rewrites the route after a delay, then use the access picker before the rewrite completes. 4. For hire-needs-approval, finish a task with a pending linked approval. The feedback links `/approvals/<id>`. **Paperclip version or commit** master `36101ebcb9` (cohort 15); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`, `runtime-support.ts`): after the harness follows the agent's link, it waits until the URL settles on the project's canonical route key. This reuses the #21667 approach for agent profiles: `settleAgentProfileRoute` now calls a shared `settleRouteKey`, and `projectRouteKey` follows the app's `projectRouteRef`. The owner's picker selection (`addProjectAccessMember`) now retries as one unit. If the dialog or picker closes, or an option detaches, it reopens them, searches again and selects again. It continues only when the picker shows the person and Add is enabled. - `skills.author-skill` (`integrator-skills.ts`): removed the keyword-regex checklist grader, the waiver regexes, and the reply regexes for "global activation" and "attached to me" claims. The saved-state checks stay: one agent-created writable skill, its current revision authored by the responding agent, the same bytes on the skill and the revision, the Skills/Studio inspection with reload, and assignment scope from saved assignments and audit rows. The checklist check now requires the setup literals `375`, `WCAG` and `AA` as whole tokens in the saved SKILL.md and in the rendered editor. Whether the skill still asks for keyboard-only navigation and visible labels, and whether it waives a check, is left to human review. Oracle version `skill-authoring.v10`. - `inbox-search.show-decisions` (`internal-decision-search.ts`): removed all reply regexes, including the `Riley` requirement and the wording-based separation checks. The grade now uses saved state (pending decisions unchanged, answered history unchanged, each person's own attention queue and resolver) and rendered links to the exact approval, the question's task and the history task. Oracle version `decision-search.v3`. - Product (`native-completion-feedback.ts`): the completion feedback now links `/<prefix>/approvals/<id>` and `/<prefix>/issues/<identifier>`. This fixes the approval link and the four sibling task links in the same file. - `agents.hire-needs-approval` (`runtime-hiring.ts`): the harness now clicks each reply's rendered link to its approval and records where the UI lands. The grade accepts a prefixed link, or an unprefixed link that the board redirect lands on `/<prefix>/approvals/<id>` with the proposed hire visible. Without such a link, the harness still opens the approval directly so later phases keep their evidence, and the grade fails. Oracle version `runtime-hiring.v4`. - No eval YAML changed, so the TestOps manifest does not change. `chat-catalog-descriptors.json` records the three oracle version bumps. ## Verification - Red/green tests (each new test fails on the old code and passes on the new code): - `internal-privacy-share-route.test.ts` (real Chromium): the project page rewrites its route after 0, 400 and 1500 ms, and one case closes the picker while its first results render. With the old flow, the 400 ms rewrite and the closed picker both time out on the option click (the cohort-15 failure). With the fix, Riley is added on the settled route in all cases. - `integrator-skills.test.ts`: the retained cohort-15 SKILL.md passes. The old grader rejects it. A stale skill body, another author, and a lost 375 width still fail. - `internal-decision-search.test.ts`: the retained cohort-15 reply and a reworded reply pass. The old grader rejects the retained reply. A missing or wrong approval, question-task or history-task link fails. - `native-completion-feedback.test.ts`: a pending linked approval produces `[the pending approval](/<prefix>/approvals/<id>)`, and a pending review stage produces a prefixed task link. The old code produces `/approvals/<id>`. - `runtime-hiring.test.ts`: unprefixed links that the redirect lands on the right approval pass. The old grader rejects them. A link that did not land, landed on another approval, was not clicked, or is another href still fails. - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: all passed (one negative control in `internal.test.ts` changed from a reply wording to "approved while listing"). - `pnpm test:e2e:runner:typecheck` and `server` `tsc --noEmit`: passed. - `server` vitest for `native-completion-feedback`, `native-current-wake-completion-feedback` and `heartbeat/run-completion`: 201 tests passed. - Narrow paid run of the four full case ids on this branch: run 43657390762 (results are added in a comment). ## Risks - Low risk. The product change only adds the company prefix to links in agent feedback text. It adds one indexed company read, and only on the branches that return a link. - Less is graded automatically, by design. `skills.author-skill` no longer checks that the skill keeps the keyboard-only and visible-label checks or that it does not waive a check. `inbox-search.show-decisions` no longer checks how the reply names resolvers or separates the person's decision from others' and from history. `skills.author-skill` no longer checks whether the reply describes the assignment scope truthfully. A person reviews this meaning. Saved state and links still catch the plausible broken outcomes listed above. - `agents.hire-needs-approval` now needs a reply link that really opens the approval. A reply without a link to its approval fails as before. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
68f641ba0b |
fix(recovery): continue a native run cut off by a sandbox client timeout (#21736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs an agent's provider turn in a sandbox (for example Daytona) and records each failure on the run and its native coordinator > - A Daytona SDK request can hit its client-side deadline during a provider turn. The run then fails with `native_session_interrupted` in an ambiguous state, because the turn started and its outcome is unknown > - That failure blocks the task under a board-owned watchdog hold. No recovery owner selected the hold, because its cause is not an execution-reconciliation cause. The task never ran again, and only a person's Retry could continue it > - Recovery from an infrastructure fault must work automatically, without a person > - This pull request lets the execution-disposition pass hand that hold to the existing fresh-continuation fallback > - The benefit is that the task continues once, in an isolated fresh run, and the cut-off turn is never replayed ## Linked Issues or Issue Description **What happened?** A native run on a Daytona execution environment (native runtime, an OpenRouter model) started its provider turn. A Daytona request then failed with `HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline (DaytonaConfig.requestTimeoutMs, or the per-call operation/execution timeout); any operation already started on the server may still be running.` The run failed with `native_session_interrupted`. The task had no further run: no automatic retry and no continuation. **Expected behavior** The task continues automatically, exactly once, in a fresh run from its saved work. The cut-off provider turn is not replayed. **Steps to reproduce** 1. Start a native run for an assigned task. Let its provider turn start (a `turn.started` event is saved). 2. Make a sandbox request in that turn fail with the Daytona client-side timeout error. 3. The executor saves the coordinator as `terminal_failure` with `recoveryMode: ambiguous_state`, blocks the task, and writes an active board hold with cause `native_session_interrupted`. 4. Run the execution-disposition worker (`settleUnrecoverableExecutions`). Before this change, nothing happens and the task stays blocked. **Paperclip version or commit** `master` at |
||
|
|
01c1e708e9 |
fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself
## Linked Issues or Issue Description
**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.
**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.
**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.
**Paperclip version or commit**
`master` at
|
||
|
|
a78be8b601 |
feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On each warm turn Core sends the plugin a `prepare` operation with the full launch requirements for the new run > - The requirements' `requirementsDigest` covers the run identity, and the run identity changes on every turn > - So a plugin cannot tell that only the run moved, and it must redo its whole preparation every warm turn > - This pull request gives the plugin two digests: one that stays the same while its preparation stays valid, and one that names the run > - The benefit is that a plugin can transfer only the run identity when nothing it reconciles changed ## Linked Issues or Issue Description **What existing behavior does this improve?** The `prepare` operation that Core sends to an environment plugin for a plugin Runner task, on cold starts and on warm attaches. **Subsystem affected** Native runtime, plugin Runner tasks, the Runner launch contract, and the plugin SDK environment task contract. **Current behavior** A warm attach sends the full requirements. The only digest available, `requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it changes on every turn even when the launch is otherwise the same. **Proposed behavior** Core computes two digests for every `prepare`: - `preparationDigest`: the launch requirements an environment reconciles when it prepares a Runner: - the runner, lease and session identity; - provider, model and settings; - the prompt and instruction text; - the runtime context's instruction and skill bundles, working copy and connection instructions. It leaves out the `runId`, `turnId` and `itemId`, and the runtime context's MCP assignment and binding (with `aggregateDigest`, which hashes the assignment). MCP reaches the Runner through Core's tool authority on every run attachment, never through files the environment publishes. Core already continues a warm session across an MCP-only change (`nativeRuntimeContextsMatch` with tool refresh). The binding is also named per run (`native-mcp:<runId>`). So the digest stays the same across warm turns of one Runner, lease and session, including when the agent's MCP access changes. - `runDigest`: `preparationDigest` plus everything it leaves out: `runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is unique to one run's launch. A plugin that advertises `launchDigestVersion: 1` in its task description gets both fields on `prepare`. **Reason and benefit** A plugin can compare `preparationDigest` with its saved preparation and transfer only the run identity when they match, instead of a full preparation. **Breaking changes** None. Plugins without `launchDigestVersion` get no new fields. ## What Changed - `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add `runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The preparation digest drops the MCP assignment and binding; the run digest takes the full requirements and includes them. Both use the same canonical JSON as `runnerLaunchRequirementsDigest`, with their own schema tags so the three digests never collide. Export them from the package and the server's vendored runner shim. - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `preparationDigest` and `runDigest` to the `prepare` operation, and optional `launchDigestVersion: 1` to the task description. The capability gates both digests. - `server/src/services/native-runtime/plugin-runner-task.ts` and `native-session-executor.ts`: send both digests on `prepare` when the description advertises `launchDigestVersion: 1`. - `server/src/services/environment-task-runtime.ts`: recompute both digests from the requirements, and refuse a `prepare` whose digests are missing in part or do not match. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK public source pin. - Tests: - the preparation digest is stable across runs and changes when the runner, lease, session, instructions, model, approval policy or settings change; - with a runtime context, an MCP-only change (the run's binding, no binding, a changed access set) keeps the preparation digest and changes the run digest; - a change to the instruction bundle, instruction entry, a skill bundle, the skill set, the working copy or the connection instructions changes the preparation digest; - the run digest changes per run; - Core sends the digests only when advertised; - the runtime refuses forged or partial digests before it calls the plugin. ## Verification - `npx vitest run src/live/runner-launch-preparation.test.ts` in `packages/paperclip-runner`: 26 passed. - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed. - `npx vitest run source-sdk-public-inputs.test.mjs source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in `tests/runner-e2e`: 59 passed. - `pnpm typecheck` in `server` and in `packages/paperclip-runner`: clean. - Not run locally: the full suite. CI covers it. ## Risks Low risk. The digests are new optional fields behind a plugin capability version. Core computes them and checks them again before it calls the plugin, so a plugin can rely on them. They prove only that the launch requirements are unchanged. A plugin that skips work on a match must still check its own state, such as files it published into the sandbox. If the runtime context carries values that change per run, the preparation digest changes too, so a plugin does its full preparation. That is safe, but the plugin saves nothing. The MCP binding is the one such value Core sets per run today, and MCP is excluded entirely. A plugin that does publish MCP configuration into its sandbox must not rely on this digest for it. Values outside the requirements are not covered at all: the account identity and credentials, the GitHub broker, project IDs, and the plugin's own runtime revision. A plugin must keep checking those itself. A match lets a plugin skip publishing its launch configuration, not computing the descriptor: `prepare` must still return a launch whose `requirementsDigest` matches the current requirements, which carry the current MCP assignment and binding. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8cc26221af |
perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On a warm turn Core attaches the kept Runner to the new run: the plugin prepares the new run, then Core checks managed project authentication > - A warm attach has no submit step, so Core sends a separate `check_project_auth` operation after `prepare` > - That separate operation is one more plugin dispatch and Launcher round trip on every warm turn, about 0.12–0.18 s, and it is serial > - This pull request lets the plugin do the check inside `prepare` and say so, so Core does not send the second operation > - The benefit is a faster warm turn with the same proof and the same failure behavior ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm attach of a plugin Runner task (`plugin-runner-task.ts` `prepareLaunch`). When `launched` is true, Core always calls `checkProjectAuth` after `prepare`. **Subsystem affected** Native runtime, plugin Runner tasks, and the plugin SDK environment task contract. **Current behavior** Cold starts fold the check into submit (`projectAuthCheckAtSubmit`). Warm attaches send `prepare`, then a separate `check_project_auth`. Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s on each warm turn. **Proposed behavior** The `prepared` result has a new optional flag, `projectAuthChecked`. A plugin sets it only when it ran the managed project-authentication check inside this preparation, for this launch's configuration, provider and projects, and the check allowed them. Core then records the same lease proof that an allowed `check_project_auth` records, and skips the separate check. **Reason and benefit** One fewer serial round trip on every warm turn. The plugin already runs the check for this task; this removes a dispatch, not a check. **Breaking changes** None. The flag is optional. A plugin that does not set it, or that saw a refusal, gets the current behavior. Plugins detect the flag with the SDK schema before they send it. ## What Changed - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `projectAuthChecked: true` to the `prepared` result, with a doc comment that defines it. - `server/src/services/environment-task-runtime.ts`: when a `prepared` result has `projectAuthChecked`, record `runnerTaskProjectAuthValidation` for the launch's configuration, provider and projects, without `atSubmit`. This is the proof that `submit` and `read_file` require. - `server/src/services/native-runtime/plugin-runner-task.ts`: do not send `check_project_auth` when the prepared result has `projectAuthChecked`. - Tests: a warm attach with the flag makes only `prepare`; without it, `prepare` then `check_project_auth`. The runtime test covers all four combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`. ## Verification - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server/`: 157 passed. - Not run locally: full typecheck and full suite. CI covers them. ## Risks Low risk. Core trusts the flag the same way it trusts a plugin's `check_project_auth` answer and `projectAuthCheckAtSubmit` today. A refusal still goes through Core's own check, so the conflict error is unchanged. The plugin side ships separately and sends the flag only when its SDK accepts it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, tracing queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5ba548112 |
feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line that a real-agent ramp harness checks (#21695) > - Four finish-line criteria need numbers from inside each server process: main-thread CPU, run event lateness, agent API time before the handler, and missed lease renewals > - The server had no way to give them, so every harness run was at best "incomplete" > - This pull request adds one instance-admin endpoint that returns those numbers per process over a short sliding window, off by default, and connects the harness to it > - The benefit is that a ramp run can measure every server criterion it can see, with near-zero cost when on and no cost when off ## Linked Issues or Issue Description Refs #21695 **Subsystem affected** Server (instance routes, run event append, native lease renewal), database client (statement counter), and the scale-ramp harness in `scripts/scale-ramp/`. **Problem or motivation** The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the server has no endpoint for it. The criteria `main_thread_cpu`, `run_event_lateness_p99`, `agent_api_pre_handler_p99` and `lease_renewals_missed_zero` are therefore always "not measured", and no run can pass. **Proposed solution** Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of one server process over the last 50 to 60 s. Only an instance admin can read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start, default off) turns on collection and the route. Off, the path answers 404 and no collector starts. **Alternatives considered** - The postgres.js `debug` option for the statement counter: rejected, because it captures a stack (`new Error()`) for every query. - Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected, because it includes GC and libuv threads. It is returned as `processCpu` instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux only, null elsewhere). - Default on: the measured cost is negligible (numbers below), but the event-loop monitor wakes the process every 20 ms when idle, and this is a new instance-wide surface. It stays off; the runbook says how to turn it on for a ramp run. ## What Changed - `server/src/services/server-metrics.ts`: a per-process collector. It uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20 ms), `process.cpuUsage()`, the main thread's `/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native `perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing on the hot path awaits, allocates per call or does I/O. - `server/src/routes/instance-server-metrics.ts`: `GET /api/instance/server-metrics`, guarded by the existing `assertInstanceAdmin` (the check that `/instance/database-backups` and other instance routes use). It returns numbers, a random per-process `processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no SQL, names, IDs or hostnames. - `server/src/app.ts`: when the flag is on, two middlewares time agent API requests from arrival to the API router, the route is mounted, and shutdown stops the collector. When the flag is off, nothing is mounted. - The snapshot also carries `dedicatedConnections` from `dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit` since start. The harness ignores it today; it shows dedicated-connection queueing in a restart burst. - `packages/db/src/statement-count.ts` and `lazy-transaction-queries.ts`: a Drizzle logger that increments one counter per statement replaces Drizzle's no-op logger. No edit to `client.ts`. - `paperclip-control-plane-port.ts`: one call records runner `emittedAt` to committed row for each committed (not duplicate) runner event. - `native-execution-lease.ts`: one call counts a lease whose renewal deadline passed. - Harness: missed lease renewals are summed per `processId` (several processes, or a restart, no longer cancel out). A metrics URL on the target origin gets the board token in the `Authorization` header; `PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin gets no token. - Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the serializer's own output. A server test asserts the serializer produces exactly this file, and a harness test parses it and checks every field. - `rules/company-scope-exceptions.mdc`: adds the snapshot as an instance-wide exception with its reason. The runbook in the plan doc documents the path, the flag, the credential and each field. | Field | Provided | Source | | --- | --- | --- | | `mainThreadCpu` | yes (Linux; null elsewhere) | `/proc/self/task/<pid>/stat` utime+stime | | `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` | | `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s p99 in the window | | `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT or raw driver calls) | | `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue time | | `runEventLatenessP99Ms` | yes (null with no events) | runner event append | | `agentApiPreHandlerP99Ms` | yes (null with no agent requests) | arrival and router-entry middlewares | | `leaseRenewalsMissed` | yes, counter since process start | native lease deadline timer | ## Verification - `cd server && npx vitest run src/__tests__/instance-server-metrics-routes.test.ts src/__tests__/server-metrics.test.ts`: through `createApp` on the embedded test database. 200 with the exact field set for an instance admin; 403 for a company owner who is not an instance admin, for an agent key, and unauthenticated; 404 and no collector when the flag is off; values move after agent requests, statements, a late run event and a missed renewal; the body has no IDs, keys or SQL. - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks (red/green checked for the lease hook by removing the call). - `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts src/client-options.test.ts`: the counter counts statements inside transactions and savepoints. - `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture contract and the board-token header. - `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in `packages/db`: pass. Overhead, measured on a shared, loaded laptop (load average 8 to 20), median with min and max: | Measure | Off | On | | --- | --- | --- | | Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) | 3.36 ns (3.32 to 6.16) | | CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us (20.3 to 42.4) | 32.3 us (20.7 to 45.5) | | CPU per HTTP request, client and server in one process (2,000 x 6) | 67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) | | Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08% (0.05 to 0.09) | | Record one run event (Date.parse + histogram) | - | 148 ns (135 to 188) | | One snapshot | - | 55 us | The statement and request differences are inside the run-to-run spread. The direct cost is about 3 ns per statement, against about 30 us of CPU per statement. ## Risks - Low risk. Off by default; off, only the statement counter runs, and it costs one increment more than Drizzle's no-op logger. - `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other platforms. The harness then uses `eventLoopUtilization`. - Run event lateness compares the runner's clock with the server's clock, so clock skew adds to it. Negative values count as zero. - With several processes, each request reaches one process. The harness keeps the highest gauge and sums counter growth per process, but a process it never samples is not seen. - The statement counter does not count BEGIN/COMMIT or raw postgres.js calls. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
11b09b7234 |
refactor: remove doc/execution-semantics.md and its references (#21699)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents read the docs under `doc/` for background on run execution > - `doc/execution-semantics.md` grew to 6,310 lines of per-PR notes > - It was hard to read and it drifted from the code > - The code and tests are the source of truth for execution behavior > - This pull request removes the file and every reference to it > - The benefit is one less stale document that readers can mistake for a requirement ## Linked Issues or Issue Description **Issue type** Outdated or unclear documentation. **Where is the issue?** `doc/execution-semantics.md`, and the docs, rules, skills, maps, code comments and log text that link to it. **What's wrong?** The file is 6,310 lines of notes added one PR at a time. Much of it repeats or contradicts the code. Readers and agents treat it as a contract, but the code and tests decide the behavior. **Suggested fix** Delete the file. Remove each link to it. Keep the documents that link to it. Point at code where a pointer is still useful. Do not move the content to another doc. Refs #10664 (an open docs PR that edits this file; it becomes obsolete). ## What Changed - Delete `doc/execution-semantics.md`. - Remove links and pointers in `doc/PRODUCT.md`, `doc/LOW-TRUST-PRESETS.md`, `doc/TASK-WATCHDOG.md`, `doc/composer-stop.md`, `doc/native-workspace-finalization-recovery.md`, `doc/run-log-events.md`, `doc/runner-internals.md` and `doc/workspace-repositories.md`. - `doc/TASK-WATCHDOG.md` now points at `server/src/modules/active-run-watchdog/` for the silent active-run watchdog. - Remove the link in `rules/upgrade-safe-run-state.mdc` and `maps/feature-map/recovery.md`. - Update the skills `diagnose-why-work-stopped`, `fix-failed-paperclip-run` and `terminal-bench-loop`. The `fix-failed-paperclip-run` skill now points at `server/src/services/heartbeat/` and `server/src/services/recovery/`. - Remove the file from the source anchors in `.devin/wiki.json`. - Remove the doc name from four code comments in `server/` and `ui/`. - The failed-attachment log message no longer ends with a link to the removed doc. Two tests now check the message text instead of the link. - Leave two dated plan files under `doc/plans/` unchanged. They are history. ## Verification - `git grep -i execution-semantics` finds no hits outside `doc/plans/`. - Every relative Markdown link in the changed files resolves. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs` passes (808 tests). - `pnpm verify` was not run locally. The worktree has no `node_modules`. CI runs the changed Vitest files. ## Risks - Low risk. Most changes are docs and comments. - One runtime log line is shorter. It no longer has a link. No logic changes. - Open PRs that edit the deleted file, such as #10664, will conflict. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and shell access. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
adc3356ad3 |
fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native (Paperclip Runner) run claims its `native_run_finalizations` coordinator before it starts any provider work. The claim is one transaction. It locks the coordinator row, then the run row. > - A writer that holds the run row and then waits for the coordinator row closes a lock cycle. PostgreSQL then aborts the claim with a deadlock (`40P01`), because the claim waited first. > - The aborted claim rolls back whole and starts no provider work. But the raw database error failed the run before its first turn. > - The failure path then preserved the prepared workspace. That stamped the unclaimed coordinator `native_continuation_requires_reconciliation`, and `failRun` used it as the run's error code. The task waited for reconciliation. > - A transient database fault must never end a run like this. The target rate of runs that need Retry is 0%. > - This pull request retries the claim transaction on transient lock contention with the shared bounded helper. > - The benefit is that a fresh run that loses a lock race at its claim continues to its provider turn by itself. ## Linked Issues or Issue Description **What happened?** On a hosted QA stack, a fresh native assignment run (not a retry) failed after about 50 s. The run status was `failed`. The error code was `native_continuation_requires_reconciliation`. The error text was `Database query failed (40P01)`. The stderr was empty. The task needed reconciliation. **Steps to reproduce** 1. Prepare a native run with an unclaimed coordinator (`observed`, attempt 0), as runtime selection leaves it. 2. In another transaction, lock the run row. 3. Start the native session. Its claim locks the coordinator and waits for the run row. 4. In the other transaction, lock the coordinator row. 5. PostgreSQL aborts the claim with `40P01`, and `executePaperclipNativeSession` rejects with that error. **Expected behavior** The claim retries after the competing transaction commits. The run reaches its provider turn. **Paperclip version or commit** `master` at |
||
|
|
958eda6f16 |
fix(native): wait for a live outgoing server before resuming its warm session in a deploy overlap (#21697)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap. The outgoing server still serves while the incoming server starts new runs. > - A warm agent session stays with the server that ran its last turn. That server closes the session about 1 s after it leaves new run starts to the replacement (#21627). > - A follow-up that started within about 1 s of its predecessor's finish read the session as suspended, while the outgoing server was still closing it. The follow-up did not wait. It started a second Runner with the wrong OpenCode launch files, and the Runner refused the launch. > - The same-run retry then went to the outgoing server, which had already left run starts. The run failed, and a blocked parent showed a Retry during its scheduled resume. > - This pull request makes the follow-up wait for the live outgoing server, rebuilds the admitted launch files on the new Runner, keeps same-run retries on the replacement, and shows the scheduled resume for a blocked parent. > - The benefit is that a follow-up during a deploy overlap finishes without a failed run or a Retry banner. ## Linked Issues or Issue Description Refs #21627 **What happened?** A zero-downtime release-swap test (Gate B, wave B1 on master |
||
|
|
97a167093d |
fix(queue): send a message into a starting run without a cancel or recovery errors (#21661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can queue a message while an agent works, then send it into the running turn > - A run that is still starting takes queued messages into its first turn, and Interrupt hands off to them > - Both rules only accepted a queue that a user started > - A user's message can merge into an earlier automatic wake that waits behind the task, and that wake keeps its system requester > - Then the starting run did not take the message, Interrupt cancelled the boot, and the task showed "Couldn't finish this attempt" with Retry and "Waiting for execution recovery" under the composer until the cancelled boot settled > - This pull request makes both rules judge the messages and the recorded Interrupt intent, not the wake's requester, and shows an Interrupt hand-off as sent > - The benefit is that sending a queued message feels immediate and shows no error ## Linked Issues or Issue Description **What happened?** A run started from a resolved blocker. The user queued a message 20 seconds later, while the runner was still starting. The message merged into an older automatic deferred wake. The queue offered Interrupt instead of sending it. Interrupt cancelled the starting run. For about 7 seconds the task showed "Couldn't finish this attempt. Your message is saved." with Retry, and the composer showed "Waiting for execution recovery. Your message is saved." Then the message was delivered. **Expected behavior** The starting run takes the message into its first turn. When an Interrupt does stop a run, the message shows as sent, with no error banner, Retry, or recovery text. **Steps to reproduce** 1. Have an automatic deferred wake for a task (for example from an earlier blocked execution). 2. Start a run for the task with a Paperclip runner agent. 3. While the runner is still starting, queue a message and click the send control. ## What Changed - `deferredWakeJoinsFirstTurn`: a system or unattributed wake qualifies. Each message must still be the responsible user's own (`firstTurnEligibleComment`). A wake from another user or an agent keeps its own delivery. - Task interruption presentation: find the Interrupt queue by its recorded `queuedCommentInterrupt` intent, not by `requestedByActorType = user`. The hand-off now shows as working for a merged queue too. - Queued-comment read: while a deferred queue carries an Interrupt intent, report an `interrupt_handoff` wait instead of the recovery message. - UI (`TaskChatQueuedMessages`, `IssueDetail`): an `interrupt_handoff` response counts as sent. The messages stay out of the queue and no status or error line shows. ## Verification Red/green tests. Each failed before the fix and passes after it: - `server/src/__tests__/issue-queued-comments-routes.test.ts`: "gives a booting run's first turn the user's message merged into an earlier automatic wake"; the Interrupt fallback case now expects `interrupt_handoff` on the response and on a poll. 144/144 pass. - `server/src/__tests__/task-interruption-query.test.ts`: the hand-off case runs for a user-started and a system-started queue. 32/32 pass. - `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`: "shows an Interrupt as sent while the stopped turn hands off". 43/43 pass. - `first-turn-queued-input.test.ts` 8/8, `issue-queued-comment-queue` 7/7. Server and UI typecheck pass. Root cause from the staging wake diagnostics: the run started at 20:26:54, the message was coalesced at 20:27:17 into a deferred `issue_execution_deferred` automation wake from the previous day, the Interrupt at 20:27:20 recorded `execution_recovery`, and the false-positive hold resolved at 20:27:26. Not run: local runner E2E steering cells. The OpenCode chat-features cells require the trusted CI host. The Codex and Claude `revise-while-running` chat cells failed before steering on a fixture workspace path (`workspaces/` vs `isolated-workspaces/`), which this change does not touch. Staging verification follows the deploy. ## Risks - Low. A wider set of deferred wakes can join a starting run's first turn. Message eligibility, the claim receipt, and exactly-once delivery are unchanged. - An Interrupt intent that never delivers shows "sent" in the composer. The task notice still shows a failure with Retry after its existing grace period. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
648b238be1 |
perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime releases owners of failed runs. When the owner's controller is in a process space that Paperclip cannot verify, it shows controller namespace attention on the task instead. > - On a hosted instance with zero active runs, this writer made about 21% of all work signals. It inserted a new active attention row for the same task about every 26 seconds. > - The tasks were already done or cancelled. Recovery resolved each row as `source_terminal`. The active slot was then free, so the next owner-release pass inserted a new row and published a graph recovery signal again. The loop did not stop. > - A live task had the same defect: after the fresh-continuation fallback resolved its attention, the next pass made the same incident again. > - This pull request stops the writer from surfacing attention for finished tasks. It also stops the writer from making an incident again when that run authority was resolved after its lease expired. Those owners are no longer release candidates and do not set the release deadline. > - The benefit is that an idle instance does not write or publish anything for these runs. Attention still shows once for a live task. ## Linked Issues or Issue Description **What happened?** The failed-run owner release (`settleFailedRunOwner`, expired-owner path) called `surfaceNativeControllerNamespaceAttention` for every expired foreign owner that had no active attention. It did this even when the source task was `done` or `cancelled`. Recovery resolved that attention as `source_terminal`. The next pass found no active attention and inserted a new board row. Each insert published `notifyIssueGraphRecovery` and the other execution-control signals. The stranded-task check then ran again for the task graph. A hosted instance collected about 45,000 rows on 13 finished tasks in this way. **Expected behavior** A pass that changes nothing writes nothing and publishes nothing. A finished task never gets controller namespace attention. A live task gets one incident for each exact run authority. A resolved incident stays resolved until the authority changes. **Steps to reproduce** 1. Make a native run that failed, with an `observed` coordinator whose foreign lease has expired and whose controller host is not recorded. 2. Mark the task done or cancelled, or let the fresh-continuation fallback resolve the attention of a live task. 3. Run the execution-finalization worker with the issue recovery workers for 30 minutes. 4. On master, each task gets about 40 to 60 new attention rows. The writer publishes thousands of signals. ## What Changed - `surfaceNativeControllerNamespaceAttention` (shared helper): before it creates attention, it checks for a resolved incident for this run that was resolved at or after the owner's current lease expiry. If one exists, it writes nothing. - New `settledNativeControllerNamespaceAttention` SQL predicate. It is shared by the helper, the release candidate page, and the release deadline query, so all three use the same rule. - `settleFailedRunOwner` (expired-owner path) does not surface attention when the task is `done` or `cancelled`. The restart-recovery claim path has the same guard. - `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt` skip foreign owners whose task is finished or whose incident is settled. A pass that has nothing to do selects nothing, and no immediate deadline is set again. Local owners keep the normal proof-based release. - The owner release proofs did not change. A foreign owner is still never released without proof. - Tests: `server/src/__tests__/native-failed-run-owner-release-idle.test.ts` (real PostgreSQL, real coordinator and workers, fake clock, statement and signal counters) and three new cases in `native-failed-run-owner-release.test.ts`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-failed-run-owner-release-idle.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/services/native-runtime/native-restart-recovery-isolation.test.ts`: 80 tests pass. - Red on master, green with this change. The test seeds one done task and one cancelled task (each with an earlier resolved `source_terminal` incident) and one live task, then runs 30 idle minutes: - master: 126 attention inserts, 249 attention writes, 7,024 signals, 8,025 statements. - this branch: 1 attention insert (the live task, surfaced once), 2 attention writes (that insert and the fallback's resolution), 36 signals, 350 statements. All of this is startup work. Nothing repeats. - After the fresh continuation settles a live task's attention, six unrelated finalization wakes over 30 minutes: - master: 3 attention inserts, 90 signals, 551 statements, and the release deadline stays due. - this branch: 0 attention writes, 0 run writes, 6 signals (the test's own wakes), 153 statements, and the release deadline is null. - New unit cases: a done or cancelled task never surfaces, and surfaces once when it is reopened. A settled incident stays settled over repeated passes. A renewed authority (a new lease expiry) surfaces one new incident. - The three new unit cases fail on master and pass on this branch. - `tsc --noEmit` for `server` passes. ## Risks - Low risk. The change only removes writes. It does not weaken a release proof, an authorization, a budget, a pause, or an approval check. - An incident that a person resolved or dismissed after the lease expired is not made again for the same authority. A renewed authority (a later lease expiry) still makes a new incident. - A superseded incident (status `cancelled`) still shows again when the superseding action clears. This keeps the current behavior. - The historical rows that the loop made are not removed. This change only stops new rows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21621, #21616, #21617 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
64c7122dae |
perf(native-runtime): stop re-surfacing settled controller namespace attention (#21675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime releases owners of failed runs. When the owner's controller is in a process space that Paperclip cannot verify, it shows controller namespace attention on the task instead. > - On a hosted instance with zero active runs, this writer made about 21% of all work signals. It inserted a new active attention row for the same task about every 26 seconds. > - The tasks were already done or cancelled. Recovery resolved each row as `source_terminal`. The active slot was then free, so the next owner-release pass inserted a new row and published a graph recovery signal again. The loop did not stop. > - A live task had the same defect: after the fresh-continuation fallback resolved its attention, the next pass made the same incident again. > - This pull request stops the writer from surfacing attention for finished tasks. It also stops the writer from making an incident again when that run authority was resolved after its lease expired. Those owners are no longer release candidates and do not set the release deadline. > - The benefit is that an idle instance does not write or publish anything for these runs. Attention still shows once for a live task. ## Linked Issues or Issue Description **What happened?** The failed-run owner release (`settleFailedRunOwner`, expired-owner path) called `surfaceNativeControllerNamespaceAttention` for every expired foreign owner that had no active attention. It did this even when the source task was `done` or `cancelled`. Recovery resolved that attention as `source_terminal`. The next pass found no active attention and inserted a new board row. Each insert published `notifyIssueGraphRecovery` and the other execution-control signals. The stranded-task check then ran again for the task graph. A hosted instance collected about 45,000 rows on 13 finished tasks in this way. **Expected behavior** A pass that changes nothing writes nothing and publishes nothing. A finished task never gets controller namespace attention. A live task gets one incident for each exact run authority. A resolved incident stays resolved until the authority changes. **Steps to reproduce** 1. Make a native run that failed, with an `observed` coordinator whose foreign lease has expired and whose controller host is not recorded. 2. Mark the task done or cancelled, or let the fresh-continuation fallback resolve the attention of a live task. 3. Run the execution-finalization worker with the issue recovery workers for 30 minutes. 4. On master, each task gets about 40 to 60 new attention rows. The writer publishes thousands of signals. ## What Changed - `surfaceNativeControllerNamespaceAttention` (shared helper): before it creates attention, it checks for a resolved incident for this run that was resolved at or after the owner's current lease expiry. If one exists, it writes nothing. - New `settledNativeControllerNamespaceAttention` SQL predicate. It is shared by the helper, the release candidate page, and the release deadline query, so all three use the same rule. - `settleFailedRunOwner` (expired-owner path) does not surface attention when the task is `done` or `cancelled`. The restart-recovery claim path has the same guard. - `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt` skip foreign owners whose task is finished or whose incident is settled. A pass that has nothing to do selects nothing, and no immediate deadline is set again. Local owners keep the normal proof-based release. - The owner release proofs did not change. A foreign owner is still never released without proof. - Tests: `server/src/__tests__/native-failed-run-owner-release-idle.test.ts` (real PostgreSQL, real coordinator and workers, fake clock, statement and signal counters) and three new cases in `native-failed-run-owner-release.test.ts`. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/native-failed-run-owner-release-idle.test.ts src/services/native-runtime/native-failed-run-owner-release.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/services/native-runtime/native-restart-recovery-isolation.test.ts`: 80 tests pass. - Red on master, green with this change. The test seeds one done task and one cancelled task (each with an earlier resolved `source_terminal` incident) and one live task, then runs 30 idle minutes: - master: 126 attention inserts, 249 attention writes, 7,024 signals, 8,025 statements. - this branch: 1 attention insert (the live task, surfaced once), 2 attention writes (that insert and the fallback's resolution), 36 signals, 350 statements. All of this is startup work. Nothing repeats. - After the fresh continuation settles a live task's attention, six unrelated finalization wakes over 30 minutes: - master: 3 attention inserts, 90 signals, 551 statements, and the release deadline stays due. - this branch: 0 attention writes, 0 run writes, 6 signals (the test's own wakes), 153 statements, and the release deadline is null. - New unit cases: a done or cancelled task never surfaces, and surfaces once when it is reopened. A settled incident stays settled over repeated passes. A renewed authority (a new lease expiry) surfaces one new incident. - The three new unit cases fail on master and pass on this branch. - `tsc --noEmit` for `server` passes. ## Risks - Low risk. The change only removes writes. It does not weaken a release proof, an authorization, a budget, a pause, or an approval check. - An incident that a person resolved or dismissed after the lease expired is not made again for the same authority. A renewed authority (a later lease expiry) still makes a new incident. - A superseded incident (status `cancelled`) still shows again when the superseding action clears. This keeps the current behavior. - The historical rows that the loop made are not removed. This change only stops new rows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #21621, #21616, #21617 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
172c267338 |
fix(runner): count returned validation rejections and give a copyable call in repeated-failure corrections (#21666)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents change Paperclip records through the `call_api` tool, and the native runtime runs each tool call through a repeated-identical-failure guard (#21360, #21398) > - Some model hosts drop arguments. A model then sends the same failing call again and again, and each attempt costs a model turn > - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input hash > - The guard counted these refusals and stopped running the call after the second failure. But its correction showed only a `"body": {...}` placeholder, so the model had no concrete call to copy. Also, the guard did not count a route's own 400/422 when it came back as an `ok: false` result > - This pull request puts a copyable call in each correction and counts returned 400/422 rejections. It never counts transient failures > - The benefit is that a model gets the exact call to send on the second failure, and the third identical validation failure does not run, whether the rejection is thrown or returned ## Linked Issues or Issue Description Refs #21360, #21398 **What happened?** In chat-eval cohort #14, `questions-and-approvals.review-plan` (run 470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH /api/issues/{id}` with no `body` 7 and 10 times, all with the same input hash. The pre-dispatch check (`assertBodyPresent`) refused each call and returned HTTP 400. The guard corrected the second failure and refused the rest. But the correction said only `Resend with a top-level body argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The model kept sending the identical call, and each refusal cost another model turn: 2 min 18 s in review-plan and 24 s in private-task. Also, when the route itself returns a 400/422 validation response, `call_api` gives back `{ ok: false, status: 422, ... }` as a result and does not throw. The guard did not count that, so an identical invalid request reached the route every time. **Expected behavior** On the second identical deterministic failure, the agent gets a correction with a call it can copy that passes validation. On the third, Paperclip refuses to run the call and tells the agent to change the arguments or to stop and report. Transient failures (5xx, timeouts, rate limits, conflicts) always run again. **Steps to reproduce** 1. Run a native-runtime agent on a host that drops the `body` argument. 2. Have it update a task with `call_api PATCH /api/issues/{id}`. 3. Look at the tool results. They contain only a `{...}` placeholder. A route's 422 returned as `ok: false` is never counted. **Paperclip version or commit** master |
||
|
|
0ab73d4e30 |
feat(native-runtime): ask the Runner for provider-started turns on native ACPX Claude runs (#21456)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. The transport can ask for it (#21607), and a controller restart during it is safe (#21612). > - No run does this yet, because Core does not ask the Runner for it. > - This pull request makes the native runtime of Core ask, for ACPX Claude. It is the only part that changes what a production run does. > - The benefit is the goal of the whole work: a run with a background command that is still running continues when the command finishes, and does not fail with `native_finalization_missing`. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. **This pull request:** Core sets the option for native ACPX Claude runs. This is part 3. Its base branch is `fix/provider-turn-restart-recovery`. Refs #19912 (design). **What existing behavior does this improve?** What a native ACPX Claude run does after a turn that ends with no result while a background command is still running. **Subsystem affected** Core, the native runtime: `server/src/services/native-runtime/native-session-executor.ts` and `native-provider-turn-wait.ts`. ACPX Claude only. **Current behavior** - The run ends at the end of the turn. The Runner restarts Claude, the background command stops, and the run fails with `native_finalization_missing`. **Proposed behavior** - The native runtime sets `providerStartedTurns: true` in the transport options for ACPX Claude, at the two places that build those options. One function, `nativeProviderStartedTurns`, decides this and also decides if the consumer gets the wait options, so the two cannot be different. - A turn that ends with no result while a background command is still running now waits, for a maximum of 10 minutes, for the turn that Claude starts by itself. That turn is consumed as the next turn of the same run, and its result settles the run. - Other providers and other ACPX agents do the same as today. - Limit of the Runner (see #21421): after a follow-up turn, the Runner waits again only while a background command is still running. A command that finishes while Claude reports on an earlier command gets no follow-up. **Reason and benefit** - The opt-in is in the last part, so each earlier part can merge with no change for a production run. ## What Changed - `server/src/services/native-runtime/native-provider-turn-wait.ts` (5 lines added, 1 removed): `nativeProviderStartedTurns`. - `server/src/services/native-runtime/native-session-executor.ts` (4 added, 2 removed): the option at the two places that build the transport options, and the same function for the wait options. - `server/src/services/native-runtime/native-session-executor.test.ts` (4 added, 2 removed): the two native paths set the option for ACPX Claude and not for other providers. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it, and the run does the same as today. - **A deploy during a run:** a turn that started before the deploy has no opt-in and ends as today. The next turn has the opt-in. - **Rollback of Core:** the previous Core does not set the option, so the Runner does not wait. ## Verification The items below were run on the tree that was verified before the split into three parts (commit `1b15ded1c4` has that tree, byte for byte). The head of this branch (`23e2c13ad3`) is that commit on the new head of #21612; its three server files are the same. - `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`: all passed. - `server/src/services/native-runtime/native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,732 passed, 21 skipped. - The end-to-end behavior is tested in parts 1 and 2 with the real `paperclip-runnerd`, where the test sets the option in the way that this part does. - Live run with a real Claude account, not in CI, at the adapter level of the Runner (see #21316): the cycle of Claude was given as a turn with its own usage, and each Paperclip tool call was admitted. Not run: a live run through Core, `paperclip-runnerd` and the real sidecar with a real Claude account (that path needs an API key, not a local sign-in); a remote sandbox; the full suite. CI supplies the broad coverage. ## Risks - **This part turns the behavior on for each native ACPX Claude run.** A run with no background command is not changed: the Runner waits only when a shell is still running at the end of a turn that has no result. - **This part must not merge before #21612.** Without #21612, a controller restart during a wait or during a provider-started turn fails the run. - **A run can now stay open up to 10 minutes longer** when the background command does not finish or Claude does not start a turn. The agent computer stays active for that time. - **A waiting run can show as "suspicious" after 5 minutes of silence** in the run watchdog. The watchdog only labels a native run; it does not stop it. - **Follow-ups that the Runner cannot wait for are lost,** as the limit above says. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
d4498ae7ac |
fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.
Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.
**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.
**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.
**Paperclip version or commit**
`master` at `df1b12ad8e`.
## What Changed
- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.
## Verification
- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.
## Risks
- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
890dcbdd66 |
fix(native): wait for a live warm session owner during a deploy overlap (#21627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, master |
||
|
|
aaec67b891 |
fix(native): close idle warm sessions in parallel across agents at shutdown (#21597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - At shutdown, the server closes its idle warm Runner sessions before
it prepares the hot-restart snapshot
> - It closed them one at a time, and each full close (checkpoint, then
two stopped-copy collections) takes 3-5 s
> - In a zero-downtime swap test, five closes used the whole 20 s
pre-drain budget. Hot-restart preparation timed out, a checkpoint was
cut mid-way, and shutdown logged "hot-restart shutdown preparation
failed"
> - The same test logged 34 "Delivery reconciliation failed" errors, all
from directory cleanup retries that found the agent directory lock busy
> - This pull request closes idle warm sessions in parallel across
agents, one at a time within an agent, and treats a busy directory lock
as a quiet retry
> - The benefit is that shutdown preparation fits its budget with
several warm sessions, and expected lock contention no longer looks like
a failure
## Linked Issues or Issue Description
**What happened?**
`closeIdleWarmNativeSessions` awaited each warm session's full close in
turn. In swap 2 of a release-swap test, five idle sessions closed at
22:28:42, :46, :51, :56 and :59 (one "transport_close_requested"
diagnostic each). The `hot_restart_prepare` step then timed out at 19.9
s. Separately, `recoverDirectoryRelease` threw
`AgentDirectoryOperationBusyError` when another close held the agent
directory lock. The delivery coordinator logged each one at ERROR level.
I first checked if these sessions should have taken the cheap
released-session detach. They could not: `entry.released` is set only
after a successful `session.release()` through a release/resume-capable
plugin Runner task. The Runner state files from all sandboxes in the
test recorded only `run.prepare`, `session.open`, `turn.start`,
`runner.drain` and `runner.suspend` commands, and no `runner.release`.
The Daytona daemon logs show only `/process/execute` and
`/process/session` calls, and no Runner-task API. So no session in that
test was ever released, and a full close was the only correct shutdown
for them.
**Expected behavior**
Idle warm sessions of different agents close at the same time. Two
sessions of the same agent never close together, because they collide on
the agent directory advisory lock. A cleanup retry that finds that lock
busy stays pending until its claimed retry time and logs no error.
**Steps to reproduce**
1. Keep five or more idle warm sessions on remote sandboxes.
2. Send SIGTERM.
3. The closes run one after another and `hot_restart_prepare` times out.
**Paperclip version or commit**
`master` at
|
||
|
|
009c8884a4 |
perf(attention): read blocker retention state once per page (#21595)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items. One
kind is `blocker_attention`: a blocked task whose terminal blocker needs
a person.
> - When the queue loads a page of blocked tasks, it reads `keep` and
`archived_at` from `decision_retention` for each of them, one full-row
`select` per task.
> - A company hint walks every page. On a hosted instance with 2,112
blocker attention items and one active run, these per-task reads ran at
about 16 per second.
> - This pull request reads the retention state for the whole page in
one narrow statement. It also scopes three remaining run-path attention
hints to the task graph.
> - The benefit is one retention read per page instead of one per
blocked task, with the same decisions.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The `blocker_attention` eligibility read in
`server/src/services/attention-retention.ts` (`eligible()`), and three
run-path producers of company-wide attention hints.
**Subsystem affected**
Server background retention (`attention-retention.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-review-participant.ts`,
`native-runtime/native-workspace-reconstruction.ts`).
**Current behavior**
- For each blocked task whose blocker attention is terminal (and for the
owner variant), `eligible()` ran `select <all columns> from
decision_retention where company_id = $1 and source_kind = $2 and
source_id = $3`. It did this during `load`, once per task, on every
company hint.
- The hosted statement is identified by that column prefix: `id,
company_id, source_kind, source_id, source_activity_at, …`.
- I checked every other full-row reader of `decision_retention`. None
fires per run event:
- `syncItems`: attention summary and list requests, which a board calls
once per load; retention `check` and the sweep, which run for due
entries only; Keep and Archive actions.
- `getState`: user actions only.
- `archiveReviewedManifest`: proposal acceptance only.
- `decisions.list`: only when archive proposals exist.
- `native-provider-capacity-retry`, `native-review-participant` and
`native-workspace-reconstruction` sent `{ companyId }` attention hints,
which make the queue walk every page.
**Proposed behavior**
- `pageEvidence` reads `source_id`, `keep` and `archived_at` for every
blocker root and every terminal blocker on the page in one statement
(ids passed as one JSON parameter). `eligible()` uses that map when it
has page evidence.
- A missing row means "not kept, not archived", exactly as the per-task
read returning no row did.
- The per-entry `check` path still reads one row, now only `keep` and
`archived_at`.
- The three producers send `notifyIssueAttentionRetentionWork` with
their task (and run). That is the task, its parents and the tasks it
blocks, with a company fallback past 100.
- A capacity retry changes the failed-run items of its task.
- A review participant claim changes the review of its task.
- A workspace reconstruction resolves the recovery action of its task,
which can also change the blocker attention of the tasks it blocks.
- The configuration, lifecycle, import and fallback producers keep their
company hints.
**Reason and benefit**
Background work must not grow with the backlog. A page now costs one
retention read instead of up to 100.
## What Changed
- `server/src/services/attention-retention.ts`: batched and narrowed
blocker retention read; `PageEvidence.blockerRetention`.
-
`server/src/services/native-runtime/native-provider-capacity-retry.ts`,
`native-review-participant.ts`, `native-workspace-reconstruction.ts`:
scoped attention hints.
- Tests in `server/src/__tests__/attention-retention-work.test.ts` and
`server/src/services/native-runtime/native-workspace-reconstruction.test.ts`.
## Verification
- `reads blocker retention state once per page, not once per blocked
task` (I saw it fail first). 160 blocked tasks, each behind its own
unassigned blocker; the blockers' retention rows are kept (40), archived
(40), plain (40) or missing (40). The full-size response capture and the
statement log measure the load of both pages:
- before: one `decision_retention` read per task (160), 26,240 result
bytes;
- after: 2 reads (one per page), 5,480 bytes, none of them selecting
`source_activity_at`.
- The same tasks are listed: only plain and missing.
- `check` still drops the kept and archived ones.
- `scopes the attention retention hint of a reconstruction to the task
graph` (I saw it fail first): before, the reconstruction sent `{
companyId }`; after, it sends a hint that names the task.
- `batches 100 blocker graphs with exact terminal timestamps and fresh
terminal visibility` pinned 200 retention reads per page (one per root
and variant). It now expects 1; the rest of that test is unchanged and
passes.
- Not separately tested: the capacity-retry and review-participant hint
changes. Each is one line that passes the task the function already
holds, the same pattern as the reconstruction test.
- `attention-retention-work` (58), `attention-retention-candidates`,
`native-workspace-reconstruction`, `native-review-participant` and
`heartbeat-run-dispatch-work` pass, each file run alone (in `server/`,
against dedicated local `_test` databases).
- `pnpm check:module-boundaries` passes.
## Risks
- The page read happens once, before the per-entry decisions of that
page. Before, each decision read its row just in time. A Keep or Archive
that lands between the page read and an entry's decision is seen by the
next load or by `check`, which re-reads under a row lock, as before.
- A run-path hint that changes more than its task graph would now miss
the wider effect. The 90-day archive delay makes a missed reload cheap.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
c3f9a0c040 |
fix(retention): evaluate native retention blockers for many sessions in one statement (#21588)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep runner checkpoints and harness backups so that a
session can resume. Retention expires them after the session can no
longer resume.
> - A retention blocker decides whether a session can still resume: a
live or recent run, an open or recently closed task, an unsettled
finalization, or a live lease.
> - The startup checkpoint pass evaluated that blocker in its own
transaction, once for each retained checkpoint, with three reads. A
session that stays blocked (for example, a long-lived open task) was
evaluated again for each of its runs on every server start. On a hosted
instance with frequent deploys and a 40-minute overlap of two servers,
this pass ran almost continuously at about 7 checkpoints/s, about 21
statements/s.
> - Backup reconciliation also used four reads and a transaction for
each run hint that reached a backup.
> - This pull request evaluates the blocker for many sessions in one
fixed-text statement. Only candidates that are not blocked enter the
existing locked transaction.
> - The benefit: a restart costs 2 statements for each page of 200
blocked checkpoints instead of 5 statements and 1 transaction for each
checkpoint, and a backup hint costs 2 statements with no transaction.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A hosted instance had only a handful of live agent runs, but its server
main thread was about 95% busy. An analysis of its most-called SQL found
three retention-blocker reads (`company_id, agent_id from
heartbeat_runs`; the session's runs; the session's issues) at exactly
7.0/s each, each in its own transaction. The startup checkpoint pass
sent them once for each retained checkpoint older than 7 days whose
session was still blocked, and it repeated the full pass after every
start.
**Expected behavior**
A checkpoint or backup that is blocked from expiring costs no
transaction and no statements of its own, because one statement
evaluates its session together with the rest of its page. It expires as
soon as its blocker clears: checkpoints on the next startup pass (the
current design), and backups at their exact deadline.
**Steps to reproduce**
1. Create 100 native sessions with 3 finished runs each (finished 10
days ago), settled finalizations, a checkpoint header for each run, and
an open task.
2. Register the startup checkpoint expiration and count statements. Then
register it again, as a restart.
3. On master each start costs 1,502 statements and 300 transactions, and
nothing expires.
**Paperclip version or commit**
`origin/master` at
|
||
|
|
f452b449fd |
fix(recovery): stop re-checking settled native runs that cannot be released (#21590)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.
**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.
**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.
`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.
**Paperclip version or commit**
`master` at
|
||
|
|
e8c79235ca |
perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `attention-retention` queue archives old attention items (failed runs, reviews, questions, recovery actions, blocked tasks). A company hint makes it reload every candidate of the company, 100 per page, with eligibility reads for each page. > - Several run lifecycle producers sent that company hint: every agent status change after a run, the execution-control writes and legacy run terminalization when they name only a run, scheduled retries, and the Runner's task monitor tool. > - Each page also read whole rows it did not use: full `heartbeat_runs` rows (recent ones carry tens of KB of `context_snapshot` and `runner_profile_json`), full `issues` rows (descriptions), and the whole agent roster with its configuration. > - This pull request names the task graph or the source kind in those hints, and reads only the columns each eligibility decision uses. > - The benefit is fewer company reloads per run event and far fewer result bytes per reconstruction, with the same decisions. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds on that pull request's change to `execution-control-notifications.ts`); review that one first. **What existing behavior does this improve?** The attention retention queue (`server/src/services/attention-retention.ts`) and the producers that wake it on run lifecycle paths. **Subsystem affected** Server background retention (`attention-retention.ts`, `retention-work-notifications.ts`, `execution-control-notifications.ts`, `heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `native-runtime/paperclip-runner-tool-authority.ts`). **Current behavior** - `heartbeat/agent-status.ts` sends `{ companyId }` on every agent status change after a run (two places). `heartbeat/retries.ts` sends it for each scheduled retry. `writeExecutionControl` and `notifyExecutionRunWork` send it when the scope names only a run. The Runner's `set_task_monitor` tool sends it for one task. - On a local fixture with 1,500 open tasks and run churn, the queue loaded 47 company candidate rows per second from these hints. - `pageEvidence` read `select *` from `heartbeat_runs`, `issue_recovery_actions` and (through `getTableColumns`) `issues`; `reviewEligibility` read `select *` from `issues`; `interactionEligibility` read the full agent roster; the per-entry `check` repeated full-row reads. **Proposed behavior** - Agent status changes send `{ companyId, sourceKind: "agent_error_alert" }`. The status moves between `running`, `idle` and `error`. All three are invokable for blocker attention (`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions (`agent-eligibility.ts`). Failed-run attention only excludes terminated agents, and this path never sets that. So only agent error alerts can change. - Run-scoped execution-control writes, legacy run terminalization and scheduled retries use `notifyIssueAttentionRetentionWork` with the run id (the run's task, its parents and the tasks it blocks; company fallback past 100). This is the same helper as #21521. The task monitor tool scopes to its task. - `notifyIssueAttentionRetentionWork` matched the run's saved task id with `issue.id::text = …`. It now validates and casts the id, so the primary key is used, as #21552 does for recovery. - Narrow reads, same decisions: - runs: `id`, `agent_id`, `created_at` and the saved task id computed in SQL with the same rules as `readRunIssueId` (`issueId` unless it is absent or JSON null, then `taskId`; only non-empty strings). - recovery actions: `id`, `source_issue_id`. - issues for evidence and `visibleIssue`: the blocker-attention input columns plus `unblock_descriptor` and `blocked_transition_at`. - review issues: `id`, `company_id`, `status`, `execution_state`, `assignee_user_id`. - agent roster for question addressees: `id`, `company_id`, `name`, `reports_to`, `status` (the `AgentOrgRow` that `evaluateAgentInvokability` takes). **Reason and benefit** A run event no longer reloads every attention candidate of the company. A reconstruction no longer transfers whole run, task and agent rows. ## What Changed - `server/src/services/attention-retention.ts`: column lists for runs, recovery actions, issues and the roster; the run probe type replaces whole rows. - `server/src/services/heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `execution-control-notifications.ts`, `native-runtime/paperclip-runner-tool-authority.ts`: scoped attention hints. - `server/src/services/retention-work-notifications.ts`: indexed lookup of a run's saved task. - `server/src/__tests__/attention-retention-work.test.ts`: two tests. ## Verification - `reads only the fields its decisions use from large runs, reviews and agents` (I saw it fail first): 50 failed runs, 50 tasks in review with large descriptions, 10 questions to a paused agent, about 40 KB of payload in each run's snapshot, profile and stdout and in the agents' configuration. It loads the three kinds with the full-size response capture (`full-database-response-capture.ts`). Before: `heartbeat_runs` 6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After: under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions are the same before and after: 50 failed runs, 40 reviews and 10 questions. - `scopes run lifecycle attention hints to the run's task graph and agent status to agent alerts` (I saw it fail first): `writeExecutionControl` and `notifyExecutionRunWork` with a run id send the hint for the task and its parent. `finalizeAgentStatus` sends the agent-alert hint. Before, all three sent `{ companyId }`. - `attention-retention-work` (57 tests), `attention-retention-candidates`, `attention-service`, `execution-control-work` and `heartbeat-agent-status-recovery` pass, each file run alone (in `server/`, against a dedicated local `_test` database). - `pnpm check:module-boundaries` passes. Not done in this pull request, and why: - I did not reproduce `syncItems` re-syncing unchanged rows from the retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old recovery-action and blocker retention rows, run churn), `syncItems` ran 0 times in a 180-second window and `decision_retention` reads were 0 per second. `check` calls `syncItems` only for entries that are already due (older than the 90-day archive delay), and a due entry is archived and leaves the candidates. So I did not batch `syncItems` or skip checks of unchanged rows. - `readIssues` still selects the `issues` columns that blocker attention needs. I did not narrow `issueService.listBlockerAttentionBatch`. ## Risks - A producer that changes more than the run's task graph would now miss the wider effect. Company-level changes (agent configuration, company status, budgets) keep their company hints. The 90-day archive delay makes a missed reload cheap: the item is reloaded on the next hint that covers it. - The saved task id now comes from SQL. The SQL mirrors `readRunIssueId` for strings, absent keys and JSON null. A non-string `issueId` value now yields no task, as before. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
340157d1d6 |
perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.
**Subsystem affected**
Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).
**Current behavior**
`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.
**Proposed behavior**
Design. For each task queue, a run event of run R on task T can change:
| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |
The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.
- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.
Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:
- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.
Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:
1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.
Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.
Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.
- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.
**Reason and benefit**
Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.
## What Changed
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.
## Verification
New tests. I saw each red test fail before the change.
- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).
Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.
Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).
| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |
New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.
With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).
## Risks
- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
e977c3b922 |
feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Part 2 gave the consumer a wait with a limit and two inputs: a hook that ends the wait early, and a start time. > - This pull request gives both from the server for a native ACPX Claude run. A new comment from a person ends the wait. The start time comes from an event that is already saved, so a restart of Core does not make the wait longer. > - The benefit is that a person who comments does not wait up to 10 minutes for the agent to see it. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 3. Its base branch is `feat/core-provider-turn-consumer`. Refs #19912 (design). **What existing behavior does this improve?** Nothing visible yet. What Core gives to the consumer of a native ACPX Claude run. **Subsystem affected** Server: `server/src/services/native-runtime`. **Current behavior** - A comment from a person during a run is queued as a deferred wake. The next run delivers it. This does not change. - Core has no wait for a provider-started turn. **Proposed behavior** - For a native ACPX Claude run, Core gives `executeNativeSession` the option `providerTurnWait`. - One delivery worker does the person-comment check for all runs that wait. The server registers it at startup with the shared delivery work coordinator, as it registers the cost accounting worker. The worker listens to the existing notification for deferred wakes. On each notice it runs one query for each run that waits: a comment from a person on this issue, made after the start of the run, that a deferred wake of this agent names. A comment that the query finds ends the wait of that run. The run then takes the path of today, and the next wake delivers the comment as today. - If a database read fails, that run stays owed a check, and the coordinator runs the worker again after 5 seconds, as for each other delivery worker. A read that fails for one run does not stop the check of the other runs. When no run is owed a check, the worker has no deadline and does no query. - The start time of the wait is the time of the first saved `turn.completed` event of the run that has `awaitingProviderTurn`. With no such event it is empty. - Each query has the company id in its condition. **Reason and benefit** - The comment window starts at the start of the run, not at the start of the wait. A comment made during the prompt turn must not wait for the limit. ## What Changed - `server/src/services/native-runtime/native-provider-turn-wait.ts` (new, 109 lines) and 4 lines in `native-session-executor.ts`. - `server/src/index.ts` (4 lines): the registration of the worker, its stop at server close, and its place in the list of delivery workers that must be ready. `delivery-work-notifications.ts` (1 line): the queue key of the worker. The key needs no database change, because each topic uses the one shared channel. - `server/src/vendor/paperclip-runner/index.ts`: one type export. - Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one case in `native-usage-accounting.test.ts` (25 lines). ## State saved by the previous release - No new persisted state and no schema change. The start time is read from `heartbeat_run_events`. - **A run that the previous release started:** it has no waiting terminal, so the start time is empty. - **Core restarts during a wait:** the new process reads the start time, so the limit is the original one. The consumer then continues the wait (#21454), and the worker checks the comments again from the start of the run, so a comment that was made before the restart still ends the wait. - **Rollback of Core during a wait:** the previous Core stops at the waiting terminal and finalizes the run as `native_finalization_missing`, as today. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres with the real coordinator: a comment from a person in a deferred wake ends the wait; a comment from an agent, a comment before the run, and a comment of a different company do not; the start time comes from the first waiting terminal, and a run of the previous release has none. - The 6 new tests: a read that fails one time is tried again after 5 seconds and finds a comment that is already saved, with no new notice; a notice during a read that fails gives a new check; a read that fails for one run does not hold back a second run, and each run ends only for its own comment; a run that starts to wait during a check is checked; a wait that is aborted is removed, nothing checks it again and no deadline stays; with no run owed a check there is no deadline and no query in one hour of test time. - The test helper now registers the wake intent before it writes the deferred wake, in the same transaction. This is the order of each production writer (`modules/wake-queue/adapters/postgres.ts` and `services/heartbeat/queue.ts`). Production code did not change for this. - Mutation checks: a read that fails clears the owed check (3 tests fail); an aborted wait is not removed (the abort test fails). - Also run, unchanged and green: `delivery-work-coordinator.test.ts`, `budget-enforcement-worker.test.ts`, `server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`. The 7 files together: 1,699 passed, 7 skipped. - `native-usage-accounting.test.ts`: a prompt turn and the provider-started turn after it are each counted one time, with the events as the executor receives them. The cost of the provider-started turn is the session total in the cumulative channel, and it is not added two times. The accounting code is not changed. Not run: a run through the executor with a database; the full suite. CI supplies the broad coverage. ## Risks - **One more query at each start of a native ACPX Claude run,** on `heartbeat_run_events` by company and run. - **The worker is registered in each server process, and it is idle with no wait.** A notice with no run that waits does no query. No wait starts before part 4. - **After an abort during a retry, the 5 second retry of the coordinator runs one more time.** It finds nothing owed and does no query. The coordinator has no call to cancel the retry of one worker. - **A process that does not register the worker does no comment check.** The server registers it with no condition at startup, and native runs execute only in that process. Only a unit test can be in this state. - **A comment ends the wait but does not interrupt a turn that is running.** That is the rule of part 2. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3ae7aba643 |
perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server uses `withDedicatedDbConnection` (packages/db) when an action must hold one database session outside the main pool: advisory transaction locks, lock keepalive, and `pg_settings` reads for workspace lifecycle, finalization, source locks, skill caches and status cards > - Each call built a new postgres.js client, opened a new physical connection (TCP, TLS and startup through the pooler), built a new drizzle client with the full schema, and then ended all of it > - On staging the dedicated application name shows about 21 connection setups per second, and the main thread is pinned (event-loop utilization 0.965) > - This pull request keeps a small set of idle single-connection clients per owner and reuses them > - The benefit is that a dedicated action costs about 0.2 ms of main-thread time instead of 4 to 8 ms, with the same exclusive-session guarantee ## Linked Issues or Issue Description **What existing behavior does this improve?** `withDedicatedDbConnection` opens and closes one new database client and one new physical connection for every call. **Subsystem affected** Database client (`packages/db`) and the server services that take dedicated advisory-lock sessions. **Current behavior** Each call builds a new postgres.js client, opens a new connection (TCP, TLS and startup through the pooler), and builds a new drizzle client with the full schema. On staging (Neon statement statistics, 326 s window) `SET application_name='paperclip-workspace-finalization-lock'` ran 21.4 times per second. Only this factory uses that application name. In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2% inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that is this per-call setup. **Proposed behavior** A caller still gets exclusive use of one session for the whole action. The session and its drizzle client are reused by later actions, so the per-call cost is the action's own queries, not a full connection setup. **Reason and benefit** The server main thread is saturated on staging (event-loop utilization 0.965). Reuse removes about 5 ms of main-thread work per call. **Breaking changes** None. The exported function keeps its signature. `closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are new exports. ## What Changed - `packages/db/src/client.ts`: each `createDb` owner now has one set of dedicated single-connection clients, shared with every private client derived from that owner (so nested dedicated actions use the same set). - An action takes an idle client (last in, first out) or opens a new one. It never waits. Callers hold advisory locks and nest dedicated actions, so a waiting queue could deadlock. - On success the client returns to the idle set if fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed. - When the action throws, the client is closed and never reused. This covers a failed or cancelled transaction, a broken socket, and an interrupted lock wait. - When an action ends the client itself (the source-lock cancel path calls `$client.end`), the client is never reused. - Idle clients keep the owner's idle timeout and max lifetime, so postgres.js still closes idle physical connections after `DATABASE_IDLE_TIMEOUT_SECONDS`. - New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle set and stops retention. Ending the owner `$client` does the same. A later action still works with a connection that closes on release. - `server/src/index.ts`: shutdown closes the dedicated set next to the other private pools. - `doc/DATABASE.md`: documents the idle cap, the "not an admission limit" rule, and the transaction-scoped session state contract. - Session state audit: every current caller uses `pg_advisory_xact_lock` / `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or temporary tables. A reset round trip per call is therefore not added. ## Verification - New `packages/db/src/dedicated-connection-pool.test.ts` (embedded PostgreSQL, 10 tests). - Red on `origin/master`: 200 sequential calls used 200 distinct sessions (`expected 200 to be 1`). All 10 tests failed. - Green with this change, 3 runs in a row: 200 sequential calls use 1 session; 7 concurrent callers each hold a different session for the whole transaction, and afterwards at most 4 stay open and later calls reuse them; a failed transaction's connection is closed; a terminated socket during the action is not reused; an idle connection terminated by the server reconnects; a client the action ended is not reused; transaction-local settings do not leak; shutdown and owner `$client.end` close every retained connection; nested actions do not block. - Focused suites pass: packages/db `work-signals`, `prepared-queries-postgres`, `response-capture-postgres`, `checkpoint-pool`, `client`, `client-options`, `client-teardown-registry`, `postgres-connection-recovery`, `client-disconnect-replay` (132 tests). Server suites that use dedicated connections: `skill-package-store`, `status-card-evaluation-work` (updated: it now proves advisory locks are released, since sessions are reused), `agent-instruction-working-copies`, `workspace-recovery-retry-copyback`, `native-execution-lease-batch`, `company-skills-empty-runtime`, `decisions-service`, `legacy-controller-lease`, `server-startup-feedback-export` (467 tests). - `pnpm verify --explain` selects 2779 files for a `packages/db/src/client.ts` change. Per AGENTS.md I ran the focused checks above and let CI give the broad coverage. - Benchmark: 200 calls of `withDedicatedDbConnection(db, d => d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))` against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating before/after process pairs per mode. Main-thread time is `eventLoopUtilization().active`. | Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call (median, range) | | --- | --- | --- | --- | | TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) | | TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) | | Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) | | Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) | At the staging rate of about 21 calls per second, this removes roughly 100 ms of main-thread work per second (TLS figures; staging CPUs may be slower). ## Risks - A retained connection that the server or network drops while idle: postgres.js reconnects on the next query when it has seen the close (tested). If the drop and the reuse race, the first query of that action fails like a main-pool query would. Before this change, a dedicated action always started on a fresh connection. - Through a transaction-mode pooler, `SET application_name=...` is a server-side parameter switch, so its count may not drop as much as the client-side connection count. Measure the deployed effect with pooler client connections and main-thread CPU, not only that statement. - Concurrency above 4 opens extra connections exactly as before; they close on release. Total retained connections per owner grow by at most 4 idle clients, which still close after the idle timeout. - An action that sets session-level state would now leak it to the next action. The contract is documented on `withDedicatedDbConnection` and in `doc/DATABASE.md`; no current caller does this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
351b69941c |
perf(checkpoint): write a routine controller checkpoint save in one statement (#21553)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote runner's controller saves a checkpoint after each runner event it commits, on a 3-connection pool, before it acknowledges the event > - Each save is a transaction of 4 statements (5 where commit notifications are on): BEGIN, the owner lock with the transaction id, the write, COMMIT > - What forces the transaction is the live-run recovery wake that every save registers. Recovery does not react to a routine save > - This pull request writes a routine save (new or changed event, command and cursor records) with one statement that checks the owner and writes, and keeps today's transaction for every other save > - The benefit is one statement instead of four for most checkpoint saves, on the pool that limits one process in a busy minute ## Linked Issues or Issue Description Refs #21528 (owner lock returns the transaction id). Refs #21549 (load model at recorded rates; the plan records the checkpoint pool as a per-process limit). **What existing behavior does this improve?** `nativeRunnerCheckpointWriter` in `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Subsystem affected** Controller checkpoint persistence for remote native runs. **Current behavior** Every save locks the coordinator row (`native_run_finalizations` `for update`), checks the owner, calls `notifyLiveRunRecovery` (6 recovery queues), and upserts the changed parts, in one transaction. **Proposed behavior** A routine save is one statement: a CTE locks the coordinator row only while the lease owner, boot id, generation and an unexpired lease (`clock_timestamp()`) all match, and the upsert writes only through that row. It registers no recovery wake. If the owner does not match, it writes nothing and the save takes today's full path, which runs the complete owner check (successor rules included) and fails with `native_runner_checkpoint_owner_lost` when it should. A save is routine when it has changed parts, no reset, no removed parts, no header change, no predecessor, and the run is its own authority. Every other save (the first save of a controller, which is a reset, a save after a failed save, header or retirement changes, removals, source-run consumers) is unchanged. **Reason and benefit** At recorded rates about a third of one process's statements are checkpoint saves. In a busy minute the 3-connection pool cannot keep up at 500 agents (#21549). **Breaking changes** None. No schema change. ## What Changed - `ROUTINE_CHECKPOINT_SAVE_SQL` and the routine branch in `nativeRunnerCheckpointWriter`. ## Why a routine save needs no recovery wake I read the six consumers of `notifyLiveRunRecovery`: - `nativeProcessSessions`, `nativeQuestionCancellations`, `activeExecutionRecovery`, `settledNativeProcesses` (`live-recovery-work.ts`): each selects candidates from `heartbeat_runs` status, runtime mode, `native_run_finalizations` phase, lease and retry times, the context snapshot and process identity. None reads checkpoint rows to decide eligibility. - `strandedAgentStatus` (same file, `agents` rows) and `activeRunSilence` (`active-run-watchdog/adapters/deadline-work.ts`, `heartbeat_runs.last_output_at`): neither reads checkpoint rows. - The checkpoint rows matter to recovery in two places: `listExpiredNativeRunnerAuthorities` requires that a checkpoint exists, and the remote bootstrap proof reads its parts. A checkpoint first appears with a reset save, which keeps its wake. A routine save only adds event and command parts, which can only take a run further from the bootstrap proof; recovery reads the current rows when its own deadline (lease expiry) fires. - The intent of the wake also fenced a recovery pass against a save in flight. The routine save is one atomic statement and holds the coordinator row lock while it writes, so a recovery claim that takes that row waits for it, and a reader sees the save whole or not at all. ## Verification - Red/green on embedded PostgreSQL (`native-runner-checkpoint-save.test.ts`); the new tests failed against `origin/master` first: - A routine save is one statement (`with owner as ... for update ... insert into native_runner_checkpoints`) and registers no recovery wake (before: 4 statements and a wake). - A reset save keeps BEGIN, the owner lock with the transaction id, and COMMIT, and still registers its recovery wake. - A routine save that waits while another connection takes over the lease (new owner and generation) falls back to the full path and fails with `native_runner_checkpoint_owner_lost`; the checkpoint is unchanged. - The same race where the new owner also resets the checkpoint (deletes every part and writes its own header): the save fails the same way and the checkpoint holds only the new owner's header. - A lease that expires while the routine save waits for the coordinator row lock (the holder does not change the row) writes nothing: the statement judges the lease after the lock, as `lockNativeRunnerCheckpointOwner` does. My first version checked the lease in the locking CTE itself, and PostgreSQL does not re-check a condition after a lock wait on an unchanged row; the existing `native-runner-restart-recovery.integration.test.ts` test "fences checkpoint delivery when ownership expires during an unchanged row-lock wait" failed on it, and so does the new test. Both pass now. - The existing test "refuses a save after the owner's lease expired and writes nothing" passes through the new fallback. - Mutation checks: with the owner conditions removed from the statement, both race tests and the expired-lease test fail; with the lease check moved back into the locking CTE, the lock-wait expiry test fails. - Other focused suites, one file at a time: `native-runner-restart-recovery.integration` (231 passed, 14 skipped), `native-runner-restart-checkpoint-cost.integration` (1; it greps for `insert into "native_runner_checkpoints"`, so the statement quotes the table name), `native-recovered-owner-followup` (97), `native-controller-recovery.integration` (30), `native-runner-checkpoint-authority.integration` (1), `native-restart-recovery-isolation` (1), the port suite with conformance (105), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51 tests pass; on this busy machine its teardown hook timed out, as it has before on master). `pnpm --filter @paperclipai/server run typecheck` passes. - Statements per routine save: 4 → 1 (5 → 1 where commit notifications are on). I did not rerun the load model for this change: the machine is too busy for a fair timing comparison today, and the statement count is exact. - CPU per operation (a local script, not committed: 600 routine saves of one event part and the cursor for one run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per save 0.38–0.56 ms before and 0.10–0.12 ms after (encryption included in both); wall time per save 0.45–0.73 ms before and 0.22–0.30 ms after. ## Risks - A routine save no longer wakes the six recovery queues. Their own deadlines and the 30 s reconstruction still run; see the reading above for why no decision depends on a routine save. - The owner check of the routine statement accepts only the exact active owner. Any other state goes to the unchanged full path, so a successor or a finished result keeps today's rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b7922570e1 |
fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.
Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.
Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on
|
||
|
|
f6c5986b61 |
feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service
## Linked Issues or Issue Description
**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.
**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.
**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.
**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.
**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.
## What Changed
- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.
## Verification
- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.
## Risks
- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1dee321844 |
perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every native event that announces work to another part of the server (chat progress, a final response, an evidence outcome, recovery) registers a wake in its transaction before it writes the event > - Such an event takes six statements: BEGIN, the run lock read, a replay probe, a read of the transaction id for the wake, the append, and COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock, the transaction id, the write, COMMIT > - In the agent scaling load model these two paths are now the largest per-agent cost after #21511 and #21520, and other events fall behind first at 600 agents > - This pull request returns the transaction id from the lock read that each path already sends, and moves the replay probe into the run lock read when it is safe > - The benefit is four statements for such an event and four for a checkpoint save, with the same locks, the same wake order and the same replay rule ## Linked Issues or Issue Description Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The statement count of native events that register a wake (`appendNativeSourceEvent`) and of controller checkpoint saves (`nativeRunnerCheckpointWriter`). **Subsystem affected** `packages/db/src/work-signals.ts` (one new export), `server/src/services/heartbeat-run-events.ts`, `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Current behavior** `signalDatabaseWork` sends `select pg_current_xact_id()` for the first wake of each transaction. A native event with a wake reads the run lock, then probes for its replay in a separate statement, because a statement that waits for the lock keeps its earlier snapshot. **Proposed behavior** - The run lock read and the checkpoint owner lock read also return `pg_current_xact_id()::text`. Both already lock a row, so the transaction has an id at that point. `primeDatabaseWorkTransactionId` gives it to the transaction's wake state, and the wake sends no statement for it. - For an event that may register a progress, response, reassessment or retention wake, the run lock read also reads `next_event_seq` from its own snapshot and checks for a replay in that snapshot. Every event append advances `next_event_seq` in its transaction. So when the locked row has the same `next_event_seq` as the snapshot, no append committed in between, and the probe result is current. Otherwise, or when the probe finds a replay, the separate probe runs as before. **Reason and benefit** Fewer round trips while a pool connection is held in a transaction. In the load model these paths hold the main pool and the three-connection checkpoint pool longest per event. **Breaking changes** None. No schema change. ## What Changed - `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`, exported from `@paperclipai/db`): inside a transaction whose id is not yet known, it records the id. Outside a transaction it does nothing. An uncertain commit still resolves with `pg_xact_status` on the same id. - `appendNativeSourceEvent`: the lock read returns the transaction id when the event has any wake. For a probing wake it is one statement: a CTE that reads `next_event_seq` from the snapshot, a CTE that locks the run row (`for no key update`, unchanged), and `exists (...)` for the replay. The wake is still registered after the replay check and before the append. - `lockNativeRunnerCheckpointOwner` returns the transaction id with its owner lock and primes the wake state. Its return value and every caller are unchanged. ## Verification - Red/green on embedded PostgreSQL; each new assertion failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "registers a native event's wakes before its write and only when the event is new": four statements (before: six), the wake's intent and settlement are published, and a replay publishes no wake. - New: "finds a replay that another writer committed while the lock read waited, and registers no wake for it". Another connection holds the run row (the test waits until PostgreSQL reports the append waiting for that lock), stores the same event and moves `next_event_seq`; the waiting append returns `duplicate`, runs the separate probe, and publishes no wake. With the `next_event_seq` comparison removed, this test fails. - New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the owner lock with the transaction id, the write, COMMIT (before: five statements), and its recovery wake is published; a save after the lease expired writes nothing and fails with `native_runner_checkpoint_owner_lost`. - `work-signals.test.ts`: a primed transaction sends no id statement for its wakes; a primed transaction whose connection is killed after its write stays pending and then resolves through `reconcileDatabaseWork`, with no row stored. - Added after merging `origin/master` (with #21511, #21520 and #21436) into this branch: - "gives the wake the transaction id of the transaction that took the run lock": inside a caller's transaction, an event with a progress wake sends no separate `pg_current_xact_id()` statement; the transaction's connection is then killed, and `reconcileDatabaseWork` checks `pg_xact_status` with exactly the id that `pg_current_xact_id()` returns on that same transaction. Priming a wrong id makes it fail. - "primes nothing when the lock read fails, so the transaction is not left uncertain": a lock read that fails on `lock_timeout` leaves no pending work; after the same failure inside a savepoint, the next signal on that transaction still sends its own `pg_current_xact_id()`. Priming before the lock read makes it fail. - Both call sites pass only the value returned by the statement they just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)` then `primeDatabaseWorkTransactionId(tx, run.xid)`) and `lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then `primeDatabaseWorkTransactionId(tx, xid)`). - Retest on the combined tree, one file at a time: atomic-append 28 (including the equivalence test and both race tests), the port suite with conformance 105, checkpoint-save 2, `live-recovery-work` 51, `work-signals` 25, runner `native-session-runtime` 297; server typecheck passes. No code change was needed for the combination. - Other focused suites pass, one file at a time: the port suite with its conformance suite (100), `chat-native-progress-work` (35), `native-observer-replay` (1), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-recovered-owner-followup` (97), `native-restart-recovery-isolation` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25). - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). Stacked: this pull request merged locally with #21511, and the model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model applies the port's batching rules itself; that mode is not in any pull request). Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. "Before" rows are `origin/master` from the same session. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 / 12,000 | 1 / 70 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 / 24,000 | 2 / 1,274 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 / 72,000 | 1,136 / 4,980 | | 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 | 71,958 / 72,000 | 1,319 / 5,712 | | 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 / 48,000 | 1,974 / 3,835 | With the three changes, one process finishes 99.9% of the offered work at 600 agents, but its event-loop utilization is 0.90–0.93, above the 0.75 target, and p99 lateness is about 5 s. It saturates between 600 and 800 agents. - Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5 where commit-time notifications are on). ## Risks - The transaction id is read in a statement that already assigns one (a row lock), so no transaction gets an id it did not have before. - The probe in the lock read is used only when `next_event_seq` did not change between the snapshot and the lock. Every writer of `heartbeat_run_events` moves `next_event_seq` in the same transaction (native and ordinary appends, the run coordinator store, the retry queue). A future writer that inserts events without it would make this probe miss a concurrent replay; the append statement then still finds the replay and stores nothing, but the wake would already be registered (a spurious wake, not a lost or duplicate event). - This pull request and #21511 both edit `heartbeat-run-events.ts`, in different functions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1f388ed385 |
Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm sandbox keeps an agent's Runner alive between runs, so the next run on the task starts fast > - When a run ends, the server closes the idle warm Runner in-process. A separate SQL worker stops the sandbox when its warm window ends (#21410, merged today) > - Both steps start at the same moment. The sandbox stop wins, so the idle close fails. The failed owner stays registered, and the next run throws `native_instruction_materialization_handoff_unavailable` before any provider work > - Recovery must just work: a follow-up run must never fail because a predecessor's warm copy is gone > - This pull request lets the idle close finish before the sandbox stops, prepares instructions fresh when the warm copy cannot be handed off, and releases an owner whose sandbox the host already gave up > - The benefit is that automatic follow-up runs start normally after a warm window ends ## Linked Issues or Issue Description Refs #21472 (base of this stacked PR), #21410 (introduced the warm-window stop). **What happened?** In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three automatic follow-up runs failed before any provider work with `native_instruction_materialization_handoff_unavailable`: run `5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a release swap. In each case, the predecessor's idle close had failed first with `provider_state_unreadable` / "Daytona sandbox lease … is no longer active". The cause: the warm-window stop from #21410 stopped the sandbox at the same moment that the idle close started. For the first run's predecessor, the close was requested at 20:30:34 and the sandbox got SIGTERM at 20:30:37.8. **What did you expect?** The follow-up run prepares its instructions fresh and continues. The warm-window stop does not interrupt the host's own idle close. **Steps to reproduce** 1. Run a warm Daytona task until it ends, so its warm window starts. 2. Let the warm window end while the idle timer closes the Runner. 3. Start a follow-up run on the same task. It fails with `native_instruction_materialization_handoff_unavailable`. Two of the three tasks then ended stranded. That downstream part (the claim-gate cancellation with `execution_reconciliation_required` and the stranded notice) is owned and fixed by another session. It is **not** in this PR. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts` `claimWarmNativeInstructionCopy`: after it retires a stale warm owner whose copy cannot be adopted, it returns `null`. The caller then prepares instructions fresh. Before, it threw `native_instruction_materialization_handoff_unavailable`. - `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window. The in-process idle close (close grace plus the 30 s proof read) finishes first. - `native-session-executor.ts` idle timer: when an idle close fails, it reads the lease rows for that sandbox. If every row is no longer `active` or `retained` (the host gave the sandbox up), it releases the in-process owner. The next run then prepares fresh. - `doc/execution-semantics.md`: one paragraph about the stop margin and fresh preparation. ## Verification Red → green (each test fails on the base `35ba76d0ef` and passes on this branch): - `native-session-executor.test.ts` › "hands off collection before probing or retiring a warm owner: unavailable" — red: throws `handoff_unavailable`; green: returns `null` (fresh preparation). - `native-session-executor.test.ts` › "forgets a warm owner whose idle close failed only once the host gave up its sandbox: pending_cleanup / released / retained" — red on all three; green: the owner is released for `pending_cleanup` and `released`, and kept (retired by the successor) for `retained`. - `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the warm runner time to finish before stopping its sandbox" — red: the sandbox stops 3 s after the window; green: it stops only after the margin. Two existing tests ("keeps a sandbox still inside its warm window, and wakes when the window ends", "reconciles a warm sandbox left running across a restart") now expect the deadline with the margin. Local runs on this branch: - `server`: `tsc --noEmit` clean. - `vitest run src/__tests__/warm-sandbox-expiry.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed. - `vitest run src/services/native-runtime/native-session-executor.test.ts -t "hands off collection before probing or retiring a warm owner|forgets a warm owner whose idle close failed"`: 10/10 passed. Not run: a Daytona wave with this change. ## Risks - Warm sandboxes now stop up to 2 minutes later than the configured window. This is a small cost increase per warm sandbox. I chose a fixed margin over an in-process "is a close running?" check because the stop is pure SQL. The margin works when two servers run at the same time (the server that runs the stop may not be the server that owns the Runner), and it needs no busy-loop or cross-server signal. - The margin does not cover a re-armed idle timer or a close that takes longer than 2 minutes. Fresh preparation (fix 1) and the owner release (fix 3) cover those cases. - Not run on Daytona. The evidence is from the wave above and the unit tests. - In the wave, a `restore_failed` copy-back onto the just-restarted sandbox was also seen. I did not find its root cause. It is not changed here. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a82fa96540 |
perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane port commits streamed text deltas of a run in batches (#21460): the first text at once, later deltas after at most 250 ms > - Each batch is cut when its timer fires, even when the previous batch of the same run has not committed yet > - When the database is slow, every run then queues one small batch per 250 ms behind its slow commit. The number of statements stays the same exactly when the database has the least room for them, and the queue grows > - This pull request cuts a batch only when the previous batch of the run has committed. Deltas that arrive while a batch commits all join the next batch > - The benefit is fewer, larger statements under load, with no change when the database keeps up, and no longer wait for any delta ## Linked Issues or Issue Description Refs #21460 (batched deltas). Refs #21511 (one statement per delta batch). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** How `PaperclipControlPlanePort` groups queued text deltas into batches when a batch commit is slower than the batch time. **Subsystem affected** Server run event ingest: `server/src/services/native-runtime/paperclip-control-plane-port.ts`. **Current behavior** `#flushStreamingEvents` takes the queued deltas when it runs (on the 250 ms timer, at 100 deltas, or before any other event) and chains their commit behind the previous batch. If one commit takes two seconds, the run queues about eight batches, each one statement, behind it. **Proposed behavior** `#flushStreamingEvents` chains one cut behind the previous batch and takes the queued deltas when that cut runs. While a cut waits, later flush requests join it. A run has at most one batch committing and one cut waiting. **Reason and benefit** Under load, the number of delta statements per run falls to about one per commit time instead of one per 250 ms. When commits are faster than 250 ms, nothing changes. **Breaking changes** None. Rows, receipts, order and the first-text rule are the same. ## What Changed - `#flushStreamingEvents` queues one cut on the run's commit chain (`#streamingFlushQueued`). The cut takes every delta queued at that time, sets the first-text flag, commits the batch, and then releases the pending count of that batch. - A flush while a cut waits returns the same chain, so a caller that waits for it (every other event, `checkpointSession`, `completeRun`, `replayEvents`, `provesNoRunProviderWork`) still waits for every delta queued before it. - A batch can now hold more than 100 deltas when commits are slow. The 100-delta rule still asks for a flush; the batch is one statement with a fixed number of bind parameters (#21460, #21511). ## Verification - Red/green: new port test "collects the deltas that arrive while a batch commits into one next batch, before the next other event" (`paperclip-control-plane-port.test.ts`). A transaction holds the run row, so the first batch waits. Six more deltas arrive 15 ms apart with a 5 ms batch time, then a non-streaming event. Before: 8 inserts (one per timer). After: 3 inserts (the waiting batch, one batch of six, the other event). Rows are stored 1–9 in order and every receipt resolves. - The whole port suite with its conformance suite passes: 101 tests. Other suites, one file at a time: `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `chat-native-progress-work` (35). I ran the four streaming port tests two more times; they pass each time. - `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). The model applies the port's batching rules itself, so for these runs I changed my local copy only (not in this pull request): with `DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append chain and takes the queued deltas when the cut runs, as the port now does. All rows run with #21511. Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. | Agents | Batching | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 / 12,000 | 3 / 3,994 | | 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 / 24,000 | 5,721 / 15,238 | | 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993 / 24,000 | 1,896 / 4,193 | | 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 / 36,000 | 2,280 / 30,985 | | 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 / 6,057 | 58,984 / 72,000 | 11,465 / 25,361 | | 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 | 24,877 / 48,000 | 24,544 / 31,331 | The process does more work per second with this change (more other events finish), so statements per second rise; delta lateness falls. 600 agents are still not carried reliably: other events fall behind. #21528 removes two statements from each such event and one from each checkpoint save; its description has the stacked results. ## Risks - A delta never waits longer than before: it already waited for the previous batch of its run. When the database keeps up, batches are the same as before. - Under load, one batch can be large. The live view then gets one larger step instead of several small steps behind the same wait. - A failed batch now rejects more deltas at once (every delta that joined it). The runner resends them, as before; each resend is idempotent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0bba04675c |
perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - After #21460, text deltas commit in batches, but each batch still takes five statements: BEGIN, the run lock, a replay probe, one insert, and COMMIT. A single event without a wake takes four > - In the agent scaling load model, the process issues about 36 statements per agent per second and saturates its one thread between 200 and 400 agents > - This pull request writes a batch of in-order native events, or one such event, with one statement that commits on its own > - The benefit is about half the statements per agent, less CPU per agent on the main thread, and less time on a pool connection, with the same rows, receipts, cursor and locks ## Linked Issues or Issue Description Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The append of native source events that continue their source's stored cursor: batched text deltas, the first delta after another event, and every other native event that registers no wake. **Subsystem affected** Server run event ingest: `server/src/services/heartbeat-run-events.ts`, and the port comment in `paperclip-control-plane-port.ts`. **Current behavior** A delta batch takes BEGIN, a run lock read (`for no key update`), a replay probe, one insert, and COMMIT: five statements. One delta that commits at once takes four. In the load model at 200 agents the process sends about 6,700 statements per second at event-loop utilization 0.72. **Proposed behavior** The same append is one statement. Its update of the run row takes the same row lock. It requires the stored cursor to end just before the batch, requires the `next_event_seq` that its own snapshot read, and finds no replayed event. Then it moves the sequence and the cursor and inserts every row. If any check fails, it writes nothing and the append takes the existing locked transaction. **Reason and benefit** Fewer statements free the main thread and the shared pool for other agents. In the load model at 200 agents, statements per second fall from about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to 0.56. **Breaking changes** None. No schema change. Rows, receipts and the cursor are the same. Servers from before this change keep working on the same runs. ## What Changed - `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one statement: a CTE reads `next_event_seq` from the statement snapshot, an `update heartbeat_runs ... returning` locks the run row and moves `next_event_seq` and the source cursor, and an `insert ... select from jsonb_array_elements($9)` writes every row. The text is fixed, with nine bind parameters for any batch size. - **Why the `next_event_seq` check.** A statement that waits for the run row lock keeps the snapshot it took before the wait. Its replay probe cannot see events that the lock holder committed. Every event append (native, ordinary, the run coordinator store, the retry queue) moves `next_event_seq` in its own transaction. So the update requires the value that its snapshot read. PostgreSQL re-checks that condition on the newest row version after the wait, and any append committed meanwhile makes the statement write nothing. The unique indexes on `(run_id, source_event_id)` and `(run_id, source_instance_id, source_seq)` are a second guard: a unique violation from this statement also falls back. - A source without a cursor starts at 0. The probe still finds any stored event of that source, so a run that an older server wrote falls back and recounts. - `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the one statement first for native events without a wake, outside a caller's transaction. Every other case (a replay, a gap, an event with a wake, a cursor that lags, a run changed meanwhile, a binding mismatch, a run deleted meanwhile, a caller's transaction) takes the existing transaction, unchanged. - Inside a caller's transaction the append keeps its SAVEPOINT path. A failed statement there would abort the caller's transaction, and the fallback could not run. - `assertWritable` runs before the statement is sent and after it returns (see Risks). ## Verification - Red/green on embedded PostgreSQL. I ran the new tests against the previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests in the file pass with the change (`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`): - One statement for an in-order event and for a batch (before: 4 and 5). - A writer revoked before the send sends nothing; a statement that fails on its third row stores no row and no sequence; the resend commits. - A writer revoked while the statement runs: the rows stay, the receipt is rejected, and the resend is a replay with the same rows. - A batch waits for Stop on the run row and appends after it; a batch and a single event that wait for a run deletion fail with `heartbeat_run_event_binding_mismatch` and store nothing. - An older server stores events 2 and 3 while it holds the run row, without the cursor: the waiting batch 2–4 falls back, returns duplicate, duplicate, committed, and the cursor ends at 4. - Two connections append the same 200-event stream in batches of 10: each event is stored once, in source order, sequences 1–200 without a gap, one `committed` and one `duplicate` receipt per event. - A 7,000-event batch (more than any bind parameter limit at 10 parameters per row) is one statement. Before: it fell back to one statement per event and timed out at 15 s. - A run without a cursor starts at its first event; a run with events from an older server replays and then continues. - Equivalence: 12 random streams of 30 steps (in-order batches, gaps, replays of earlier ranges, fills of gaps, ordinary events, conflicting replays). Each step goes to two runs, one through the new path and one forced through the one-by-one locked path. Receipts, errors, rows, `next_event_seq` and cursors are equal at every step. - Mutation check: with the `next_event_seq` check and the unique-violation fallback removed, the older-server race test fails. - The port suite (`paperclip-control-plane-port.test.ts`, with the package conformance suite) passes: 125 tests. Two tests changed: the streamed-event test now expects one statement and the new revocation behavior, and the batch test checks that the five deltas are one statement sent before the boundary event. - Other focused suites pass, one file at a time: `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `chat-native-progress-work` (19), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51). In 2 of 3 runs with this change, all 51 tests passed but the file's `afterAll` (closing the side pools and the embedded database) hit its 30 s timeout on this busy machine; the third run passed cleanly, and one run on `origin/master` passed. The native events in that file all register a wake, so they do not take the new path. - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in #21418 (per-source counts stop when the window closes). I did not change its batching rules for this pull request. Model rates: 20 deltas/s, 1 other event/s, 0.3 API requests/s per agent; default pool of 10 connections; embedded PostgreSQL on the same, shared machine. I ran "before" (`origin/master`) and "after" one after the other, with a pause between runs, and I give the load average at the start of each run. The machine was busy with other work (load 10–15 on 18 cores), so compare rows that ran close together. No run had a failed operation, a failed checkpoint save or a lost lease. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 / 24,000 | 5,721 / 15,238 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 / 36,000 | 2,280 / 30,985 | | 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135 / 52,222 | 36,000 / 36,000 | 698 / 3,422 | At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization and 7,051 → 3,210 statements/s. One process with this change alone still does not carry 600 agents: at 400 and above the 10-connection pool and the shared database are the limit (event-loop utilization stays below 0.75 while other events fall behind). #21520 and a third pull request for the remaining statements of other events and checkpoint saves continue from here; their description has the stacked results. - Statements: a delta batch 5 → 1; one delta or one native event without a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save (5). ## Risks - **`assertWritable` window (read first).** Before, the writer check ran inside the transaction, before the first statement and before COMMIT. A revocation while the statements ran rolled the append back. Now a revocation while the one statement runs does not roll it back: the rows commit, and the caller gets the error instead of the receipt, so no observer runs and the runner gets no ack. This check guards a controller that hands its run to a successor at restart (`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the earlier check was not atomic with COMMIT either, so a revocation during the COMMIT round trip already committed; the window grows from one COMMIT round trip to one statement round trip plus its wait for a pool connection; (2) the append is idempotent by source event id and payload hash, so the successor sees the committed row as a replay of the event that the runner resends; (3) the old controller never acts on the rejected receipt. A new test covers exactly this. - A crash loses nothing new: the statement commits or not as a whole, and a resend after an unknown outcome is a replay. - The check on `next_event_seq` makes the statement write nothing whenever any other event of the run committed after its snapshot. Under heavy contention on one run, more appends take the fallback. The fallback is the previous code path. - The statement text is fixed SQL sent through the driver (`$client.unsafe`), like the rest of the event stream statements. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2178c3df88 |
fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at |
||
|
|
47a2c82d35 |
fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at |
||
|
|
089748fbf9 |
fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.
## Linked Issues or Issue Description
**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.
**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.
**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.
**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
guidance: "Reconcile the existing operation before repeating it. ..." }
```
No related public PR exists. I searched open and closed PRs for this
fence.
## What Changed
- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
- A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.
## Verification
- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.
## Risks
- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
|
e82c4f0a31 |
fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**
This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it
## Linked Issues or Issue Description
**What happened?**
Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.
- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.
**Expected behavior**
A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.
**Steps to reproduce**
1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.
**Paperclip version or commit**
`master` at
|
||
|
|
6d8ebc4a85 |
fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can bind a company secret to an environment variable for an agent, an environment, a project, or a routine > - Core resolves these values for each run and gives them to the agent process or to the Runner > - Core stores the run output: the run log, the run excerpts, adapter events, and native (Runner protocol) transcript events > - The run-output redactor did not know the resolved secret values. Only failure diagnostics and the `meta.env` keys of `adapter.invoke` were masked > - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the secret in plaintext, and every person who can view the run could read it > - This pull request gives the resolved values of secret-bound keys to the run-output redactor and to the native control-plane port > - The benefit is that an accidental print of a secret does not put the secret into stored run output, for local and native runs ## Linked Issues or Issue Description **What happened?** An agent printed a secret-bound environment variable during a run (for example with `env` or `echo $STRIPE_KEY`). Core stored the value in plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in the transcript events. Anyone who can view the run can read the value. **Expected behavior** Core masks the value of each secret-bound environment variable before it stores run output. Core does not mask plain (non-secret) configured values. **Steps to reproduce** 1. Create a company secret and bind it to an agent environment variable with `secret_ref`. 2. Configure the agent so that its command prints the variable. 3. Run the agent and open the run log. The value shows in plaintext. **Related** Refs #21330. That PR keeps configured secrets out of OpenCode's own storage and model requests inside the Runner. This PR is the Core side: it masks secrets in the run output that Core stores, for every run type. The two changes are independent. ## What Changed - `secretEnvironmentValues(env, secretKeys)` in `agent-identity-redaction.ts` selects the raw values of keys in `secretKeys` (the keys that came from `secret_ref` or `user_secret_ref` bindings). `secretOutputForms` derives the forms to mask: the full value, the value without outer whitespace, the JSON-escaped form, and each line of a multi-line value (PEM armor lines are not added). It skips a form that has fewer than 8 characters after trimming. This is the same 8-character limit that diagnostic redaction uses for unknown values. - The output redactor keeps these secrets in a separate set, beside the existing agent private-key values. The private-key values are masked everywhere, as before. Environment secrets are masked only in output text. They are not masked in protocol fields: `schema`, `type`, `kind`, `status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids, refs, codes, revisions, versions, digests, timestamps, and names that end in these words (for example `eventType`, `turnTerminalState`, `reportedWorkDisposition`). So a secret that equals a protocol word (for example `completed`) cannot change an event type or a terminal state. - The streaming match now scans left to right. At each position it tries the longest value first. If a longer value could still start at that position, it holds the text until the next chunk, also when a shorter value already matches there. So the tail of a longer secret cannot leak when a shorter secret is its prefix. - `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to the run's output redactor immediately after it resolves them. This one redactor already masks the local adapter log stream, the excerpts, live log events, adapter events, `adapter.invoke` metadata, the adapter result, and the canonical native log export. - For a native run, `executeHeartbeatRuntime` stores the delivered secrets (only values that have a form to mask) encrypted under a new context key, `paperclipOutputSecretRedactions`, in one transaction. It then adds all stored values of the run to the redactor. A run that reconnects to a live Runner after a `"latest"` secret rotation then masks the value that the Runner holds and the new value. This store reuses the encryption and fingerprint logic of the run redaction registry, but it is separate: API read redaction does not apply its values, so a secret cannot change ids, event names or terminal states in API responses. API reads hide the new key like the existing registry key. - `executeHeartbeatRuntime` gives the native executor the run's secrets and the observability credentials (`observabilitySecretValues`). - The native executor gives the secrets to its failure-diagnostic redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks them in each PRP event before it persists the event, with the existing delta-aware handling for values that are split across `item.delta` events. This covers native runs on the local Runner and on plugin-hosted agent computers. - After a controller restart, a new control-plane port cannot know the text that the old port held back, because that text is never stored. On the first delta of each stream it has not seen, the port masks leading text that matches the tail of a secret (4 or more characters). It skips text that can start a secret, and it keeps checking the stream while a delta falls entirely inside such a tail. - The native executor classifies failures from the original error. It masks only the agent private key in place, as before. It masks environment secrets in a separate redactor that it uses only for text that it writes or logs (`failureDetail`, failure messages, stderr diagnostics). Core masks the rethrown error before it stores it. - `coalesceIntoActiveExecutionRun` (wake queue) writes the merged context with `withRunRedactionHistory`, which takes `paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from the locked current row. Before, a coalesced wake built from an older context could delete either key. This also protects the existing registry. This is a change outside the secrets module, in the one place that replaces a running run's context concurrently. - When an adapter or native run fails, the heartbeat keeps the original error for retry decisions and rethrows it. The copies that it saves in `workspace_finalize` metadata (`errorMessage`) and in a failed session goal action go through the run's output redactor first. - `docs/deploy/secrets.md` describes the mask and its limits. ## Verification - `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new, embedded Postgres): a real `process` adapter run with a `secret_ref` bound to `"latest"`. The agent prints the secret split across two writes, in JSON, and on stderr. The stored log, run row, and run events do not contain the value. The plain value and a 5-character secret are not masked. The test then rotates the secret and runs again. The second run masks the rotated value, and a SHA-256 digest in the output proves the run received the rotated value. Red-green: it fails without the new line in `execution-configuration.ts`. - `server/src/services/native-runtime/paperclip-control-plane-port.test.ts`: the port masks a secret in a persisted tool event and in a value split across two `item.delta` events; plain and short values stay (red-green: fails without the port change). A new test binds secrets equal to `interrupted`, `completed`, `succeeded` and `satisfied`, then appends a `turn.interrupted` event, checkpoints, and completes the run. The event type, payload status, terminal states and criterion status stay unchanged and the result is saved (red-green: fails when protocol fields are masked). - `server/src/services/heartbeat/runtime-execution.test.ts`: a value stored by an earlier attempt, the current value, and an observability header value all reach native dispatch and the heartbeat redactor. Short values are not stored. The shared run redaction registry stays empty, and an API read hides the new key (red-green: fails without the store read). - `server/src/services/native-runtime/native-session-executor.test.ts`: a native failure that contains the secret is masked in failure writes, logs and the rethrown error, and the executor gives the secrets to the control-plane port. - `server/src/services/agent-identity-redaction.test.ts` (new): value selection, the minimum length, multi-line and JSON-escaped forms, nested values, every split point of a streamed value, overlapping secrets (`abcdefgh` and `abcdefgh1234567890`) split at the shorter value, protocol fields in events, terminal states and deltas, and secrets added by a later attempt. - Restart, retry and coalescing (each fails without its fix): a port test restarts between two deltas of a secret and stores `echo ` and `***REDACTED*** done`; an executor test binds `timed out` and keeps `provider_session_open_timeout`; a wake-queue adapter test keeps both redaction keys when a coalesced wake replaces the context. - `server/src/services/heartbeat/run-output.test.ts`: the local output path masks the log, excerpt, live event, adapter event, and adapter metadata. - `pnpm --filter @paperclipai/server typecheck` passes. The changed test files and the other users of the redactor (`agent-identity`, `run-log`, `heartbeat-case-handoff`, `native-run-log-export.integration`) pass in full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify` exits 2 (broader verification required) because `docs/deploy/secrets.md` is a global CI input, so required CI supplies broad coverage. ## Risks - This is a guard against accidental leaks. It is not a security boundary. An agent can change a value before it prints it (for example, encode it in base64 or print one character at a time), and Core cannot detect that. - Core does not mask a secret value or line that has fewer than 8 characters. This prevents a short value from erasing ordinary output. A secret that short is not masked in run output. - Core does not mask environment secrets in protocol fields (see What Changed). If a secret appears in such a field, it stays. These fields hold protocol words and generated identifiers, not agent output. - If a value is common text of 8 or more characters, Core masks that text wherever it occurs in that run's output text. This only affects runs that bind that secret. - If a key is secret-bound in one source and a later source sets the same key to a plain value, `secretKeys` still includes the key, so Core masks the plain value. This is the existing meaning of `secretKeys`. - For native runs, Core now stores the delivered secrets encrypted in `contextSnapshot` under `paperclipOutputSecretRedactions`, with the same local encryption as the existing run redaction registry. It adds one transaction per native run that has secrets. API reads do not apply these values and hide the key. - After a controller restart in the middle of a secret, a tail shorter than 4 characters can still be stored. A new stream that starts with the tail of a secret (4 or more characters) is masked at its start even without a restart. This is a small amount of extra masking. - Core masks values at write time. Run output that Core stored before this change is not changed. - No schema change and no migration. The new fields are optional and in-process only. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
defed64af9 |
fix(native-runtime): retry a native failure write after a brief lock timeout (#21494)
An interrupted native turn records its same-run resume in one transaction under a 1 s lock_timeout. When another transaction briefly held a row that the write needs (for example the agent row that the new recovery action references), the insert failed with 55P03 and the whole write rolled back. The run kept its dead lease, and the task needed reconciliation instead of the recorded automatic resume. Retry the whole failure write on transient lock contention with the existing bounded helper. A rolled-back attempt has no effect, so a fresh attempt is safe. The regression test holds the agent row until one attempt is cancelled by its lock timeout, so only a retry can complete the write. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5df1ada9c1 |
feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21335 lets a plugin task provider that restarts the Runner on `resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant run's harness credential > - A provider restarting the Runner also needs the agent's configured environment, which Core sends on `resume` only during a session handoff > - Without it, a Runner a provider moves to a new computer starts without the agent's configured environment variables > - This pull request widens that opt-in to both launch inputs, under a name that says so, before any provider adopts the old name > - The benefit is that a provider can restart a parked Runner on resume with the same credential and environment a `submit` would carry ## Linked Issues or Issue Description Refs #21289 Follows: #21335 **What existing behavior does this improve?** The opt-in resume launch inputs added in #21335. **Subsystem affected** Plugin SDK environment task schema (`describe` capability), the plugin Runner task owner's `resume`, and the plugin environment task docs. **Current behavior** With `resumeCredentialVersion: 1`, `resume` carries `credential`. `configuredEnvironment` is sent on `resume` only for a session handoff. **Proposed behavior** The capability is renamed `resumeLaunchInputsVersion: 1`. With it, `resume` carries `credential` as before and also `configuredEnvironment` (same builder as `submit`) when the provider advertises `configuredEnvironmentVersion: 1`. Session handoff is unchanged. **Reason and benefit** A provider that restarts the Runner on resume gets both launch inputs it needs. **Breaking changes** The capability name changes from `resumeCredentialVersion` to `resumeLaunchInputsVersion`. No provider advertises the old name yet (it merged in #21335 and has no adopter), so nothing changes behavior. ## What Changed - Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the SDK `describe` result schema, executor plumbing, owner input, docs and tests. - With the capability, `resume` also adds `configuredEnvironment` when the provider advertises `configuredEnvironmentVersion: 1`. A normal resume does not mark the variables as delivered, since a live Runner keeps the environment it started with. - Docs: the "Release and resume" section describes both fields; the "Configured environment" section points to it. - The runner-e2e SDK public-input pin is updated. ## Verification - SDK vitest `tests/environment-task-release-resume.test.ts` and `tests/environment-tasks.test.ts`: 68 passed. - Server vitest `plugin-runner-task.test.ts` and `environment-task-runtime.test.ts`: 153 passed, including a normal resume with the capability (credential sent, variables added only with `configuredEnvironmentVersion: 1`, delivered flag unset) and redaction of both fields from metadata writes and sanitized errors. - `native-session-executor.test.ts -t "uses a task-only"`: 5 passed. - runner-e2e `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts`: 46 passed. - `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed. - Not run locally: the full suite and `pnpm verify`; required CI covers them. ## Risks - A provider restarting the Runner gets the claimant run's current variables, which can differ from the ones the parked Runner started with. That is intended. - Rollback: revert this pull request. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs), implemented by a sub-agent and reviewed before opening. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4cf04d64cb |
fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts a Runner in a remote sandbox for each agent run. A bounded same-run retry (at most 3 startup attempts) handles transient startup failures. > - A server restart can stop a run during its first Runner bootstrap. Restart recovery then claims `bootstrap_incomplete`. It stops and quarantines the unfinished bootstrap and launches a fresh bootstrap on the same run. > - The executor builds its "unclaimed bootstrap" proof only when `restartRecovery` is not set. Thus the fresh bootstrap of a `bootstrap_incomplete` restart can never prove that a transient failure is safe to retry. > - A transient HTTP 502 from the Runner endpoint request in that fresh bootstrap was then classified as `ambiguous_state`. The run stopped and needed manual board recovery. > - This pull request gives the unclaimed-bootstrap proof to a `bootstrap_incomplete` restart as well as to a first attempt. > - The benefit is that a transient endpoint failure after a restart uses the bounded automatic retry and does not need a person to recover the run. ## Linked Issues or Issue Description Refs #20414 (related: it fixes executor admission of a `bootstrap_incomplete` restart at state-root migration. It does not change the retry evidence that this PR changes.) **What happened?** A server restart interrupted the first Runner bootstrap of a remote native run. Restart recovery claimed `bootstrap_incomplete` and launched a fresh bootstrap on the same run. The fresh bootstrap registered its controller checkpoint. Then `runner.ingress.acquire` failed with HTTP 502, before any Runner connection or provider event. The run ended with `nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3, and the next action "resolve the ambiguous session state". The error was `Runner connection endpoint could not be acquired (HTTP 502).` (`runner_ingress_acquisition_failed`). **What did you expect to happen?** The transient 502 occurred before the Runner connected and before any provider work. It should use the bounded same-run startup retry, as it does on a first attempt. **Steps to reproduce** 1. Start a native run on a remote sandbox target. 2. Restart the server during the first Runner bootstrap, so that restart recovery claims `bootstrap_incomplete`. 3. Make the next `runner.ingress.acquire` call return HTTP 502. 4. See that the run stops as ambiguous state and does not schedule a startup retry. ## What Changed - Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`. It returns true for a first attempt and for a `bootstrap_incomplete` restart. It returns false for `reattach_existing_runner`, `reattach_remote_runner` and `resume_dead_runner`, because these continue an existing Runner or provider session. - In `native-session-executor.ts`, use this helper in place of `!input.restartRecovery` when the executor builds the unclaimed-bootstrap evidence for a startup failure. - Add `native-bootstrap-proof.test.ts` for the helper. - Add a case to `native-startup-retry.integration.test.ts`. It runs the real executor under a `bootstrap_incomplete` restart claim, and the fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before the Runner connects. ## Verification - `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart of an incomplete bootstrap" failed before the fix and passes after the fix (5/5 tests pass, vitest 5.1.0). - `native-startup-retry.integration.test.ts`, case "transient 502 after a bootstrap_incomplete restart": the run must settle as `retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1 and a set `nextAttemptAt`. Without the executor change, the executor builds no bootstrap evidence and the run settles as `ambiguous_state`. CI runs this suite; I could not run it locally. - The other guards for a retry did not change. The checkpoint must be unclaimed, there must be no provider events and no session checkpoint, and the 3-attempt startup budget applies. If the earlier attempt opened a provider session, provider events exist and the run still fails closed as ambiguous. - I could not run the full typecheck and the real-Runner integration suites locally (no Rust toolchain in the sandbox). CI must run them. ## Risks - Low risk. The change only affects a `bootstrap_incomplete` restart whose fresh bootstrap fails during startup. Such a failure now gets the same retry decision as a first attempt. All the other evidence checks still apply, so a run with provider activity still stops for manual recovery. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with tool use (shell, file edit, and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
342fb308bd |
perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - Text deltas (`item.delta`) are about 85% of that process's SQL statements: each delta is one transaction of four statements on the shared 10-connection pool > - At 50 agents and 20 deltas per second, the process falls behind and unrelated agent requests wait for the pool > - This pull request commits text deltas in batches: one transaction for all deltas that arrive within 250 ms, with one multi-row insert > - The benefit is about 7 times fewer transactions and 5–6 times fewer statements per delta, with no delay for the first text of an item ## Linked Issues or Issue Description Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a of the agent scaling plan in #21418. **What existing behavior does this improve?** The persistence of native text deltas. `native-session-runtime` awaited `PaperclipControlPlanePort.appendEvent` for each event, and the port ran one transaction per event. **Subsystem affected** Server run event ingest (`paperclip-control-plane-port.ts`, `heartbeat-run-events.ts`) and the runner package's turn consumer (`native-session-runtime.ts`). **Current behavior** Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In the load model with 50 agents (deltas only), that is about 4.0 statements and 1.0 transaction per delta. **Proposed behavior** The turn consumer starts the next append without waiting for a delta's receipt. The port collects deltas and commits them in one transaction with one insert. In the same load model, that is about 0.7 statements and 0.14 transactions per delta. **Reason and benefit** Fewer transactions free the shared pool and the main thread for other agents' requests. This is the largest per-agent cost in the scaling plan. **Breaking changes** None. The row shape does not change. A port without the new capability keeps the earlier serial behavior. ## What Changed - `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends native source events of one run and one source in one transaction. When the events continue the stored source cursor (from #21435), it sends four statements for the whole batch: the run lock, one replay probe, one `insert ... select from jsonb_array_elements(...)` of every row, and the commit. Any other batch (a replay, a gap, an event with a wake, or a run without a cursor) appends its events one by one in the same transaction, with the single-append checks. - **Storage choice: one row per delta, written by one statement per batch.** I did not store one row per batch. The transcript, live events, run log export, chat progress, replay (`replayEvents`), recovery cursors and the UI all read one row per delta. One statement per batch keeps every reader unchanged, and rows written before this change read the same. - `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An `item.delta` appended with the new `streaming` option goes into a queue: - It commits at once until a delta with text commits after the latest other event. So the first text of each item, and the time to first token, never wait for the timer. - Later deltas commit after at most 250 ms, or at once when the batch has 100 deltas. - Every other event, `checkpointSession`, `completeRun`, `replayEvents` and `provesNoRunProviderWork` first commit the queue. A non-streaming event is never committed before the deltas that preceded it. - A delta appended without `streaming` commits at once, so callers that wait for each receipt are not slowed. - The receipt of a delta settles only after its batch commits and its observers (live event, executor callback) run, in order. - If a batch fails, every delta in it rejects and nothing from it is stored. - `native-session-runtime` `consumeTurn` appends deltas with `streaming` and does not wait for their receipts (at most 256 pending). Before any other event, any result, and the governed-wait settlement, it calls `flushStreamingEvents()` and then waits for all pending receipts, so a boundary never waits for the timer. A failed delta stops the turn before a later event is appended. The consumer joins every pending delta append before it settles, also when the provider stream fails. - If one observer of a committed batch fails, only that delta's receipt rejects. The other rows are still published and observed. - `ControlPlanePort` contract: optional `pipelinesStreamingEvents`, `flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`. ## Verification - Red/green. The new tests failed before the change and pass after it: - Runner: "starts text delta appends without waiting and flushes them before any other event" (`native-session-runtime.test.ts`). The port in the test has no timer, so the turn finishes only if the boundary flushes. Before: the appends were serial, and later the boundary waited for the timer. - Runner: "joins pending delta appends before a failed turn settles". Before: the turn settled while a delta commit still ran. - Port: "observes every committed delta of a batch when one observer fails". - Port: "commits the first text delta at once and later deltas in one batch before the next other event". It checks one transaction and one insert for five deltas, flush before a non-streaming event, receipts only after commit, and immediate commit of the first delta with text. Before: each delta committed alone. - Port: "flushes queued deltas on the timer and before a replay read, and rejects every delta of a failed batch". It checks the crash-before-commit case: a revoked writer fails the batch, nothing is stored, and the resent deltas commit idempotently (the stored one is a duplicate). - Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`: one lock, one probe and one insert per batch; replay idempotency and payload conflict; a batch that fails before commit stores nothing and its resend commits; the one-by-one path for no cursor and for gaps. - Focused suites pass: `heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`, `paperclip-control-plane-port` (including the package conformance suite), `paperclip-semantic-conformance`, `native-observer-replay`, `native-event-boundary-golden`, `live-recovery-work`, `native-interaction-bridge`, `chat-native-progress-work`, and `native-session-runtime.test.ts` in the runner package. - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript` and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418, `AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls the store directly, so for this run I added a local `BATCH_MS` mode that applies the port's batching rules (first delta at once, 250 ms timer, 100 cap, flush before other events) and does not wait for delta receipts. That mode is not in this pull request. The machine was very busy (load average 40–60), so absolute figures are noisy; per-delta ratios are the stable result: | Deltas only | Statements/s | Transactions/s | Deltas done of offered | Statements per delta | Transactions per delta | Lateness p50 / p99 | | --- | --- | --- | --- | --- | --- | --- | | One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 | 1.00 | 3,074 / 5,384 ms | | Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 / 2,858 ms | | One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 | 3.99 | 1.00 | 6,558 / 11,464 ms | | Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17 | 187 / 556 ms | With all sources on, the machine was saturated (event-loop utilization 0.96–0.98) in both modes and the results were not usable (764 and 12,800 of 20,000 deltas done). ## Risks - **Ack and crash (read first).** In the live runnerd path the runner's ack does not wait for the `heartbeat_run_events` row today. `DurablePrpControlPlane.#event` acks after the in-memory journal commit. Streaming events are not in the controller checkpoint. So a delta that is acked but not yet appended is lost if the server crashes, and the runner does not resend it. This is the earlier behavior. This pull request makes that window up to 250 ms longer. The final text still survives through `item.completed`, which commits immediately and in order. This pull request does not change `durable-prp-control-plane.ts`. - Live text in the UI arrives in steps of up to 250 ms after the first text of an item. - A delta failure now surfaces at the next non-delta event or at the end of the turn, not at the delta. The turn stops before it appends a later event. - The governed-wait drain and the control-event replay still append one event at a time. They do not carry deltas. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12aa08721c |
feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.
Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on
|
||
|
|
b5adf35119 |
test(runner): guard that a relaunch after a lost launch acknowledgement leaves one Runner (#21477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) that the server launches in a sandbox. > - A deploy can stop the server after the launch command reached the sandbox but before the server read back the Runner's process identity (a lost acknowledgement). > - The database then has no PID for the run, so the successor server launches the Runner again on the same state folder. > - Two Runners must never serve the same run's state. #20852 proposed a large launch-intent design for this window. Since then, #21079 added a fence that stops a leftover Runner on the same state folder before each launch. > - This pull request adds a Linux regression test that drives the real launcher through the lost-acknowledgement window. > - The benefit is that a future change to the launch path cannot bring back a second Runner without failing this test. ## Linked Issues or Issue Description Refs #20852 (superseded: current master already prevents the duplicate Runner through #21079). Refs #21079. **What happened?** #20852 claimed that a relaunch after a lost launch acknowledgement could start a second Runner for the same run. Nothing proved whether current master still has this problem. **Expected behavior** After a lost acknowledgement, the successor's relaunch stops the orphaned Runner. Exactly one Runner serves the run's state folder, and the process marker names it. **Steps to reproduce** 1. Launch a remote Runner, and lose the launch RPC's acknowledgement before the identity read (the old server stops). 2. Relaunch the same run on the same sandbox from a new launcher (the successor). 3. Count the live `paperclip-runnerd` processes that serve the run's `--state-dir`. ## What Changed - New `server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`. It is a test only and changes no product code. - It runs only where `/proc` exists (Linux CI), like the existing #21079 fence tests. - It drives the exported `createRemoteRunnerProcessLauncher` with a local command runner. The old launcher's runner runs the launch command, then throws `remote_launch_ack_lost`. After that, every command fails, because the old server is gone. - A small bash stand-in for `paperclip-runnerd` keeps the Runner's argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty lineage. - The test checks these results: the orphan was alive and had no PID on the handle; the successor's Runner has a new PID; the orphan stops; exactly one Runner serves the state folder; the marker names the successor's PID. ## Verification - Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I extracted from this head: - master scripts: `runners after successor relaunch [61], orphan alive false`. Result: one Runner (PASS). - the same scripts without the #21079 stale-state fence (counterfactual): `runners after successor relaunch [24, 56], orphan alive true`. Result: duplicate Runner (FAIL). - macOS: `npx vitest run src/services/native-runtime/remote-runner-lost-launch-ack.test.ts` reports the file as skipped (1 skipped), as intended, because there is no `/proc`. - The test runs for real only on Linux CI. CI on this head is the first real run of the vitest file. ## Risks - Low risk. The change is a test only. - The test starts real local processes. Its teardown kills only the PIDs it found on its own temporary state folder, and it never signals PID 0. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, Docker, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
11f4a76743 |
perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every native agent session. Each session gets the app tools (MCP tools) that are assigned to its agent. > - Each live or warm session kept two private copies of its assigned tool catalog: the gateway descriptors and a canonical copy of every input schema. > - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap for each session. With 800 tools, it was about 15 MB. Warm sessions stay for 5 minutes after the last turn, so idle agents also hold this memory. > - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the server cannot get there. > - This pull request keeps one deeply frozen catalog for all sessions of the same company that get the same admitted tools and schemas. > - The benefit is that each additional session on the same catalog adds about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is faster. ## Linked Issues or Issue Description Refs #21415 (releases a finished session's tool authority). Refs #21440 (adds the catalog memory benchmark that this change extends). **What existing behavior does this improve?** `createAssignedMcpTools` in `server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call kept every gateway descriptor (with its provider metadata) and a separate canonical clone of each input schema. Each `definitions()` call also made a new `structuredClone` of every schema. **Subsystem affected** Native runtime: assigned app tools (MCP) for native agent sessions. **Current behavior** Retained heap grows by one full catalog for each session. Measured with the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100 sessions. Each `definitions()` call takes 14 to 41 ms because it clones every schema. **Proposed behavior** Sessions of the same company that get identical admitted tools share one frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms. **Reason and benefit** The server must hold 1,000 concurrent agents in one process. The catalog was the largest per-session heap cost. Revocation and authorization do not change: every call is still authorized again by the gateway, and every new session still reads a fresh listing. **Breaking changes** None in the API or data. The direct definitions that `definitions()` returns are now frozen. A consumer that writes to a returned definition or schema gets a `TypeError` and does not change other sessions. No production consumer writes to them (see Risks). ## What Changed - `server/src/services/native-runtime/assigned-mcp-tools.ts`: - A session keeps only a small frozen record for each tool (gateway name, display name, description, risk), plus the frozen schema and the frozen definition. It does not keep the gateway descriptors. - New per-process index of shared catalogs. The key is the SHA-256 of an identity string: `[companyId, [[assignedName, gatewayName, displayName, description, risk, sha256(JSON of the input schema)] for each listed tool, in listing order]]`. On a hit, the code also compares the full identity string, so a hash collision cannot give a wrong catalog. - The identity is computed from the fresh listing that the gateway returns for this session. The gateway applies all per-agent, per-run, per-profile, and policy filtering, and the approval description suffix, before it returns the listing. Thus two sessions share only if they were given exactly the same tools, names, descriptions, risks, and schemas in the same company. Without `companyId`, nothing is shared. - Schemas, definitions, tool records, and the definition arrays are deeply frozen. Search results still return cloned copies. - Sessions that start at the same time with the same catalog wait for one build and share it. - The index holds each catalog through a `WeakRef`, so it never keeps a catalog alive. The index has at most 512 entries and 256 MiB (accounted schema and description bytes). The oldest entries are removed first. A catalog that is larger than the byte limit is not indexed. - New `close()` on the assigned tools. When the last session that holds a catalog closes, the index entry is removed. A `FinalizationRegistry` also removes entries whose catalog was collected. - `definitions()` returns the shared frozen definitions for the session's work mode. It no longer clones every schema on each call. - `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`: `close()` also closes the assigned tools. The session executor already closes the authority when it revokes a session's tool authority. - `server/src/services/native-runtime/native-session-executor.ts`: one line. It passes the run binding's `companyId` to `createAssignedMcpTools`. - `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes `companyId`, adds `--companies`, measures session start and `definitions()` time, and returns fresh copies of fixed rows (like database reads). - Tests in `assigned-mcp-tools.test.ts` and `paperclip-runner-tool-authority.test.ts`. ## Verification Red/green tests in `server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared assigned tool catalogs"): - Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema catalog. The heap for the other 19 sessions must be less than two catalogs. Red on master: `expected 14025976 to be less than 3788112`. Green with this change. - Shared deeply frozen schemas, also for sessions that start together. Red on master (`Object.is` equality fails). - Mutation isolation: writes to a returned definition, schema property, and `required` array, and a delete, throw `TypeError`. Changes to the gateway's source rows and to a search result do not change any session. Red on master (no throw). - Release and bounds: the entry stays while one session holds it and goes when the last session closes. `close()` is idempotent. A new session after release builds a new catalog. 517 companies keep the index at 512 entries or fewer. Red on master (no `close` or stats). - Guard tests: two companies with identical catalogs do not share. Sessions without a company do not share. Fewer tools, an edited schema, an edited description, or a different risk do not share. A revoked or edited tool is served from fresh discovery. On master these pass, because master never shares. To prove they test the key, I removed `companyId` and the schema digest from the identity: three guard tests failed, and they pass with the full key. - An existing test changed: "does not project gateway secrets..." now expects a write to a returned schema to throw `TypeError` (before, the write changed only a private clone). The isolation check after the write is the same. - `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes the assigned tools. Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one company). Command: `cd server && node --expose-gc --import tsx scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N> --schema-bytes 5700 --sessions <S> --turns 5`. | Tools | Sessions | Total before | Per session before | Total after | Per session after | | --- | --- | --- | --- | --- | --- | | 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB | | 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB | | 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB | | 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB | | 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB | | 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB | The first session also uses about half the heap of before, because the session no longer keeps the gateway descriptors. When all sessions are dropped without `close()`, the catalog is collected (0 bytes held after release). Session start, interleaved A/B in one process (old and new `createAssignedMcpTools` alternate, 40 starts each, median of the last 35, includes a fresh copy of the listing; the machine had a load average of 25 to 65): | Tools | Listing copy only | Before | After | | --- | --- | --- | --- | | 400 | 22.6 ms | 68.0 ms | 37.6 ms | | 800 | 30.5 ms | 72.5 ms | 41.2 ms | A start that reuses a catalog does no schema clone and no admission. It still serializes and hashes each schema to compute the key. The first start of a new catalog does the same admission work as before (the cold first start was about 600 to 750 ms in both versions on this busy machine). `definitions()` median: 14 to 41 ms before, 0.01 ms after. Other checks: - `cd server && npx vitest run src/services/native-runtime/assigned-mcp-tools.test.ts src/services/native-runtime/paperclip-runner-tool-authority.test.ts src/services/native-runtime/runner-api.integration.test.ts src/services/native-runtime/runner-tool-results-prp.integration.test.ts src/__tests__/deployment-connections.test.ts`: 204 passed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "assigned|tool authority|released|MCP"`: 31 passed. - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10 passed. - `cd server && npx tsc --noEmit`: pass. - `pnpm check:module-boundaries`: pass. - `pnpm verify` selects a broad plan (the benchmark script is a broad input), so CI supplies the full suite. ## Risks - Frozen definitions: the direct definitions now go to the runner drivers as frozen objects. I checked the consumers. `PaperclipRunnerToolAuthority.definitions()` filters and spreads them. The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them, spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP bridge), or serialize them. Their types are `readonly Readonly<Record<string, unknown>>[]`. The first-party semantic tool definitions are already shared module constants that go through the same path without clones. If a consumer that I did not find writes to a definition, it gets a `TypeError` instead of a silent change. Catalogs over 256 tools or over the listing budget use the two on-demand tools and are not affected. - Authorization: the shared data is only the result of a listing that the gateway already authorized for that session, keyed on the full content plus the company. Execution still goes through the gateway with the session's own token on every call. Search still intersects with a fresh listing. A new session still reads a fresh listing, so a revoked or edited tool takes effect for new sessions and for existing sessions at the same points as before. - Several server processes: the index is a per-process memo of the listing content. Each process builds its own catalog from its own listing. Nothing is stored or shared outside the process. - The index is bounded (512 entries, 256 MiB) and holds catalogs only weakly. Index eviction only stops future sharing. It never frees memory that a session uses. - No schema change. No migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs, heap and timing benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
754b1f36d3 |
perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every running native agent. Each run holds a native execution lease that the process must renew before it expires (60 seconds for a remote run, 20 minutes for a local run) > - Each run renewed its lease on its own timer, in its own transaction, on a private pool of two connections. At 1,000 remote runs that is about 50 transactions each second through two connections > - When renewals queue behind each other, a renewal can miss its 60 second deadline, and the run is lost although the server is healthy > - Each renewal also woke six live-run recovery queues for its run. The recovery owner then re-checked a healthy run about every 20 seconds, although a renewal only moves the deadline later > - This pull request renews all leases of one process with one shared timer. All leases that are due go into one transaction of fixed size, the same model that the legacy controller lease already uses. A renewal no longer wakes recovery, as #20924 did for the legacy lease > - The benefit is that the renewal cost no longer grows with the number of runs, and all safety properties of a single renewal stay the same ## Linked Issues or Issue Description Refs #20924 (the same change for the legacy controller lease), #21242. **What existing behavior does this improve?** Native execution lease renewal in `server/src/services/native-runtime/native-execution-lease.ts`. Each run had one `setInterval` and one transaction per renewal: statement timeout, row lock, recovery notification, update, commit. **Subsystem affected** Server: native runtime, controller lease renewal on the liveness connection pool. **Current behavior** The renewal cost grows with the number of runs. A local load model with 1,000 remote runs sends 35.7 lease transactions and 212.8 lease statements each second, and the event-loop utilization is 0.34. All of them use a pool of two connections. **Proposed behavior** One timer for each database client renews all due leases in one transaction for each 200 runs. With 1,000 remote runs the same model sends 0.29 lease transactions and 1.7 lease statements each second, and the event-loop utilization is 0.07. **Reason and benefit** The plan for 1,000 concurrent agents needs renewals that do not scale with the run count (item 1d). A renewal that misses its deadline loses the run. **Breaking changes** None. The `NativeExecutionLeaseRenewal` contract (`stop()`, `remainingMs()`) and `renewNativeSessionExecutionLease` do not change. ## What Changed - `native-execution-lease.ts`: a renewal (batched or single) no longer calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline later. Recovery already holds a deadline at the earlier expiry. At that deadline it re-reads the row and moves to the new expiry. A lapsed lease is still found at its expiry. This change does not touch the signals of the writers that release, revoke or take over a lease, or that change the run status. This also removes six notifications for each renewal, which was 1,200 notifications for a batch of 200 runs during a deploy overlap. - `native-execution-lease.ts`: one shared schedule for each database client replaces the timer of each run. When the first lease is due, all leases that are due within a quarter of their own interval renew with it. Thus the leases of a process converge on one batch for each interval. - A batch is one transaction on the liveness pool for each 200 runs: `statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the coordinator rows, and one `UPDATE … FROM (VALUES …)`. - The batch keeps the guarantees of a single renewal: - It locks each coordinator row before it reads the expiry, as before. Thus an owner whose lease expired while another writer held the row is never revived. - It extends a row only for the exact owner: run, company, task, lease owner, attempt, controller boot id, pid, process start, and a live lease. - The confirmed deadline is sampled after the locks and before the update, as before. - A run whose row was locked and not extended fails with `native_session_lease_lost`. The other runs in the batch succeed. - These rows do not renew in the batch. They renew alone through the unchanged single-run path: - A row that another writer holds (`SKIP LOCKED`). Thus one busy row cannot hold the batch until the statement timeout. - A row that the lock did not find (another owner, or no row). The single path gives the same answer as before. - A local lease that lapsed (host sleep). Only the single path may take it back, under the run row lock. - A batch that is cancelled (statement timeout `57014` or deadlock `40P01`) extends nothing. Its runs retry after 1 second, and each run keeps its own deadline timer. Before this change, a statement timeout waited for the next 20 second tick. - Any other batch failure, also an error thrown while the statement is built, may come from one bad row. Each run of that batch then renews alone at once through the single path, so a bad row can cost only its own run. A run that fails alone while others in its batch renew keeps renewing alone (out of later batches) until a single renewal succeeds. During a database outage the single renewals also fail; those runs return to the batch at their next interval, not after a 1 second retry, so an outage causes at most one queued single transaction for each run and interval. - Runs that become due while a batch is in flight wait for that batch. A batch that runs for more than 15 seconds no longer holds back the next batch. - A tick that passed while a renewal was in flight is skipped, as a fixed interval skips it. Thus a slow database gets no extra load. - The shared timer replaces the interval timer of each run. It exists only while this process holds a lease, and it fires at the next renewal deadline. With no held lease there is no timer and no SQL. - New export `renewNativeExecutionLeasesNow(db)` renews all leases of the process at once. Tests use it. ## Verification - New PostgreSQL tests in `native-execution-lease-batch.test.ts`: - 3 runs and 60 runs each renew in 1 transaction with the same number of statements. - 450 runs renew in 3 transactions. - In one batch, the runs whose lease was taken over, whose controller boot changed, or whose lease expired fail with `native_session_lease_lost`. Their rows do not change. The healthy runs renew. A newer attempt under the same owner is not lost, as before. - Interleaving: another transaction holds a coordinator row from before its lease expires until after it expired. The healthy run in the same batch renews at once. After the lock is released, the expired owner is lost and its row stays expired. - A takeover that commits before the batch: the batch extends nothing and the run is lost. - A local and a remote lease renew in one transaction, each for its own lease length. - A poison binding (an attempt outside the integer range) makes the shared UPDATE fail with `22003`. The other 4 runs still renew in the same tick, only the poison run is lost, and the next tick renews the 4 runs in 1 transaction again. - An invalid date throws while the statement is built. The other runs renew, the broken run is lost alone, and no lease stays busy (the next call renews again in 1 transaction). - New PostgreSQL test in `live-recovery-work.test.ts` with the real registered recovery owners: another server's native run renews its lease. No recovery pass runs after the renewal. At the old expiry recovery re-reads the row, keeps the run running and moves its deadline to the new expiry. When renewals stop, recovery acts on the run within 10 seconds after the end of the last lease. - Red first: with a shim on `origin/master` that renews each lease alone, the transaction test failed (`expected 3 to be 1`) and the 450-run test failed. - The unit tests in `native-execution-lease.test.ts` now drive the batch. New tests show that a non-cancelled batch failure renews the run alone at once, and that an outage (batch and single renewal both fail) does not retry every second and loses no run. - `native-execution-lease-reacquire.test.ts`: the host-sleep tests now go through the batch, which hands the lapsed local lease to the single path. - Load model (`startNativeSessionExecutionLeaseRenewal` for N remote runs that start at random times over 20 seconds, 70 second window, local PostgreSQL, machine load average about 50): | Runs | Lease transactions/s before → after | Lease statements/s before → after | Event-loop utilization before → after | Smallest margin to the 60 s deadline | Lost | | --- | --- | --- | --- | --- | --- | | 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0 s | 0 → 0 | | 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s | 0 → 0 | The window includes the 20 second ramp, so the steady rate before is higher (one transaction each 20 seconds for each run, 50/s at 1,000 runs). After the change, one batch carries up to 200 runs. ## Risks - **Behavior change: batch-wide error handling.** One shared transaction now carries up to 200 runs, so one database error reaches all of them. A cancelled batch only skips the extension and retries after 1 second. Any other batch error sends each run to the single path at once, so one bad row (a bad value, a cast error, an error while the statement is built) costs only its own run. In a database outage, up to one single transaction for each run waits on the two-connection pool at each interval. Each run's own deadline is unchanged. - **Behavior change: no recovery signal on renewal.** See below. - During a deploy overlap the old server renews its runs one by one and the new server renews in batches. Both use the same row lock and the same conditions, so they do not conflict. Rows written by the old version have the same columns. - `SKIP LOCKED` sends a busy row to the single path, which waits for the lock as before. - **No recovery signal on renewal:** recovery of a healthy run of another server now runs once for each lease period at the old expiry, not after each renewal. A dead run is found at the same time as before: at the expiry of its last lease. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
89f486cdab |
fix(heartbeat): run one dispatch pass per agent, back off refused claims, and enforce the run limit in the claim (#21433)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs. Each trigger (a run finishes, a wake arrives, a sweep runs) asks for one dispatch pass of that agent > - An agent could run more runs than its `maxConcurrentRuns`, dispatch passes could stack up without a limit, and every pass offered the same unclaimable runs again > - On hosted staging ( |
||
|
|
aa4a06653c |
fix(native): name the recorded turn failure when a native run fails (#21235)
Failed native runs now report the runner's recorded turn-failure code instead of the generic unclassified fallback, and the Sentry run-failure fingerprint includes runner:<code>. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6ad4529c5a |
fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.
## Linked Issues or Issue Description
**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.
**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.
**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.
Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.
## What Changed
- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.
## Verification
- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit
|
||
|
|
89e6575418 |
test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs get their tools from a session tool authority, which holds the run's assigned app (MCP) tool catalog > - #21415 releases a finished session's authority, so the server heap no longer grows with each finished session > - Two behaviors of that fix had no test: a released session fails closed, and the release when a follow-up run retires its warm predecessor and then fails before it admits its own authority > - The memory saving was measured once, by hand, with no script in the repository > - This pull request adds the two tests and a benchmark script. It changes no product code > - The benefit is that a later change cannot silently bring back the leak or let a released session reach its tools ## Linked Issues or Issue Description Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate; a reviewer comment there named the follow-up failure path). **What is missing?** #21415 tests that finished sessions return the authority map to its starting size. No test checks that a released session refuses late tool calls. No test checks the follow-up path, where a follow-up run retires the previous warm session (so the release at forget time is skipped while the follow-up owns the scope) and then fails before it admits its own authority. **Expected behavior** A late tool call or completion feedback for a released session fails with `native_session_tool_authority_unavailable`. The next run of the same scope gets a new authority with only its own assignments. When a follow-up fails after it retires its predecessor, the authority map is back at its starting size after that run settles. **Paperclip version or commit** master `edb6945ff3` (after #21415, `85f4a91e70`). ## What Changed - Test "refuses a released session's late tool call and completion feedback and admits the next run with only its own assignments". - Test "releases a retired warm predecessor's tool authority when the follow-up fails before admitting its own". A follow-up of the same warm scope with changed configuration retires the predecessor. Then its assigned tool listing fails. The test asserts that the predecessor was closed, that the follow-up failed with that error, and that `sessionToolAuthorityCount()` is back at its starting value. - `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained bytes of one assigned tool catalog per session, and schema copies per `definitions()` call. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "released native session tool authority|retired warm predecessor"`: 2 passed. - Red check: with the release at settle removed from `native-session-executor.ts`, both tests fail (`expected 1 to be +0`, and the late tool call is not refused). - Retained bytes per finished per-turn session were measured through `executePaperclipNativeSession` with the real `createAssignedMcpTools` (80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On `85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794 bytes (test-harness noise), and the authority map stayed at its starting size. - `cd server && node --expose-gc --import tsx scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes 2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per assigned tool catalog. Each `definitions()` call makes 80 new schema copies (about 296 KB), and none are kept. - Full `native-session-executor.test.ts` locally, with a load average near 78 from other work on the machine: 1,483 passed and 5 failed. All 5 failures were timeouts in tests this change does not touch (the slowest took 163 s). CI gives the clean full-suite result. ## Risks - Low risk. Tests and one benchmark script only. No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df1b12ad8e |
fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop
## Linked Issues or Issue Description
**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.
**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.
**Paperclip version or commit**
`master` at
|