mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 19:02:37 +01:00
7e7452e88f707d1c05d42608b7eeacee2915c5bc
2896
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
4fe34df748 |
Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task view shows a "Retry" notice when a task needs a person to start it again. > - The product rule is that a task that continues by itself never shows this notice, not even for a few seconds. > - A parent task that waits for its subtasks showed the notice for 3.4 seconds after its run succeeded. > - The cause is the queued-message grace in the task interruption presentation: it does not apply to `blocked` tasks. > - #21200 kept `blocked` out of that grace on purpose, so this pull request adds only the exact observed shape: a parent with open subtasks and a queued task message. > - The benefit is that a parent that waits for its subtasks no longer shows a false failure, and every other blocked task keeps its Retry. ## Linked Issues or Issue Description Refs #21264, #21200 (added the queued-message grace and kept `blocked` out of it) **What happened?** A Codex parent task ran across a two-server release swap. Its run waited for two subtasks and then succeeded with the disposition `blocked`. The task status became `blocked`. A user comment (a deferred wake, status `deferred_issue_execution`) had been queued 13 minutes earlier, while the run was active. For 3.4 seconds after the run finished, the task presentation was `state: "failed"`, `canRetry: true`. Then the release drain cancelled the wake as "Queued messages were already delivered to a completed run", and the presentation changed to `resolved`. The subtasks finished later. Their `issue_children_completed` wake continued the parent, and it completed without help. The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`. The `queuedDuringRun` check shows a succeeded run with user input queued before it finished as `working` for up to 2 minutes. It applies only to `todo`, `in_progress` and `in_review`. For a `blocked` task the presentation fell through to the last line: `failed` with `canRetry: true`. The workspace finalization deferral (`native_workspace_finalization_busy`) in the same window does not change the presentation. The presentation does not read it. **How this differs from what #21200 narrowed** #21200 first applied the grace to every status. Its commit `a3fae9bb67` removed `blocked` because input queued on a blocked task is not promoted by itself, and because a recovery that gives up hands its hold to the board and must end with Retry. This change does not add `blocked` back. It adds one narrow case, and all of these must be true: - The latest run **succeeded**. A failed, interrupted or given-up run is not affected. - The task is `blocked` **and has open subtasks of its own** (not `done` or `cancelled`). Their completion wakes the parent again (`issue_children_completed`). - The pending item is an ordinary **task comment** wake (`commentId` or `wakeCommentIds` in its payload), not a Retry request (`taskRetry.sourceRunId`). - The comment was queued before the run finished, the run finished less than 2 minutes ago, and saved-work recovery is not exhausted. These are the existing bounds. A blocked task that waits for a person or a board decision has no open subtasks, so it keeps Retry as before. **Expected behavior** After the run succeeds, the parent shows `working` with no Retry while its queued message is settled. Then it shows `resolved` (message already delivered) or `working` (message promoted to a new run). If nothing settles the message in 2 minutes, Retry shows. **Steps to reproduce** 1. Assign a parent task that creates subtasks and waits for them. 2. Post a comment on the parent while its run is active. 3. Let the run succeed with the disposition `blocked`. 4. Read the task presentation in the first second after the run finishes. It shows `failed` with Retry. **Paperclip version or commit** master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - The batched wait read also reads whether the wake carries a task comment (`queuedComment`). - One company-scoped read finds which `blocked` tasks with a succeeded latest run and a queued comment have open subtasks (`waitingOnOpenSubtasks`). It runs only for those tasks. - The `queuedDuringRun` grace also applies to such a task. The `todo`, `in_progress` and `in_review` behavior does not change. - `server/src/__tests__/task-interruption-automatic-path.test.ts`, new embedded-Postgres tests: - The exact observed state: a `blocked` parent with two open subtasks, a run that succeeded 1 second ago, a user comment wake queued 13 minutes earlier. Shows `working`, no Retry. - The same parent after the wake is cancelled as already delivered: `resolved`. - The same parent after the wake is promoted to a new run: `working` on the new run. - Guard: the same parent after the 2-minute grace: `failed` with Retry. - Guard: a `blocked` task with a queued comment and no subtasks, or only `done` or `cancelled` subtasks (waiting for a person or a board decision): `failed` with Retry, unchanged. - Guard: a parent with open subtasks whose queued wake carries no comment: `failed` with Retry, unchanged. - Guard: a parent with open subtasks and a queued comment whose run failed and whose recovery hold went to the board: `failed` with Retry. - Guard: a parent with open subtasks and a queued comment whose saved-work recovery is exhausted: `failed` with Retry. - A parent whose run succeeded with no queued message stays `resolved` when its subtasks are open, blocked or cancelled. - The existing #21200 assertion (a `blocked` task with queued input and no subtasks shows Retry) is kept unchanged. ## Verification - `cd server && npx vitest run src/__tests__/task-interruption-automatic-path.test.ts` - Red on master `bbafa78fe2`: "shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled" returns `state: "failed"`, `canRetry: true`. This is the value that the wave recorded. The other 15 tests pass on master. - Green with this change: 16 of 16. - A plain `blocked` inclusion fails 3 of the guards: the #21200 assertion, "still offers Retry on a blocked task with a queued message and no open subtasks" and "still offers Retry on a parent with open subtasks when its queued wake carries no message". This shows that the narrow condition is necessary. - `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`: green (116 tests with the file above). - `cd server && npx tsc --noEmit -p .`: 0 errors. - Not verified: no new chaos wave. The flash was seen once in a loaded release-swap wave. The server test reproduces its saved state. ## Risks - Low risk. The change is presentation only. It does not change wake promotion, dispatch or Retry. - A parent with open subtasks and a stranded queued comment now shows `working` for up to 2 minutes after its run succeeded, and then Retry. Before, it showed Retry at once. - One extra indexed read (`issues_company_parent_idx`) runs only for blocked tasks with a succeeded latest run and a queued comment. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fde0328ff5 |
fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution recovery continues a task in a fresh run after a restart or a lost run left it unfinished > - #21234 paced these continuations: at most 1 per agent and 5 per company per pass, and a continuation waits while its agent already has a queued recovery run or a full queue > - A continuation that waited 5 minutes must enter the queue anyway, but that bound applied only to the full-queue check. The "agent already has a queued recovery run" check was in the page query, and it had no bound. > - On staging, an agent had queued recovery runs that did not start for hours. Its pending continuation was never read again, and its task showed "working" with nothing running or queued > - This pull request applies the 5-minute bound to both checks, stops one undeliverable continuation from using its agent's or company's slot, and stops the task view from showing "working" for a continuation that is overdue > - The benefit is that every pending continuation enters the queue within a bounded time, also after a restart, and the pacing still protects people from recovery herds ## Linked Issues or Issue Description Refs #21234, #21135 **What happened?** On staging, a task had a fresh-continuation hold with `continuationDelivery: "pending"` since about 03:30Z. More than 3.5 hours later, no run had started for the task. The task view showed the interruption state `working` with `canRetry: false`, so the user saw nothing wrong. The task's agent had 20 concurrency slots and 11 running runs. It also had queued `issue_continuation_needed` runs from 22:54Z that had not started. The delivery pass reads pending continuations with a query that leaves out every agent that has a queued recovery run (`issue_recovery_action_restored` or `issue_continuation_needed`). That condition had no time bound, so the pass never read this continuation again. The 5-minute admission bound was never reached. Across the company, 203 open tasks showed `working` while the API showed no queued or running run for them. 162 queued runs were more than 1 hour old. **Expected behavior** Every pending continuation enters its agent's queue within a bounded time, also when the agent's queue holds recovery runs that do not start, and also after a server restart. Pacing still limits how many continuations enter a queue at the same time. A task does not show "working" when its continuation is overdue and nothing runs or is queued. **Steps to reproduce** 1. Give an agent a concurrency limit of 1. 2. Put one queued `issue_recovery_action_restored` run on that agent that does not start. 3. Record a pending fresh continuation for another task of the same agent. 4. Wait more than 5 minutes, and let the 15-second continuation pass run. 5. See that the continuation stays `pending`, the task shows `working`, and no run starts. **Paperclip version or commit** master at |
||
|
|
2483cb46c2 |
feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent gets environment variables from its execution environment, its own configuration, its project, and its routine. Secrets are resolved for each run. > - A plugin environment can run the agent as a native Runner task on its own host, through the environment task contract. > - That contract has no field for the agent's variables. Runs on a plugin environment get only the variables that the plugin sets itself. > - This pull request adds a versioned contract field for the values, sends them on a session handoff, lets Codex commands inherit the variables in an external sandbox, and lets a plugin protect the variables that it sets itself. > - The benefit is that plain and secret variables work on plugin environments in the same way as on local environments. ## Linked Issues or Issue Description Refs #21233. That pull request by @lucktastic finds the same gap and proposes the same capability and field names (`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull request uses the same names, so providers implement one contract. It differs in these places: - Codex commands get the variables through an external-sandbox denylist, not through key names sent at `prepare`. A retained Runner does not need a new preparation when a key is added or removed. - A session handoff `resume` gets the values, because it starts a new Runner on another computer. Value changes on the same computer need no new field: Core's process reuse key includes the values, so Core starts a new task. - A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or warns when one is saved. - The run log names skipped variables only for an older provider. Thank you to @lucktastic for the diagnosis and the contract shape. **What happened?** An agent has environment variables, plain and secret. Its runs on a plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS` in the run lists only the variables that the plugin sets. **Expected behavior** Each run gets the agent's configured variables. **Steps to reproduce** 1. Set a plain and a secret environment variable on an agent. 2. Assign the agent to a plugin environment that provides Runner tasks. 3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a command that the agent runs. 4. The agent's names are not in the list. ## What Changed - **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`): - A provider advertises `configuredEnvironmentVersion: 1` on `describe`. - `submit` gets `configuredEnvironment`. `resume` gets it only for a session handoff. - Names, count, and sizes have the same limits as the Runner projection. - **Server**: - `createRunnerdBackendWithinSessionClaim` gives the run's configured projection to the plugin task owner. The projection has no provider credentials and no generated runtime keys. - `createPluginRunnerTaskOwner` sends the fields only to a provider with the capability. An older provider gets no new field, and the run log names the variables that were not set. The log never prints values. - **Codex** (`codex-security-config.ts`): - An externally sandboxed launch uses `inherit="all"`, `ignore_default_excludes=true`, and an `exclude` list of Codex's own credentials, instead of an `include_only` list fixed at preparation time. - A test reads the Runner's Codex environment keys, and fails if a new key is not classified. - Host launches keep the bounded allowlist. - **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args` still grants the notification files to read-only profiles when no allowlist is present. Plan mode in an external sandbox needs these grants. - **Protected keys**: - An environment driver can declare `protectedEnvironmentKeys` in its manifest. - Paperclip rejects a protected key on an agent or environment that uses the driver. The 422 has code `protected_environment_variable`. - On a project, a routine, or another agent, Paperclip saves the key. The response has a `configurationWarnings` notice that names the environments. The project and routine pages show the notice as a toast. The agent pages already show configuration warnings. - **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the provider obligations. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed SDK source pin. ## Verification - `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts` (38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too many variables, and oversized values. - `server`: `vitest run src/services/native-runtime/plugin-runner-task.test.ts` (95 pass). Values go only to a provider with the capability, never in `prepare`. `resume` sends values only for a handoff. - `server`: `vitest run src/services/protected-environment-keys.test.ts src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices, and the project route response. - `server`: all route suites that import the project, routine, environment, or agent routes (45 files, 1689 pass). - `packages/shared`: `vitest run src/validators/plugin.test.ts` (64 pass). - `packages/paperclip-runner`: `vitest run src/drivers/codex src/configured-environment.test.ts src/live/runner-provider-launch.test.ts` (444 pass). - `packages/paperclip-runner/runner`: `cargo test -p paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as before). - `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`, `packages/paperclip-runner`, `server`, and `ui`. - Not run locally: `pnpm verify --full` and browser suites. Required CI covers the final revision. ## Risks - **Existing providers:** low risk. Core sends no new field to a provider without the capability. Their runs behave as before, with one added log line when variables were skipped. - **Codex in an external sandbox:** commands now see every variable in the Runner environment except Codex's own credentials. The execution environment builds that environment from an allowlist. If the Runner gives Codex a new credential, the drift test fails until the key is classified. - **Saving env:** new rejections happen only when an installed driver declares `protectedEnvironmentKeys`. No driver declares them today. - **Providers:** they must keep the values transient and never save or log them. The doc lists these obligations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e73fc71331 |
fix(environments): continue on a new sandbox when an idle runner sandbox never starts (#21323)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - On staging, Daytona left two idle sandboxes in the `archiving` state for more than three hours, so no resume call could succeed > - The native runner leases had no verified harness backup, so the run never replaced the sandbox, even though the task files were saved > - Each retry spent six minutes on the same stuck sandbox, the bounded retries ran out, and the task waited for a person to press Retry > - This pull request continues on a new sandbox when the provider has proven it cannot start the old one and the workspace is saved, and it keeps the old sandbox > - The benefit is that the task recovers by itself and a sandbox that may hold unsaved runner state is never destroyed ## Linked Issues or Issue Description Refs #20626, #21113 **What happened?** Eight Codie runs on staging failed in one hour with `reusable_sandbox_resume_failed`: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` They came in two clusters (05:59 to 06:00Z and 06:32 to 06:38Z). Each run took about six minutes, so the 330-second idle resume budget from #21113 was applied. All runs were already `transient_failure_retry` wakes. Investigation of one instance (lease `c23d6120…`, run `d16c272e…`): - The lease was `released` at 02:37Z with a confirmed stop-and-retain. No live run held it, so the lease was idle and the extended budget applied. - A read-only Daytona API read at 06:5xZ showed the sandbox in state `archiving` (desired state `archived`), last updated 03:38Z, one hour after the stop (the auto-archive interval is 60 minutes). The second sandbox (`e0e6cfe5…`) showed the same state since 03:35Z. Daytona does not start a sandbox while it archives, so every resume call timed out. More time or a new HTTP connection could not help. - The task workspace was clean and had a committed saved snapshot. - The agent uses `paperclip_runner`. The last run on the sandbox failed while it saved the runner session, so the lease had no `nativeHarnessBackup` stamp. `reusableSandboxReplacementIsSafe` requires a verified harness backup for a runner lease, so the run never replaced the sandbox. It failed strictly, the bounded retries (2) ran out, and the task waited for a manual Retry. **Expected behavior** When an idle sandbox never starts through the whole provider start deadline and the task files are saved, the run continues on a new sandbox automatically. A sandbox that may hold unsaved runner state is not destroyed. **Steps to reproduce** 1. Run a `paperclip_runner` task on a reusable Daytona sandbox. Let the last run end without a harness backup (for example, the runner exits while it saves the session). The workspace is clean with a committed snapshot. 2. Let Daytona start to archive the stopped sandbox and get stuck in `archiving`. 3. Wake the task. Every `environmentResumeLease` call times out, the run fails after about 330 seconds with `reusable_sandbox_resume_failed`, and the retries fail the same way. **Paperclip version or commit** Staging Core `0c04ff76eb` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `paperclip_runner` (native runner) on a Daytona sandbox environment. **Root cause** `server/src/services/reusable-sandbox-resume-replacement.ts`: a runner lease without a verified harness backup was never replaceable, for any failure. The only other path was to retry the same sandbox, which cannot work while the provider is stuck. Replacement also always retired the old lease to `pending_cleanup`, so allowing it without a backup would have destroyed the only copy of the runner session state. ## What Changed - `reusable-sandbox-resume-replacement.ts`: `reusableSandboxReplacementIsSafe` becomes `reusableSandboxReplacementPlan`. It returns `retire` (same rules as before), `keep`, or `null`. It returns `keep` for a runner lease without a verified harness backup only when the caller passes `providerCannotStart`. All other conditions still apply: idle lease, same workspace, clean workspace, committed snapshot, this run is the writer, and no access refusal. - Add `keepReplacedReusableSandboxLease`. After the replacement lease is recorded, one guarded write marks the old lease with `reusableSandboxKeptAfterReplacement`. The lease keeps its status and its sandbox. Nothing destroys it. - `environment-runtime.ts`: set `providerCannotStart` only when the idle start deadline was extended and the run then used all of it on timeouts. Lease selection skips a lease with the keep marker, so later runs do not spend time on it and obsolete-lease cleanup does not destroy it. - The replacement records `replacement` / `resume_failed` with no harness backup. The native executor already starts a fresh session in a new sandbox in this case (`freshReplacement`), and workspace preparation restores the committed snapshot. - The keep plan needs every resume call to time out. A rate limit, a reset, or any other failure in between keeps the strict behavior. - `reusable-sandbox-kept-marker.ts`: the marker key and a SQL guard. Every automatic teardown path skips a marked lease: retained-orphan listing and settle, `destroyReusableSandboxLease` (obsolete cleanup, expiry, run destroy), `destroyClaimedSandboxLease` (issue and workspace teardown), recovered-workspace cleanup, and abandoned-file release. Only an operator-consented environment delete still destroys it. - `doc/execution-semantics.md`: document the rule. ## Verification - Red before the fix: the new staging-shape test in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (released runner lease, no harness backup, clean committed snapshot, every resume times out) failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Review round: "keeps an idle sandbox with no harness backup when some resume calls failed for another reason", "never destroys a sandbox kept after replacement through the run or workspace teardown paths", and the retained-orphan case "keeps a warm sandbox while a replacement kept it" each failed before the fix and pass after it. `environment-retained-orphan-sandbox.test.ts`, `abandoned-workspace-files-hold.test.ts`, `workspace-recovery-preparation-gates.test.ts`, `workspace-recovery-admission.test.ts`: all pass. - Green after the fix: 22/22 pass in the replacement suite. The run gets a new sandbox within 330 seconds, makes no destroy call, the old lease stays `released` with the keep marker, and the next run resumes the new sandbox only. With unsaved work or no saved snapshot, the run still fails strictly and does not mark the old lease. Earlier tests (a refusal without a backup, a live owner, a dirty workspace with a timeout) are unchanged. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/__tests__/reusable-leases-default.test.ts`: 215 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - Not run: a live replacement on staging. It is verified after deploy on the next wake of the affected tasks. ## Risks - A runner task that hits this path starts a fresh runner session. The files come from the saved snapshot, and the task, comments, and run history are kept. Before this change, the task stopped and waited for a person. - The kept sandbox stays with the provider (archived storage) until someone removes it. This is the cost of never destroying unsaved runner state. - The rule needs the full extended start deadline (about 330 seconds) of timeouts, so a slow but healthy start still resumes the same sandbox. - No schema change. Old leases follow the same rule. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
503a736195 |
feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - `.github/workflows/runner-image.yml` builds the Runner alone and publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`, one time for each set of inputs > - The build used no cache, so every run compiled the daemon again. The first publication took 25m46s from push to tag > - The toolchain and the daemon change only with the Rust sources. Most commits that change the Runner's inputs do not touch them > - This pull request lets the build legs read a registry build cache for those stages. Only a new job can write the cache, and only for a `master` push > - A weekly run builds with no cache, so the cache cannot hide a broken build for long > - The benefit is about 3 minutes less on each build leg when the Rust sources did not change, with no change to the published image ## Linked Issues or Issue Description Refs #21230 (publication of the Runner image), #21229 and #21204 (the build this change makes faster). No issue exists for this change. **What existing behavior does this improve?** The Runner image build in `.github/workflows/runner-image.yml`. It builds the same image in less time. **Current behavior** Each build leg starts from an empty builder. It installs the Rust toolchain and compiles the daemon in every run. In the first `master` publication (run 43365670428) the amd64 leg spent 7m22s in the build step. The toolchain and the cargo build took 3m26s of that. **Proposed behavior** The build legs read `ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages up to the compiled daemon. A new `cache` job writes that cache after a publication. A scheduled run builds with no cache one time each week. **Reason and benefit** The stages before the last one read the Rust sources, the protocol files and the toolchain pin. In the last 30 days, 438 `master` commits changed an input of the image, and at most 181 of them changed a file those stages read. For the other commits, and for each pull request that does not touch those files, the build step is about 3 minutes shorter on each leg. **Breaking changes** None. The `Dockerfile` and each file in the image identity are unchanged. The published digest for a set of inputs does not change. This is part 4 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build (this pull request) This pull request uses no code from parts 1 to 3. It is stacked on them because all four parts edit `doc/runner-image-contract.md`. ## What Changed - The build legs pass `--cache-from type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` to the first build command. They do not log in. Their permissions stay `contents: read` and `packages: read`. - A new `cache` job writes the cache with `--cache-to type=registry,...,mode=max` and `--output type=cacheonly`. It runs only for a `master` push in the canonical repository, for inputs that had no image. Its permissions are `contents: read` and `packages: write`. It has no `id-token` and no `attestations` permission. - The `cache` job needs the `publish` job, and no job needs the `cache` job. A cache write cannot delay or fail a publication. - The `cache` job builds the target `runner-artifacts-runnerd` and no later stage. The last stage takes the commit as a build argument, so no other commit can reuse it. The install and its dependency lifecycle scripts are in that stage, so the job with the write token does not run them. - The `cache` job has one concurrency group for each architecture, a job time limit and a step time limit. - A `schedule` trigger starts a weekly build. It passes `--no-cache` to the first command and reads no cache. It runs all checks and publishes nothing. A fork's schedule runs nothing. - `server/src/__tests__/runner-image-workflow.test.ts` pins the new security properties: which jobs hold `packages: write`, that the build legs have no credential and cannot write the cache, the conditions of the `cache` job, that the `publish` job does not depend on it, and that a scheduled run cannot publish or write the cache. - `doc/runner-image-contract.md` has a "Build cache" section. ## Verification Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm` container (Linux arm64, Node 24.22.3): - `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55 tests pass on both. - `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass on both. The container needs `jq`. - `pnpm verify` exits 2 for this change, because a workflow file is a global CI input. Required CI supplies the broad coverage. - Each new assertion fails when the workflow breaks its rule. I checked seven such edits, for example: the `publish` job needs the `cache` job, the `cache` job gets `id-token: write`, a build leg gets `--cache-to`. Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2 (the version the hosted runners use), and an empty builder for each build: | Build of both variants | Time | Result | | --- | --- | --- | | No cache, cache ref does not exist | 260 s | An importer error in the log, exit 0 | | One changed TypeScript file, with the cache | 149 s | Toolchain and cargo steps cached | | Unchanged inputs and a new commit, with the cache | 129 s | Toolchain and cargo steps cached | | Each stage in the cache (a whole-build cache, which this change does not write) | 16 s | Both OCI archives complete | | `--no-cache` on the first command | 272 s | No cached step in the first command. The second command reuses all of it | - The images built from the cache pass `runner-image-layout.mjs check` and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256 as the daemon of a cold build. - A cache ref that does not exist, a host that refuses the connection, a name that does not resolve, and the real `ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not exist yet) each give an importer error in the log and a complete build with exit 0. - The `cache` job command on an empty builder with the cache present takes 6 s. It downloads no layer. Not tested before merge: - The `cache` job. It runs only on `master`. - The scheduled run. It starts only from the default branch. - The cache package. It does not exist. I did not test a private package, which answers an anonymous read with an authorization error. I expect the same importer error and a cold build. - A fully cached build on the hosted runners. The local test used the same BuildKit version and a different buildx version (0.37.0, and the runners have 0.38.1). ## Risks - **A second job can write packages.** Before this change, only the `publish` job held `packages: write`, and it runs no build. Now the `cache` job holds it too, and it runs BuildKit on the repository's `Dockerfile`. GitHub cannot limit that token to one package, so it can also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are: the job runs only for a `master` push in the canonical repository, the token goes only to the registry login, the build steps run inside BuildKit without it, the job does not run the install, and the job cannot sign. An image that it wrote has no attestation. - **Part of a published image can come from an earlier run.** The daemon in a published image can come from a step that an earlier `master` run compiled and cached. The attestation names the run that published, not the run that compiled. BuildKit reuses a step only for the same instruction and the same file content, but a party that can write the cache package can replace what a later build reuses. Write access to the cache package must stay as narrow as write access to the image package. - **A cached download is not repeated.** The toolchain installer, the toolchain and the crates are not downloaded again until the inputs of their step change. The weekly cold build finds a download that is gone. It does not compare its output with the cache. - **A registry failure after the cache import fails the leg.** A cache that cannot be read at the start is not a failure. A layer download that fails later is a failure. Run the job again. - **The cache package starts private.** GitHub creates it on the first write. A maintainer must make it public one time. Until then each build runs cold and nothing fails. - **The first cache write waits for an input change.** The merge of this pull request changes no input of the image, so its `master` run builds nothing and writes no cache. - **A cancelled cache job marks its run as cancelled.** When three cache writes for one architecture wait at the same time, GitHub cancels the middle one. The image of that run is published. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code. Tool use and shell execution. It wrote the change, ran the local builds and tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
4f4d823cfc |
feat(environments): forward the warm idle window to plugin lease acquisition (#21305)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugin environment drivers run agent tasks on remote computers, and warm leases keep a parked Runner between runs for an idle window that Core records > - Some providers stop an idle computer on their own schedule (for example a sandbox auto-stop interval) > - If that provider timer is longer than the Runner idle window, the provider keeps paying for a computer whose Runner is already gone; if it is shorter, a warm resume can find the computer stopped > - The plugin cannot align the two because it never sees the idle window at acquire > - This pull request forwards the warm idle window to `environmentAcquireLease` as `runnerLifecyclePolicy` > - The benefit is that a provider can stop its computer just after the Runner idle window, without Core knowing anything about the provider's placement ## Linked Issues or Issue Description Refs #21289 Refs #21066 **What existing behavior does this improve?** Warm lease setup for plugin environment drivers. **Subsystem affected** Server environment runtime (`createPluginEnvironmentDriver`) and the plugin SDK acquire parameters. **Current behavior** Core records the warm lifecycle policy (`runnerTaskLifecyclePolicy`, including `idleTimeoutMs`) on the lease, but `environmentAcquireLease` does not receive it, so a provider with its own idle shutdown has to guess. **Proposed behavior** For a warm lease, `environmentAcquireLease` receives `runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs }`, the same value Core records. Other lifecycle modes do not receive the field. **Reason and benefit** A provider can align its idle shutdown with the Runner idle window. **Breaking changes** None. The field is optional and additive. ## What Changed - `PluginEnvironmentAcquireLeaseParams` has a new optional `runnerLifecyclePolicy`. - Core sends it only for warm leases. - `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` documents the field. - The runner-e2e SDK public-input pin is updated for the SDK change. An earlier revision of this PR also made Core try `environmentResumeLease` before a session handoff. That part is removed: placement is the plugin's decision, so Core should not decide to retry the same computer before moving the session. The plugin will own that inside its own resume instead. ## Verification - `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed. - `vitest run src/__tests__/environment-runtime.test.ts -t "portable task placement|delegates plugin environment leases with acquisition budget"` passed (5 cases). The portable test asserts the first warm acquire receives `runnerLifecyclePolicy` and the lease records `runnerTaskLifecyclePolicy`; the non-warm delegation test asserts the field is absent. Making the forwarding unconditional fails both delegation cases. - The full suite was not run locally; required CI covers it. ## Risks - A plugin that does not know the field ignores it. - Rollback: revert this pull request. No migration. Note: the branch name still reflects the earlier scope (resume before handoff). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
49f0030f06 |
fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts each agent's queued runs, up to the agent's `maxConcurrentRuns` > - #21234 starts person-initiated runs before system recovery runs, and it reads "person-initiated" from the run's own wake request > - When a person comments on a task that already has a queued system run, the comment joins that run. The run keeps its system wake, so dispatch ranks the person's message as system work > - Dispatch is event driven. The queued-run re-drive backstop covers only agents with no live run, so an agent with several slots keeps a free slot empty until one of its runs finishes, and runs can take hours > - This pull request makes dispatch read the person wakes that joined each queued run, and makes the re-drive backstop cover an agent that still has a free slot > - The benefit is that a person's comment starts first, and free slots fill within about a minute instead of waiting for an unrelated run to finish ## Linked Issues or Issue Description Refs #21234, #20986 **What happened?** On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76 queued runs at 03:12Z. No run started on that agent from 03:01:33Z to 03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least 2 slots were free. Other agents in the same company started runs during that time. Runs started again only after the next deploy restarted the server. Five queued runs came from a person: comments at 02:43Z, 02:44Z and 02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had the wake reasons `issue_reopened_via_comment` and `issue_commented`, the invocation source `automation`, and no person on the run's own wake request. **Expected behavior** A run that carries a person's comment ranks as person-initiated, also when the comment joined an existing queued run. When an agent has a free slot and a queued run is ready, the run starts promptly, without waiting for another run on the agent to finish. **Steps to reproduce** 1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it. 2. Queue a third run while no dispatch signal reaches the queue (for example, a claim declines it while a gate is still closed, and the gate then clears). 3. See that `listUndispatchedQueuedRuns` returns nothing, because the agent has a live run. The run stays queued until one of the 2 running runs finishes. 4. Separately, queue a system run on a task, then post a person's comment on the task so that the comment wake coalesces into that queued run. Queue older system runs on other tasks of the same single-slot agent. 5. See that dispatch starts the older system runs first. **Root cause** 1. `coalesceIntoActiveExecutionRun` (`server/src/modules/wake-queue/adapters/postgres.ts`) merges the person's comment context into the queued run and records the person's wake as a separate `coalesced` row with `runId` set to that run. The run's own `wakeupRequestId` still points to the system wake. `queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`) only read the wake behind `wakeupRequestId`. The task routes do record the comment author as the requester (`requestedByActorType: "user"`). The person was lost only at dispatch time. 2. `listUndispatchedQueuedRunPage` (`server/src/services/queued-run-dispatch-liveness.ts`) skipped every agent with any `running` or `scheduled_retry` run. That rule assumes one slot per agent. For an agent with several slots, a lost or declined dispatch is not re-driven until a run on that agent finishes. This change is not in `claimQueuedRun`, the fresh-isolated preflight, the wake-queue promoter or `fresh-continuation-fallback.ts`. **Paperclip version or commit** Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is based on master `6cb0dd0c6b`. **Deployment mode** Authenticated, public, Paperclip Cloud managed staging. **Agent adapter(s) involved** `paperclip_runner` (Codex native runtime). The fix is adapter-independent. ## What Changed - `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is a queued run of the agent (by `run_id IN (queued run ids)`, so an earlier saved comment that a run adopted is included). A run with such a wake ranks as person-initiated and is never counted as system recovery. - `listUndispatchedQueuedRuns` now returns a queued run when its agent has fewer `running` plus `scheduled_retry` runs than its `maxConcurrentRuns`, and its task has no `running` or `scheduled_retry` run. The re-drive backstop and the ownership I7 check use this query. - The query counts each agent's live runs once, in one grouped subquery that is joined to the queued runs. - Migration 0476 adds `agent_wakeup_requests_company_run_idx` (`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as `online_compatible`. `doc/DATABASE.md` describes it. - Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same limit as `parseHeartbeatPolicy`, including the single-run runner providers, which now share one constant. ## Verification - Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a queued run that a person's comment joined ahead of older system work". It fails on master (`expected null not to be null`: the older system run started first) and passes with this change. - Red/green: `server/src/__tests__/ownership-reconciliation-report.test.ts` "re-drives a queued run while its agent still has a free slot, and not once every slot is running". It fails on master (`expected [] to deeply equal [ Array(1) ]`) and passes with this change. - Red/green: "starts a queued run that adopted a person's comment saved before every queued run". It fails with a `requestedAt` bound and passes with the `run_id` lookup. - Red/green: "counts an agent's live runs in one grouped aggregate, not once per queued run" runs EXPLAIN on the page query. It fails with a per-row correlated count and passes with the grouped subquery. - New: "uses the agent's single-run limit for a runner provider that runs one turn at a time". - Two existing tests now set `maxConcurrentRuns: 1` so that they keep testing "the agent's only slot is busy". - `npx vitest run src/__tests__/recovery-wake-herd.test.ts src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts`: 71 passed. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. ## Risks - The re-drive backstop now also covers multi-slot agents with live runs. Each queued run is re-driven at most on its existing backoff (1 minute, then doubling to 15 minutes). A queued run whose task already has a live run is not re-driven. - The ownership I7 report can now list a queued run on a multi-slot agent that has a free slot. This is the stall that this change targets. - One more indexed read per dispatch pass (`agent_wakeup_requests` by company and run id). The new index is built concurrently, so the previous release keeps writing during the build. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only staging API reads in a browser tab). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0477f622d6 |
fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, and the workspace files live in those sandboxes until a save > - When a run cannot resume exactly, the fresh continuation fallback (#21135) abandons the old workspace epoch and keeps its unsaved files "recorded and unapplied" on the old sandbox > - The abandonment clears the dirty flag, the writer and the location, so the workspace looks clean. No code reads the abandonment receipt > - A later failed resume can then replace the sandbox, and cleanup can destroy it. That sandbox holds the only copy of the unsaved files > - This pull request adds one shared hold. The task continues on a new sandbox, and the old sandbox is parked until its files are saved, discarded, or kept for 7 days > - The benefit is that the fallback keeps its promise. Abandoned files are not lost ## Linked Issues or Issue Description Refs #21135 Refs #21297 **What happened?** After the fresh continuation fallback abandoned a workspace epoch, the receipt in `remoteWorkspaceAbandonedEpochs` was write-only. `reusableSandboxReplacementIsSafe` saw a clean workspace with a committed save and allowed replacement. `retireReplacedReusableSandboxLease` then moved the old lease to `pending_cleanup`, and the cleanup sweep destroyed the sandbox with the unsaved files. `nativeWorkspaceExportPendingForLease` (the check that the sweep, the retained orphan settlement and the scoped destroy paths use) also answered "nothing pending". **What did you expect?** A sandbox that holds abandoned, unsaved files stays until the files are saved, a person discards them, or a stated retention period ends. **Steps to reproduce** 1. Abandon a remote workspace epoch whose location names sandbox L. 2. Let the next run fail to resume L. 3. The replacement decision says safe, L goes to `pending_cleanup`, and the sweep destroys it. ## What Changed - New `server/src/services/remote-workspace-abandoned-files-hold.ts` holds the single predicate for "this lease holds abandoned files". It is one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a workspace in its company has an abandoned-epoch receipt whose `location.providerLeaseId` equals the lease's provider lease id, and that receipt has no `released` object. A receipt written by the current release (no new fields) counts as unreleased. - Replace and park. A failed resume of a held sandbox still lets the run replace it, so the task continues at once on a new sandbox restored from the last committed save. `retireReplacedReusableSandboxLease` parks the old lease as `retained` with the `abandonedWorkspaceFilesParked` marker (failure reason `abandoned_workspace_files_held`) instead of `pending_cleanup`. `reusableLeaseCanBeResumed` and `reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never select it again. The check is a metadata read with no query, so the run path does no extra work. - Every path that can destroy a sandbox now honours the hold: - `nativeWorkspaceExportPendingForLease` answers pending for a held lease. This covers the pending cleanup sweep (defer), the orphaned active lease sweep, the retained orphan settlement `heldBy`, `destroyClaimedSandboxLease` (terminal workspace reaper `destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the environment delete path, and run lease release with `destroy`), the `destroyRunLease` dispatcher, and the plugin sandbox teardown retry. - The run lease settlement (`environment_run_lease_active`) is the one caller that ignores the hold (`{ destructive: false }`). It never destroys a sandbox. It stops the sandbox and keeps it, or it moves a lease without a sandbox to `pending_cleanup`, where the sweep parks a held lease. If it waited for the hold, a dead run's lease would stay `active` and block the retry for the whole retention period. - `destroyReusableSandboxLease` parks the lease and returns without a provider call. This covers obsolete reusable lease cleanup, the destroy after an expired or mismatched resume, release with `expired`, and the sandbox driver's `destroyRunLease`. - The `retryPendingSandboxTeardown` dispatcher refuses a destructive teardown of a held lease. Stop-only cleanup still runs, because a stop keeps the filesystem. - The retained orphan listing (`alwaysHeldSql`) uses the same SQL fragment, so held orphans are not polled. - Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released` with a reason (`recovered`, `discarded`, `retention_expired`) and an actor. It moves the parked leases for that sandbox to `pending_cleanup` when no other unreleased receipt holds them, and it writes a `workspace.abandoned_files_released` activity entry on the task in plain language. No export path for abandoned epochs exists today. The `recovered` reason is the hook for one; this pull request adds no new export mechanism. - Retention expiry. New receipts carry `retainUntil` (7 days). An old receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt` includes the earliest retention end of any unreleased receipt. The shared cleanup worker therefore wakes at that real deadline, and a restart rebuilds it from the database. At the deadline the sweep runs `expireAbandonedWorkspaceFileHolds`. This releases each due receipt as an attributed system decision (actor `workspace-retention`) and logs the held receipts and parked leases for each company. Each expiry always writes a task entry: "Unsaved files from an interrupted run were kept for N days, and their retention period has ended." The entry adds "They are scheduled for removal." only when a parked lease was queued for cleanup. A separate entry, "... kept for N days and have now been removed.", is written only after the provider confirms the delete. Each receipt is released in its own transaction. A failed receipt is logged and skipped, and it is retried after a backoff (5 minutes, doubling, at most 6 hours). The other receipts are still processed. The entries reference a run or a task only if it still exists. If the task was deleted, the decision goes to the server log only. - Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that an unreleased receipt holds goes back to `retained` and is parked by the sweep. A sandbox that a release before this change already destroyed cannot be recovered. Its receipt ends at its retention time like any other receipt. - Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch` trims only released receipts and keeps the last 20 of them. It never drops an unreleased receipt. A new abandonment of the same sandbox and root replaces the unreleased receipt for that root. So a workspace holds at most one unreleased receipt for each distinct sandbox and root abandoned in the last 7 days. ## Verification - `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19 tests). It follows the lease through the real acquisition path with a failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`: - `gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone` - `pulls a held lease already queued for cleanup back to retained, and leaves it on the provider` - `treats a receipt written before release fields existed as unreleased` - `records a retention expiry on the receipt` - `hands the parked lease to normal cleanup once the files are recovered` / `... discarded` - `removes the files only after the retention period, and says removed only after the provider confirms it` - `does not report removal when the provider delete fails` - `records the end of retention without scheduling removal when another receipt still holds the sandbox` - `records the end of retention when it expires before the lease was ever parked` - `wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet` - `releases a receipt whose run was deleted and records it without the run reference` - `releases a receipt whose task was deleted without writing a task entry` - `keeps expiring later receipts when one fails, and retries the failed one after a backoff` - `keeps a held sandbox when the retained orphan settlement reviews it` - `lists the sandbox for the retained orphan settlement again once the receipt is released` - `keeps holding the first sandbox after many later abandonments on another sandbox` - `replaces and cleans the sandbox as before when the receipt names a different sandbox` - `replaces and cleans the sandbox as before when no epoch was abandoned` - Red on the base revision ( |
||
|
|
e363b0e52d |
fix(docker): keep local build output out of the Runner image's build context (#21312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The name of a published image is a digest of every file the Runner build receives from the build context > - `.dockerignore` keeps the Runner's own build output (`packages/paperclip-runner/dist`) out of the context, but not the eval kernel's (`packages/paperclip-eval-kernel/dist`), which the Runner's build also writes > - A tree where the Runner was built therefore has another inputs digest than a clean checkout of the same commit > - The Runner package's Vite builds write five more directories with the same problem > - This pull request adds the missing ignore rules, and a test that finds the next such directory > - The benefit is that a developer's tree and a clean checkout of the same commit name the same Runner image ## Linked Issues or Issue Description Refs #21204, which added the image identity. No issue exists for this change. **Problem or motivation** The inputs digest covers every file the build receives under `packages/paperclip-eval-kernel`. The Runner's build compiles the eval kernel and writes `packages/paperclip-eval-kernel/dist`. After a local Runner build, that directory is in the build context, so the digest of the tree differs from the digest of a clean checkout. A lookup of the published Runner image for such a tree finds nothing. CI checkouts are clean, so no published image is affected. **Proposed solution** Add `packages/paperclip-eval-kernel/dist` and `packages/paperclip-runner/dist-*` to `.dockerignore`, next to the rule for the Runner's own `dist`. Add a test that derives every build-output directory inside the image's inputs and requires a rule for each. **Alternatives considered** - Exclude the directory in the identity script only. Then the digest would not describe what the build receives, which is the property the identity is built on. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request stands alone. It is based on `master` and needs no other open pull request. ## What Changed - `.dockerignore`: two rules. - `packages/paperclip-eval-kernel/dist`: the Runner's own build writes it. - `packages/paperclip-runner/dist-*`: the output of the Runner package's Vite builds (`dist-browser`, `dist-issue-thread`, `dist-scenarios`, `dist-sdk`, `dist-standalone`). The Runner's `build` script does not write them, but a tree where one of those builds ran had the same digest drift. No tracked path matches the rule, and no later `!` rule re-includes those directories. - `server/src/__tests__/runner-image-identity.test.ts`: one new test. For each package among the identity's inputs it collects the `outDir` of every tsconfig a script compiles with `tsc -p`, the `target` directory beside every Cargo manifest a script names, and the `outDir` of every Vite configuration. It requires that the last `.dockerignore` rule matching each directory is an exclusion. Today that is eight directories. A new output directory inside the inputs fails the test until it has a rule. - No image build reads that directory from the host. The Runner stages, the `build` stage of the application image and `docker/daytona-runner` all compile the kernel themselves (`tsc -p`, not incremental), and the eval kernel is a development dependency of the Runner only. ## Verification The inputs digest, as the `runner-artifacts-identity` build target reports it, on a `linux/arm64` Docker host: | Tree | Without the rule | With the rule | | --- | --- | --- | | clean export at `62d673cc3f` | `0e6662ad…` | `0e6662ad…` | | the same, with an eval-kernel `dist` | `e4cf3095…` | `0e6662ad…` | | clean export at `bc71c23e0a` | `88fe2929…` | `88fe2929…` | | the same, with an eval-kernel `dist` | `ac331275…` | `88fe2929…` | | clean export at `731b52453c` | `88fe2929…` | `88fe2929…` | | the same, with two files in `dist-browser` | `44fe867d…` | `88fe2929…` | | the same, with one file in each of the five Vite directories | not measured | `88fe2929…` | - `0e6662ad…` is the digest of the image that `master` published at `25e9a9c2e9`, so the rule does not change the digest of a clean checkout. - A worktree after a real `pnpm --filter @paperclipai/paperclip-eval-kernel build` also gave the clean digest with the rule. - `runner-image-identity.test.ts` (42 tests) and `tests/runner-e2e/daytona-image.test.ts` (19 tests), which reads `.dockerignore`, pass on macOS and in `node:24.22.3-bookworm`. - The new test fails on the `.dockerignore` of `master` and names `packages/paperclip-eval-kernel/dist`. It fails without the `dist-*` rule and names `packages/paperclip-runner/dist-browser`. It fails without either of the two older Runner rules, and with a trailing `!` rule that re-includes one of the directories. Never run: - Any image build with the new rules beyond the identity target, including `.github/docker-context-checks.Dockerfile`. On a CI checkout the directory does not exist, so the rules remove nothing there. - `linux/amd64`. - The full test suite and the full typecheck. CI must supply them. ## Risks - **The sandbox image's content id changes one time.** `.dockerignore` is an input of that id (`tests/runner-e2e/daytona-image-content.ts`). At `62d673cc3f` the id goes from `be208823…` to `1e13a151…`. The next end-to-end run builds a new sandbox image, and the restart-chaos harness needs an image built from this tree. The image's contents do not change. - Low otherwise: the rules remove directories that no image build reads. The `Dockerfile`, `docker/daytona-runner/Dockerfile` and `.github/docker-context-checks.Dockerfile` name none of them and run no Vite build. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
03f9b83e52 |
feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work, and `master` now publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner` > - The application image will soon copy that Runner instead of compiling its own. Part 1 of this stack adds the resolver that picks the Runner for a build > - Two gaps make that next change risky. The resolver has never run in CI on `master`, so its lookup time and how often a commit has a published Runner are unknown. And `docker.yml` has no pull-request trigger, so a change to how the application image gets its Runner first runs after it merged > - This pull request adds a report-only job that runs the resolver on every `master` push, and a pull-request check that builds the Runner path of the application image > - The benefit is real data from `master` before any build depends on the resolver, and a failed check on the pull request instead of a failed image build on `master` ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** The resolver decides whether a build copies the published Runner. On `master`, a commit that changes the Runner's inputs is ahead of its publication for some minutes, and nobody knows yet how often an application build will meet that. The application image itself is built only on `master` and on tags. The change that removes the Runner's compile stages from that image will edit the `Dockerfile`, and today nothing builds those stages on a pull request. **Proposed solution** Add one job to `docker.yml` that only reports, and one new workflow that only checks. Neither changes what an image contains, and no existing job reads either of them. **Alternatives considered** - Run the resolver inside the build legs. A failure there could fail the image build. A job of its own, with `continue-on-error`, cannot. - Give the lookup the job token. The Runner package is public, so an anonymous lookup works and the job needs no credential. - Build `production` in the pull-request check. It adds three `COPY` instructions and a readability check to the two stages that this check builds, at the cost of the tool layer and two more copies of several gigabytes. - Trigger the check on every Runner source change. `runner-image.yml` already builds the Runner on those pull requests, and CI builds the server and the UI. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path (this pull request) 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `.github/workflows/docker.yml`: new job `runner_image_report`. - It runs only for a `master` push in this repository. - It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the result goes to the job summary with the line "Report only: no build read this result." - It has `continue-on-error: true`, declares no `outputs`, and no job needs it. Its failure does not fail the run. - It holds no credential. Its only permission is `contents: read`, the checkout does not keep its token, and the lookup is anonymous. It does not log in to a registry and writes to none. - Time limits: 8 minutes for the job, 5 for the step that builds, 2 for the lookup. - It is the first job in the file. Existing tests cut the other jobs out of the file by name, up to the next job or to the end of the file, so no other place leaves those tests unchanged. - `.github/workflows/docker-app-check.yml` (new): on a pull request, builds the `build` and `runner-provider-pack` targets with `PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only. - `permissions: {}` at the top, `contents: read` for the job, no credential, no `pull_request_target`. - It exports nothing (`--output type=cacheonly`) and pushes nothing. It reads the public build cache that `docker.yml` writes and never writes it. - It refreshes the lockfile in the build context first, as `docker.yml` does. - Time limits: 60 minutes for the job, 45 for the two builds. - Trigger paths, and why: | Path | Reason | | --- | --- | | `.github/workflows/docker-app-check.yml` | the check itself | | `Dockerfile`, `.dockerignore` | the recipe, and the files it receives | | `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that reuses or rebuilds the Runner | | `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp that step compares with the files the image build received | - `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90 lines): only security properties. The report job has no credential and one read permission, nothing can read it, and its two steps have time limits. The check has no credential, cannot publish, and exports nothing. In both, every action is pinned by commit and no shell script holds an expression. - `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and the check. ## Verification macOS (Node 26.12.0): - In `server/`: `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse` and `cloud-image-sentry` tests: 52 tests pass in 7 files. - `tests/runner-e2e` workflow tests (`workflow-security`, `private-trust-workflow-security`, `workflow-image-only`, `paid-runner-routing`): 27 tests pass in 4 files. - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass. - Both workflow files parse as YAML. - `docker buildx build --target runner-artifacts-context --cache-from type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64 --output type=cacheonly .` succeeds, so both flags are accepted and the cache is readable. Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same three sets pass: 52, 27 and 962 of 962. The two commands the report job runs were run by hand, with no credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca` the answer was `published`, with the two index digests that `master` published. For a commit whose image was not yet published the answer was `inline`, `absent`. Part 1 has the output. With the job at the end of `docker.yml`, two existing tests failed. They are the reason for its place: `docker-canary-promotion.test.mjs` reads the last job to the end of the file, and `release-verify-workflow.test.mjs` allows no BuildKit configuration after `merge-and-push`. Never run: - Either workflow. Only GitHub can run them. This pull request starts the check by its own path; the report job first runs on `master`. - The two builds of the check (`--target build`, `--target runner-provider-pack`) on a local host. Their time and disk use on a hosted runner are not known. - The full test suite and the full typecheck. CI must supply them. ## Risks - The check builds the whole `build` stage, with the UI and server builds. Its duration is not measured. It runs only when one of five paths changes, and its first real run is on this pull request. - `docker.yml` holds one concurrency slot for each ref until all its jobs end. The report job normally ends long before the build legs. A stalled job can hold the slot for at most 8 minutes when the legs end sooner. - `scripts/watch-release-source.mjs` warns about every job of the image run that waited more than 60 seconds for a runner. The report job is one more job that it can name. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - The lookup has 2 minutes. The registry client retries a failing request for longer than that, so a registry outage ends the step by its time limit, and the job then reports nothing. The run stays green. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
c730872974 |
fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In a chat, the agent's turn ends and the conversation waits for the person's next message. The stranded-work sweep must leave that wait alone > - The sweep recognizes a finished chat turn by `finalizationReasonCode: "conversation_turn_finished"` on the run record > - Run completion re-reads the run through the 64 KiB read projection. When the stored result is larger, that projection drops the finalization fields, and completion wrote the reduced copy back > - The chat turn then lost its finalization reason. Its reply was not presented, and the sweep queued `issue.productive_terminal_continuation_recovery` 30 s later. That is an extra run while the agent waits on a person > - This pull request keeps the full stored result through run completion, and makes the sweep read the committed decision from its durable rows > - The benefit is that a turn waiting on a person is never treated as stranded work ## Linked Issues or Issue Description **What happened?** In the paid chat eval `app-permissions.new-quarantined-action`, the CEO's chat turn yielded with a `response_wake` continuation and waited for a person to allow an app action. The stored run result was 113,937 bytes. Its saved record had the read projection's `truncated`, `truncationReason: oversized_result_json` and `originalSizeBytes` fields and no `finalizationReasonCode` or `decisionId`. The run's presentation decision was `interaction_owns_conversation_response`, so no reply comment was written. 30 s later the sweep queued `issue.productive_terminal_continuation_recovery`, and the case failed its run limit. **Steps to reproduce** 1. Finish a native chat turn that yields `response_wake`, so finalization records `conversation_turn_finished`. 2. Let the stored run result be larger than 64 KiB when run completion reads it. 3. Run completion writes back the reduced copy without `finalizationReasonCode`. 4. Run the stranded-work sweep. It queues a productive continuation run. **Expected behavior** The run record keeps its finalization fields, and the sweep leaves the waiting chat turn alone. No new run starts until the person replies. ## What Changed - `server/src/services/heartbeat/run-completion.ts`: run completion reads the run with `unsafeFullResultJson` before it merges the native result. It no longer writes the read projection's reduced copy back. - `server/src/services/native-runtime/native-response-wake-wait.ts`: `hasCommittedNativeConversationTurnWait` checks the run's committed finalization decision (`conversation_turn_finished`) through `native_run_finalizations` and `status_decisions` for the conversation's own agent. - `server/src/services/recovery/service.ts`: the sweep's finished-chat-turn skip also uses that durable check, so runs already saved with the reduced copy are recognized. - Tests: `run-completion.test.ts` keeps the finalization record when the stored result is larger than the read projection. `heartbeat-process-recovery.test.ts` finalizes a real chat turn with the eval's yielded result, saves the reduced run record, and expects two sweeps to start no run. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts -t "larger than the read projection"` — fails on master (the saved result is the truncated copy) and passes here. - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that yielded"` — fails on master (`continuationRequeued: 1`) and passes here. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts src/__tests__/heartbeat-process-recovery.test.ts` (614 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Run completion now reads the full stored result for one run. This is one row; the read projection still serves every API read. - The sweep check adds one indexed query for chat issues whose latest run succeeded and whose run record lacks the reason. - Previous-release saved state: no persisted shape changes. Runs already saved with the reduced copy are covered by the durable decision check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f79e2c5d84 |
feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - `master` now publishes the Runner alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The application image still compiles its own Runner on every build. The next steps make it copy the published Runner instead > - Before a build can do that, it must answer one question the same way as every other build: "does a published Runner exist for exactly this tree?" It must also fail clearly when the Runner it was given does not match the tree > - This pull request adds the script that answers the question, and a mode of the Runner vendor step that fails instead of rebuilding > - The benefit is that the later change to the image build is small, and that a wrong or unreachable answer can only mean "build the Runner yourself" ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** Three builders make images that contain the Runner: `docker.yml`, the merge-queue build and the release preview build. Each one would have to repeat three steps to find the published Runner of its tree: check the lockfile, compute the inputs digest in the build, and look up `inputs-<digest>`. Three copies can decide differently. A build that copies a Runner built elsewhere also has no Rust toolchain. If the copied Runner does not match the tree, `prepare-runner-vendor.mjs` starts a rebuild, and the build fails late at `cargo` with a message that hides the cause. **Proposed solution** Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to `prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets the variable in this pull request, so no workflow job changes. **Alternatives considered** - One command that builds and looks up. Two commands keep a registry credential, when a caller has one, out of the step that builds. - A wait for the publication of the commit. The decision whether the application build waits is open, so the resolver has no wait mode. It reports `absent` and stops. - A build argument that names the expected inputs. No `Dockerfile` declares one yet. The change that makes the application image copy the Runner will add both ends together. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 1 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode (this pull request) 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands that share a directory: - `inputs --dir <dir>` exports the `runner-artifacts-context` target, runs the lockfile check on that export, and builds the `runner-artifacts-identity` target when the lockfile is current. It reads no credential. A lockfile that is behind stops there: no image is published for such a tree. - `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`, `reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and `labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to `$GITHUB_STEP_SUMMARY`. - `mode` is `published` only for a complete publication whose full index names these inputs and both platforms. Both images are pinned by index digest. The slim digest is the one the full index names in `io.paperclip.runner.slim-digest`; the slim tag is never read on its own. - Every other answer is `inline`, with the last four values empty: | `reason` | Meaning | | --- | --- | | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests | | `absent` | no tag exists | | `incomplete`, `broken` | a publication stopped, or its two tags do not belong together | | `mismatch` | the full index names other inputs, or lacks a platform | | `unknown` | the registry refused, failed or could not be reached, or the lookup threw | - The resolver reuses `runnerLockState`, `readRunnerImageManifest`, `runnerPublicationState` and `runnerImageTags` from `scripts/runner-image-publish.mjs`. That file now exports `environmentCredentials`; it has no other change. - `server/scripts/prepare-runner-vendor.mjs`: with `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current build as before. When the build is not current, it prints the reason, starts no rebuild, leaves the stamp in place and exits with status 1. It also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same time. Without the variable the step is the same as before. - `doc/runner-image-contract.md` has a new section, "Choosing the Runner for an image build". `doc/DEVELOPING.md` describes the new variable. - The resolver is not an input of the Runner image, so the inputs digest does not change. ## Verification macOS (Node 26.12.0), in `server/`: - `node node_modules/vitest/vitest.mjs run scripts/prepare-runner-vendor.test.mjs src/__tests__/runner-image-resolve.test.ts src/__tests__/runner-image-publish.test.ts src/__tests__/runner-image-identity.test.ts src/__tests__/runner-image-workflow.test.ts src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6 files. 27 of them are new resolver tests and 3 are new tests of the vendor step, whose end-to-end test also has a new case. - `tsc` on the new test file, with the settings of `server/src/__tests__/tsconfig.json`: no error. Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted read-only): the same files, 179 tests pass. Real runs, on clean exports of the tree (`git archive`), with no credential, on a `linux/arm64` Docker host: - On `master` at `62d673cc3f`: `inputs` gave `lock=current` and `inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`. `resolve` gave `mode=published`, `image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162` and `slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`. These are the two indexes that `master` published for those inputs. - The same tree with one line added to a Runner source file: another digest, `mode=inline`, `reason=absent`. - On `master` at `bc71c23e0a`, which changed the Runner's inputs, before its image was published: digest `88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`, `mode=inline`, `reason=absent`. - Four deliberate faults in the resolver each make a test fail: no index check, only `absent` is inline, the identity is built for a lockfile that is behind, and the slim image is taken by tag. Never run: - A lookup with a registry credential. The package is public, so no caller needs one now. - Any workflow. No workflow calls the resolver in this pull request. - `linux/amd64`. - The full test suite, the full typecheck and `pnpm build`. `pnpm verify --explain` reports exit 2, "broader verification required", so CI must supply them. ## Risks - Low for existing behaviour: nothing calls the resolver, and the vendor step is the same unless the new variable is set. - The resolver does not verify the attestation of the image. The workflow that consumes the image must do that from the digest the resolver reports. - The resolver reports both build contexts, `runner-artifacts` and `runner-artifacts-slim`. A build that reads only one stage gets one context it does not use. - A `broken` publication is `inline`, not a failure. The Runner image workflow fails for that state. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
731b52453c |
fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns run leases and delegates remote execution to environment plugins. > - A retry must verify its old checkout on the placement that will run the task. > - A completed task must still release its provider resources. > - This change uses the existing lease for workspace verification and always calls the release hook after settlement. > - Saved settlement proof lets cleanup retry without repeating execution work. ## Linked Issues or Issue Description Refs #21237. That PR records stop proof before lease release. This PR keeps provider cleanup after settlement and changes workspace verification sequencing. Both touch task completion; preserve both changes if the merge order requires conflict resolution. **What happened?** Workspace verification ran before a portable task had a lease, so a provider could not verify the checkout on its selected placement. Core also skipped the provider release hook after successful task completion. This could leave resources allocated after the run ended. **Expected behavior** Verify the old checkout through the selected run lease. A confirmed refusal can use the authorized fresh-workspace fallback. A transport failure must remain retryable. Call the provider release hook after settlement and retry failed cleanup without changing the run result. **Steps to reproduce** 1. Retry a task whose prior checkout uses plugin-managed storage and portable task placement. 2. Verify that the plugin receives the same task lease for verification and preparation. 3. Complete a task, fail the provider release once, and reconstruct the cleanup controller. 4. Confirm that cleanup calls only the release hook and retains the completed run outcome. ## What Changed - Acquire the logical run lease before provider workspace verification and reuse it for preparation and execution. - Preserve the existing session and workspace choices. Recompute workspace-dependent inputs only after a definite refusal; acquire credentials and managed runtime inputs once. - Save the exact refusal and lease before cleanup. Use a workspace-scoped creation key only for the resulting fresh attempt. - Replay interrupted verification on the same unlaunched lease. Keep transport errors retryable and reject replay after execution starts. - Always call the provider release hook after task settlement. Pass refreshed settlement proof and route failed releases through the existing cleanup reaper. - Add database regressions for call order, restart, retained writers, refusal, transport errors, and settlement-only cleanup replay. `plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added lines extend the existing workspace transaction and lease callbacks. Keeping the checks there preserves the ownership assertion, provider result validation, and binding update in one operation. Extracting only the verification branch would split that transaction without removing a separate responsibility. ## Verification - Server TypeScript check passed: `tsc -p server/tsconfig.json --noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also passed, including all test-tree layers. - The first CI typecheck caught a readonly array in a test fixture that the narrower source check did not include. Commit |
||
|
|
61ab829d5d |
fix(queue): another agent's comment no longer hides a pending approval (#21303)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person approves a new plan while the agent is busy, the approval waits in the task's message queue. Its Interrupt control stops the current attempt and starts a fresh one with the approved plan > - The task view shows one queue at a time. A saved card response (an approval) gives way to any later queued message, so it never hides the person's own later messages > - Another agent's comment (here, the CEO's direction with `resume: true`) is also a queued message. It took over the view, and the approval and its Interrupt control disappeared > - The person could then not interrupt for the plan they approved > - This pull request lets only the person's own later messages hide a pending approval. A queue that holds only agent comments no longer hides it > - The benefit is that the Interrupt control for an approved plan stays available until the person uses it or writes a newer message ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.interrupt-with-direction`, the person approved a saved simpler plan while Morgan was busy. The approval queued with `requiresFreshSession`, and its Interrupt control showed. The CEO then posted its own direction comment on the task at 04:03:21. That comment queued as a separate deferred wake. `GET /queued-comments` then returned only the CEO comment's queue. The approval's Interrupt control (`task-chat-queued-interrupt-<approval id>`) never rendered, the click timed out, and no interrupt request was sent. The approval was not lost: its deferred wake was still saved. Only the view and its control were hidden. **Steps to reproduce** 1. Approve a plan on a busy task, so the approval waits for a fresh session. 2. Let another agent comment on the task, so its comment queues after the approval. 3. Read `GET /api/issues/:id/queued-comments`. The approval is gone from the queue. **Expected behavior** The approval stays the visible queue with its Interrupt control. A newer message from the person still takes the view, as before. ## What Changed - `server/src/routes/issues.ts`: `findQueuedCommentWake` returns a later message queue only when it holds a message from a person. A queue with only agent comments yields to a pending saved response. With no saved response it is still returned, as before. - `server/src/__tests__/issue-queued-comments-routes.test.ts`: the eval ordering (a fresh-session approval, then another agent's comment) keeps the approval as the queue, and a later person message in that queue still takes the view. ## Verification - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts -t "another agent's comment queues after it"` — fails on master (the CEO comment's queue is returned) and passes here. - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts` (all tests passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - While a saved response waits, queued agent-only comments are not shown in the composer queue. They stay saved and are delivered by the normal deferred-wake path, as they were before the person approved anything. The person's own messages are never hidden. - One indexed comment lookup per queued message wake during the queue read. - No persisted shape or API contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f82ec65800 |
fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can stop an agent run with Stop (cancel). The run must then end as cancelled > - A run that a person stops just as it starts tears down its environment lease at the same moment > - The lease-release teardown wrote `interrupted` (`lease_released_before_terminal`). It recognized only an acknowledged Stop, and it decided from a run snapshot that it read before the Stop request landed > - An interrupted run looks like an infrastructure failure, and it is a recovery candidate. That is against the person's decision > - This pull request makes a recorded person Stop win over the teardown in both orderings > - The benefit is that a person's Stop always ends the run as cancelled ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.stop-versus-pause`, the fixture cancelled a resumed run as it reached native running. The Stop request was recorded at 04:02:11.049. The teardown wrote `interrupted` with `lease_released_before_terminal` at 04:02:11.083. The native cancellation intent was still `pending`, so the teardown did not see it as a Stop. The eval needs `cancelled`. **Steps to reproduce** 1. Start a native run. 2. Record a person Stop request on the run (the cancel route does this first). 3. Let the lease-release teardown run before the cancel finalizer writes `cancelled`. 4. The run ends `interrupted`. **Expected behavior** The run ends `cancelled`. This is true when the Stop is recorded before the teardown reads the run, and when it is recorded between that read and the write. ## What Changed - `server/src/services/acknowledged-native-stop.ts`: `hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize a person Stop that the cancel route recorded (`cancelledByActorType: "user"`, a user id, and a requested or acknowledged execution cancellation or a startup cancellation fence). - `server/src/services/heartbeat/run-lifecycle.ts`: `terminalizeRunOnLeaseRelease` re-reads the run before it decides. A recorded person Stop now gives `cancelled`. The `interrupted` write is fenced on "no person Stop recorded", so a Stop that lands between the read and the write makes the write fail. The function then reads again and writes `cancelled`. - `server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`: two orderings (Stop recorded before the teardown read; Stop recorded after the read, with the stale snapshot) and a control (an agent-only request still reports `interrupted`). ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` — both new ordering tests fail on master (`interrupted`, `lease_released_before_terminal`) and pass here. - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts src/__tests__/heartbeat-run-terminalize-before-release.test.ts src/services/heartbeat/run-lifecycle.test.ts` (67 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Low. Only the lease-release teardown changes, and only when a person Stop is recorded on the run. The cancel path already wrote `cancelled` for the same run when it won the race. - The teardown now makes one more read of the run row. This is one indexed primary-key read per teardown. - Previous-release saved state: no persisted shape changes. Runs saved by the previous release carry the same Stop fields. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0c63f0fdb3 |
test(skills): budget list read bytes for fixture rows, not the bundled catalog (#21285)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The skill library list reconciles the bundled skill catalog, folders, and local sources on each read > - #18023 added a test that limits the `company_skills` bytes one warm `list()` call reads, so the folder pass and the list query keep their narrow projections > - The limit counts every `company_skills` read, so it also counts the bundled catalog rows that every company gets > - Each new bundled skill adds about 9.6 KB to that total, so pull requests that only add skills under `skills/` fail a test about projections > - This pull request measures a catalog-only company next to the fixture company and applies the same limit to the difference > - The benefit is that the test fails only when the fixture rows are read too widely, not when the catalog grows ## Linked Issues or Issue Description Refs #18023 (added the test). Refs #20760, #20764, #20767, #20768, #20769 (failed on this test). **What happened?** `companySkillService.list > keeps folder classification and stored content while narrowing inventory refresh reads` failed on five open pull requests that only add bundled skills under `skills/`. The error was `expected 1163219 to be less than 950000`. Other runs had 983607, 1035956, 982321, and 985561. The test passes on master. **Expected behavior** The test must fail when `list()` reads the unused metadata or the Markdown of company skill rows. It must not fail when the bundled catalog gets larger. **Steps to reproduce** 1. Check out master. 2. Run `git checkout <head of #20760> -- skills` to add its 34 bundled skills. 3. Run `pnpm --filter @paperclipai/server exec vitest run src/__tests__/company-skills-service.test.ts -t "narrowing inventory refresh"`. 4. The test fails with `expected 1163504 to be less than 950000`. **Paperclip version or commit** `9b613562b70b7bea9e7562ddc1087d4c1b1788b1` (master). ## What Changed - The test creates a second company that has only the bundled catalog. - The test lists both companies warm and captures the `company_skills` response-value bytes of each call. - The test applies the existing 950,000-byte limit to the difference. The difference is about 663 KB and comes only from the ten fixture rows. - The test also checks that the fixture company lists the catalog rows plus the ten fixture rows. - No service code changes. ## Verification Where the bytes come from (master, one warm `list()` call): | Read | Master catalog | Catalog of #20760 | | --- | --- | --- | | Main list query (`company-skills.ts:3324`) | 671,003 B, 20 rows | 691,694 B, 52 rows | | Bundled reconciliation `getByKey` reads (one per bundled skill) | 165,423 B, 13 calls | 471,771 B, 45 calls | | Total (old assertion) | 836,465 B, pass | 1,163,504 B, fail | | Fixture company minus catalog-only company (new assertion) | 663,153 B, pass | 663,166 B, pass | Red and green: - Old test, master `skills/` plus the `skills/` tree of #20760: fails, `expected 1163504 to be less than 950000`. - New test, same enlarged catalog: passes, difference 663,166. - New test, master: passes, difference 663,153. The new test still catches the regression it exists for. I widened each projection on purpose with the enlarged catalog, ran the test, and reverted: - Folder pass reads the full `metadata` column instead of `sourceKind` only: fails, `expected 1323122 to be less than 950000`. - Main list query also reads `markdown`: fails, `expected 1183162 to be less than 950000`. Commands: - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/company-skills-service.test.ts`: 81 passed. - `pnpm verify`: not completed. It selected 2,615 of 3,435 test files (conservative selection), and I stopped it to keep load low on a shared machine. CI ran the full required checks on the final commit: 54 passed, 4 skipped. ## Risks - Low risk. The change is in one test only. - The new test runs one more warm `list()` call, about 0.5 s more. - The bound is unchanged. A widened projection on the fixture rows still adds at least 540 KB and fails. - This test no longer limits the bytes the bundled catalog adds to each list call. Each bundled skill adds about 9.6 KB per warm call, because `ensureBundledSkills` calls `getByKey` with the full row for each bundled skill. That cost is real but separate. A fix should narrow that read, not let this test limit the catalog size. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: test-only change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (Superagent Security Scan is neutral: "Scan inconclusive") - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0b9a9593d3 |
fix(tool-gateway): advertise only the MCP capabilities the gateway token may list (#19279)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A named MCP gateway gives a client its assigned tools. Each gateway token has a list of allowed actions > - A native run token allows only `tools/list` and `tools/call` > - `initialize` always advertises the `tools`, `resources` and `prompts` capabilities, whatever the token allows > - A client that follows the advertised capabilities calls `resources/list` and `prompts/list`, and the gateway refuses both with 403 `gateway_token_action_denied` > - This pull request makes `initialize` advertise only the capabilities whose list action the token allows > - The benefit is that a client is not told to call methods that its token cannot use ## Linked Issues or Issue Description This continues #17688, which made the same change and was closed unmerged by its author on 2026-11-02 ("we are no longer contributing upstream"). That pull request reports that Claude Code requests every advertised capability at once, drops the HTTP connection after the first 403, and then has no gateway tools for the run. I did not reproduce the Claude Code part. The gateway part is below. I searched pull requests for "allowedActions tools/list", "gateway token allowed actions context tools", "paperclip_list_resources", "gatewayTokenAllowedActions", "gateway_token_action_denied", "MCP gateway capabilities resources prompts", "initialize capabilities token" and "named gateway token actions discovery", and issues for "paperclip_list_resources" and "gateway token allowedActions". Results: - #17688 (closed, not merged): this change. - #14427 (open): filters the four `paperclip_*` context tools in `tools/list` by the token's allowed actions. It does not change `initialize`, so the two pull requests do not overlap. They touch different handlers in server/src/routes/tool-gateway.ts. No issue matched. **What happened?** On master, the `initialize` handler in server/src/routes/tool-gateway.ts returns `capabilities: { tools: {}, resources: {}, prompts: {} }` for every token. For a token that allows only `tools/list` and `tools/call`, a `resources/list` call returns 403 with reason code `gateway_token_action_denied`. **Expected behavior** `initialize` advertises `tools`, `resources` or `prompts` only when the token allows `tools/list`, `resources/list` or `prompts/list`. A session with no token action list keeps all three. **Steps to reproduce** 1. Create a named gateway and a token with `allowedActions: ["tools/list", "tools/call"]`. 2. Send `initialize` with the token. On master the result advertises `resources` and `prompts`. 3. Send `resources/list` with the same token. The gateway returns 403 `gateway_token_action_denied`. **Paperclip version or commit** master at |
||
|
|
5d298b1b48 |
test(server): judge the Linux real-process boundary cases only by the processes they start (#21284)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In `local_trusted` mode, the operator's local processes get implicit board access. The local board boundary refuses that access to processes of an agent run > - #21209 added a Linux rule for a connection whose owner the server cannot inspect. The server keeps board access only if every same-user process it cannot inspect belongs to the operator > - The real-process test "keeps board access for an operator process the server cannot inspect" used that rule on the whole CI host. Other processes of the same user on the host can be uninspectable and not the operator's > - Since #21209 merged, this test failed in about one in four `PR` runs, on PRs that did not touch it. 22 failed server-test jobs between 2026-11-12 23:05Z and 2026-11-13 03:06Z > - This pull request makes the real-process cases judge only the processes that each case starts. It adds a case that pins the host-wide rule with a known uninspectable agent-run process > - The benefit is that the test is stable on shared CI hosts, and the product rule stays tested ## Linked Issues or Issue Description Refs #21209 **What happened?** `server/src/__tests__/local-agent-process-boundary.test.ts` failed in the "General tests (server)" lane of unrelated PRs. Both cases "keeps board access for an operator process the server cannot inspect" (127.0.0.1 and ::1) received `{ type: "none", source: "none" }`. They expected `{ type: "board", source: "local_implicit" }`. The failures started with the first runs that included #21209 (merged 22:52Z). They continued until at least 03:06Z. Two runner image versions failed in the same way. **Expected behavior** The test passes on a CI host that runs other processes. **Actual behavior** The boundary looks at every same-user process on the host that it cannot inspect. When the owner of a connection is hidden, one uninspectable process that is not clearly the operator's makes the boundary refuse. Examples are a process of another agent run, a process whose parent is PID 1, or a process that exits before the server reads its parent chain. The test did not control which processes were on the host. **Steps to reproduce** On Linux, as a non-root user, run `python3 -c 'import ctypes,time; ctypes.CDLL(None).prctl(4,0,0,0,0); time.sleep(60)' &`. Then make the operator request from the test. The boundary refuses it. ## What Changed - The Linux real-process cases use an inspector that wraps the real Linux inspector. For a hidden owner, it keeps only the candidate processes that descend from the process that the case started. All other parts are the real Linux inspector: the TCP table reads, the fd scan, the recheck, and the ancestry reads. - Each Linux block starts an idle, uninspectable process with `PAPERCLIP_RUN_ID`. This process stands for another agent run on the host. - A new case, "refuses board access to an operator process the server cannot inspect while another agent run's process it cannot inspect is running", uses the unwrapped inspector. It pins the host-wide rule in a real-process test. - If the test runner can inspect every process (for example, as root or with `CAP_SYS_PTRACE`), the idle process is not uninspectable. The host-wide case then skips. The other cases run as before. - No product code changed. ## Verification - Red, Linux (Docker `node:24-bookworm`, non-root user): a script runs the product module `server/src/middleware/local-agent-process-boundary.ts` from master with the same operator request as the test. With no other process, the operator gets board access 10/10. With one idle uninspectable agent-run process, it gets 0/10. With one uninspectable process whose parent is PID 1, it gets 0/10. With short-lived uninspectable processes, it gets 8/10. Agent-run requests are refused 5/5 in all cases. - Red, CI Linux: in this PR's run, the new host-wide case passes. It shows that the earlier assertion fails when one uninspectable agent-run process is on the host. - Green, CI Linux: in the same run, with that process on the host, both "keeps board access for an operator process the server cannot inspect" cases pass. Result: 27 tests passed in the shard that ran the file. - macOS: `pnpm --dir server exec vitest run src/__tests__/local-agent-process-boundary.test.ts` gives 15 passed, 12 skipped (Linux-only). - Not verified: the specific process that was on the CI host in each failed run. One fresh CI run with a temporary diagnostic found no other uninspectable process. That run passed. The diagnostic is not in the final commit. ## Risks - Low risk. The change is test-only. - Product note for a maintainer, not changed here: on Linux, an operator process that the server cannot inspect loses implicit board access when any other uninspectable same-user process on the machine is not clearly the operator's. A process of another agent run is an example. This is the rule from #21209, and the new case pins it. If this is too strict for operators, that needs a product decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Docker on Linux, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: test-only) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
af1c685f5f |
fix(secrets): return 422 for secret proposals missing a required field (#15348)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can propose a company secret or a secret binding through the API. > - The proposal route reads the request body and passes it to the service. > - The service trimmed required string fields before it checked them. > - A request that omitted name or justification threw a TypeError. > - The route then returned 500 instead of a validation error. > - This pull request checks the required fields first and returns 422. > - The benefit is that a client can tell a bad request from a server fault. ## Linked Issues or Issue Description Closes #15334 ## What Changed - Check `name` and `justification` in `createSecret` before `trim`, and return 422 when either is missing. - Check `justification` in `createBinding` the same way. - Keep the existing empty-value and malformed-value checks and the optional `key` field unchanged. - Add a regression test that covers the missing-field cases for both proposal kinds and keeps the complete-body case at 201. No documentation change is needed. The OpenAPI document already lists 422 for this route, so the server now matches the documented contract. ## Verification - Run the route suite. It passes. `./node_modules/.bin/vitest run server/src/__tests__/secret-proposals-routes.test.ts` gives 26 passed. - Revert only the service change and run the new test. It fails with `expected 500 to be 422`, which proves the test catches the bug. Restore the change and it passes again. - Run the server typecheck. It passes. `pnpm --filter @paperclipai/server run typecheck` exits 0 with no errors. - A `POST /api/agents/me/secret-proposals` that omits `justification` returns 422 after the fix and returned 500 before it. A complete body still returns 201. ## Risks - Low risk. The change adds two field checks before an existing `trim` call. Valid requests still return 201. A missing required field now returns 422, which the route already documents. ## Model Used Claude Opus 4.8 (`claude-opus-4-8`), with extended thinking and tool use. I chose the issue, set the evidence bar, directed each step, and reviewed the code, the tests, and the results before this PR. The model traced the route and the service, wrote the fix and the regression test, and ran the checks under that direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> |
||
|
|
25e9a9c2e9 |
feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Parts 2 and 3 of this stack build the Runner alone as an image with two variants, and check it on pull requests. Nothing publishes it > - Downstream sandbox and VM images can copy the Runner only from a registry, and they must be able to verify where it came from > - A published image must be exactly the bytes that were checked, and its name must never point at something else later > - This pull request publishes both variants from `master` as `ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set of inputs, and attests them > - Nothing consumes the image in this pull request. The application image build does not change > - The benefit is one verified, attested Runner that later changes can copy by digest ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image build). No issue exists for this change. **Problem or motivation** The Runner image exists only inside a pull-request check. A consumer cannot pull it. There is no rule for which commit publishes an image, no protection for its tags, and no way to verify a copied Runner. **Proposed solution** Publish from a `master` push in this repository, and from nothing else. Build only when no image has the digest of the commit's inputs. Let the build legs hold no write access. Push the bytes that the legs checked by digest, attest each multi-platform index, and write the tags last. Write the full tag after the slim tag, so that the full tag means "both exist". **Alternatives considered** - Publish from `docker.yml`. That workflow holds one concurrency slot for each ref, and the Runner image needs its own signer identity, so that its attestation cannot stand in for the application image's. - Resolve the lockfile in the workflow before a `master` build, as `docker.yml` does. Then the published Runner depends on the state of the npm registry at run time, and the digest cannot be computed again from a commit. - Hand each variant to the publish job as its own OCI archive. The two archives hold the two shared layers two times: 1.27 GB for each architecture in place of 0.85 GB. - Write the full tag first. Then a consumer can see a full image whose slim image does not exist yet. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 4 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant 4. #21230: publish both variants from `master` (this pull request) ## What Changed - `.github/workflows/runner-image.yml`: adds the trigger `push` to `master` and the job `publish`. - `identity` checks the committed lockfile against the Runner's manifests, computes the digest in the build context, and asks the registry whether these inputs are published. - `build` is the job of part 3. It also records what it built, on every event. On a canonical `master` push it uploads that. It cannot write to the registry on any event. The only token in the job can read packages, and only the lookup step receives it. - `publish` runs only for a `master` push in this repository. It builds nothing and runs no dependency code. - Lockfile rule. A `master` push builds from the committed `pnpm-lock.yaml` and resolves nothing, so the digest is a function of the commit. When the committed lockfile is behind the manifests that the Runner build reads, the run writes a notice, publishes nothing, and stays green. The commit that merges the lockfile refresh publishes the image. Pull requests and manual runs resolve the lockfile first, always build, and never publish. - Hand-off. Each leg joins its two OCI archives into one OCI layout that names the images `slim` and `full`. The layout stores the two shared layers one time. `record` then reads the layout back, each image by its name, and checks every blob against its digest. The leg's record binds the layout to both image digests, the architecture, the inputs digest, and the commit. Those digests are the digests of the archives that part 3 unpacks and checks, so the tree that is checked and the image that is pushed are the same bytes. The publish job accepts only records for its own inputs and commit, and only a layout that still has the recorded bytes. - `record` runs on pull requests too. Only the upload is for a `master` push. A pull request therefore runs the hand-off on real build output. - Publication order. The publish job pushes the four images by digest (`skopeo copy --preserve-digests`). It makes a candidate index for the slim variant, then one for the full variant that names the slim index in the annotation `io.paperclip.runner.slim-digest`. It verifies that each index names exactly the recorded images, attests both, and then writes `inputs-<digest>-slim` and, last, `inputs-<digest>`. - The full tag is the completion marker. Every "is it published" lookup reads the full tag. `publication-state` reports one of: | State | Registry | What a run does | | --- | --- | --- | | `absent` | Neither tag | Builds and publishes | | `incomplete` | The slim tag alone | Builds and publishes. It replaces the slim tag with its own image, then writes the full tag | | `published` | The full tag, and the slim tag at the digest that the full index names | Nothing | | broken | The full tag, and a slim tag that is missing or points at another image | Fails, with the steps for an operator | | `unknown` | No clear answer | Builds. The publish job fails before it writes a final tag | - Recovery rule. A run that stops between the two tags leaves the state `incomplete`. It does not block those inputs. The next run never keeps the earlier slim image: it writes its own slim tag immediately before its full tag, verifies both against its own index digests, and then reads the pair back as every later run will see it. The broken state cannot be repaired by a run, because no run moves a full tag. An operator must delete the two package versions and re-run the workflow run of a `master` push with those inputs. - Pairing rule for consumers, in the contract: resolve the full tag first, take the slim digest from the annotation of the full index, and pull the slim image by that digest. - `verify-index` accepts `linux/arm64` with the variant `v8`, which some BuildKit versions record, and no other variant. - `scripts/runner-image-publish.mjs` (533 lines): the commands `lock-state`, `record`, `verify-legs`, `publication-state` and `verify-index`. It reads archives with the reader of `scripts/runner-image-oci.mjs` (607 lines here), which can now read one named image of a layout. `lock-state` refuses a checkout, because pnpm can rewrite the lockfile of the tree that it checks. The workflow gives it an export that no build reads. - The workflow test pins the build arguments of the image build. Build arguments are not part of the image identity. - `doc/runner-image-contract.md` (197 lines): the section "Publication": the lockfile rule, the order, the pairing rule, the recovery rule, the create-only tags, the internal `candidate-` tags, the command that verifies provenance, and the step to make the package public. `doc/DOCKER.md` says where the image is published. ## Verification Tests, and the build jobs of the workflow on GitHub. Nothing in this pull request ran against a registry, and no publish job has run. - Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd): `runner-image-publish.test.ts` (75), `runner-image-workflow.test.ts` (12), the three other Runner image test files, the provider pack archive tests and the existing Dockerfile tests: 372 tests pass in 10 files. The same files pass on macOS. - The tests of the leg record failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does, and the publish script uses the one reader of part 3. - Leg records: both variants are accepted for this run's inputs and commit; other inputs, another commit, a changed layout, a record without a variant, an archive that lacks a blob, a full image that does not start with the slim image's layers, and a build that is not a publication build are refused. - Publication state: all five states, and the half-finished publication end to end. A slim tag alone lets the next run publish. A full image that is left with another run's slim tag is reported as broken. - Index: exactly the recorded images of one variant; the full index must name its slim index, and the slim index must name none; `v8` is accepted on arm64 only. - Registry lookups: a missing tag only when the registry says so; every refused, failed, or unreachable request is `unknown`. - Workflow: job permissions; the publish job and the upload are behind the canonical `master` push; the build steps receive no credential; the full tag is written last, after a strict lookup and both attestations; one export and a time limit for each build command; the two build arguments; actions pinned by commit; no expression inside a shell script. - `record` on the two OCI archives of a real `linux/arm64` build from part 3: the joined layout is 850,149,376 bytes, and both images read back by name with every blob under its digest. The two archives together are 1,273,646,592 bytes. The image digests in the record are the digests that `unpack` reports for the trees that were checked. - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `docker buildx build --check` reports nothing. The image identity is the same as in part 3 (`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`), because this pull request changes no input of the image. GitHub, at this head: - The `Runner image` workflow passes with these changes: `Build and check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each leg builds both variants, checks them, and writes the leg record. `Publish and attest` is skipped, as it must be on a pull request. Never run: - The publish job. It runs only for a `master` push. - The publish path: the registry lookups with a workflow token, `skopeo copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx imagetools create` with index annotations, and the tag writes. - The attestations, and the `gh attestation verify` command in the document. - `actionlint`. It is not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - **Manual step.** GitHub creates the `paperclip-runner` package private on the first publication. A maintainer must make it public one time. Until then, consumers cannot pull the image without credentials. - The first `master` runs of the workflow are the first test of the publish path. A failure before the tags publishes nothing. A failure between the two tags leaves the `incomplete` state, and the next run with those inputs repairs it. - A slim tag can move while its publication is incomplete. A consumer must resolve the full tag first and take the slim digest from the annotation of the full index. It must never resolve `inputs-<digest>-slim` by tag on its own. The contract says so. - `candidate-` tags are written before the attestations and can name an index that is not attested. They are internal. - The broken state needs an operator, and it fails every `master` run with those inputs until then. It can arise only when someone changes or deletes a tag outside the workflow. - **Known residual.** Two `master` runs that both start before a new digest is published can each build the image. Each build leg and the publish job look again before they work, and the publish job runs one at a time for each digest. The cost is runner time, not correctness. - A commit whose lockfile is behind the Runner's manifests has no published Runner image until the lockfile refresh merges. A lockfile that the refresh cannot fix leaves every `master` run green with only a notice, for as long as it lasts. - The tags are create-only because the workflow never moves a tag of a complete publication. The registry does not enforce that. Consumers must pin the digest and verify the attestation. - `payload.runnerSourceRevision` in a pack from this image is the commit that published the image. Code that reads it as the application's commit must change before it consumes the image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
717aae060a |
fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs execute on environment leases, and `createOwnedSandbox` records each sandbox creation so that a lost reply never creates a second sandbox > - The record key is company, environment, run and request key. The plugin request key is the environment fingerprint, so it is the same for every attempt of one run > - A same-run recovery can need a new environment after the first lease is closed with successful cleanup and a termination receipt > - The old record is still `attached` to the closed lease, so the recovery fails with "Sandbox creation has already transferred ownership" before the provider is called > - This pull request lets the creation fence accept the same stop proof that recovery already requires. A creation record whose sandbox is proven stopped becomes resolved, and a resolved record no longer holds its key > - The benefit is that same-run recovery gets one replacement sandbox, while unconfirmed cleanup still blocks a second sandbox ## Linked Issues or Issue Description Fixes #21214 Refs #20755. That pull request fixes the same failure with a different rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when the recorded lease has ended, and it accepts any ended status, including `failed`, without a termination receipt. A lease whose cleanup is not confirmed can then produce a second sandbox while the first one can still run. This pull request keeps the original key and requires the same stop proof that Core uses elsewhere (`hasRemoteTerminationReceipt`). It does not add a key generation. ## What Changed - `server/src/services/sandbox-creation-operations.ts`: when `createOwnedSandbox` finds the key held by an `attached` operation whose lease is not active, it checks the same stop proof that same-run recovery requires before it acquires: `hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for the sandbox that this operation created (`creationProviderLeaseId`, or the lease's `providerLeaseId`). - With that proof, it runs the same admission transaction one more time. That transaction first marks the old operation `resolved` (`WHERE id AND state = 'attached' AND lease_id`), and then inserts the new operation. - Without that proof, it throws exactly as before. - The check covers every source of a stop receipt: `releaseLease`, the Runner `complete` operation (which writes receipts with direct SQL for reusable leases), stop-and-retain settlement, and the receipt that `resolveRemoteTerminationLease` derives for an expired reusable lease. - A task replacement can leave an earlier operation attached to a lease that now holds another sandbox. That operation is not proven and keeps its key. - Concurrent recoveries serialize on the row update. The second one updates nothing, finds the first one's unresolved operation through the partial index, and replays or refuses as before. There is one provider call. - Migration `0475_sandbox_creation_owner_unresolved.sql` recreates `sandbox_creation_operations_owner_idx` under the same name as a partial unique index (`state <> 'resolved'`). The drop and the create are in one migration transaction. The Drizzle schema matches. - The conflict lookup in `createOwnedSandbox` ignores resolved rows. An active attached lease is still replayed. Pending and `needs_reconciliation` rows still refuse. `releaseLease` does not change. - `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the operation lifecycle. ## Verification - `pnpm --filter @paperclipai/db generate` - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck` - `vitest run src/migration-snapshot-drift.test.ts` (db) - `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48. With the service change reverted, the 5 replacement tests fail. - The 9 other server test files that read `sandbox_creation_operations`, plus `environment-runtime.test.ts`: 564/564 - `pnpm check:migrations` passes New tests in `sandbox-creation-operations.test.ts`: - A stop receipt from `releaseLease` (sandbox driver, and the plugin driver through `acquireRunLease` in the #21214 sequence), from the `complete` operation's direct write on a retained reusable lease, and derived only from the destroyed successor of an expired reusable lease each let the next creation resolve the old operation and create exactly one replacement. A retry replays that replacement. These tests fail before the fix. - Failed cleanup, success without a receipt, a receipt for another lease or another provider task, a retained lease, and pending cleanup each still throw "transferred ownership" with no provider call. - An earlier task's operation that a task replacement left attached keeps its key. - Two concurrent recoveries make one provider call. This test fails before the fix. - Reapplying migration 0418 keeps the partial index. I did not run `pnpm verify` or the full suite locally. CI runs them. ## Risks - Migration: the drop and the rebuild of the index take an exclusive lock on `sandbox_creation_operations`. The table has one small row for each sandbox creation, so the lock is short. - Rolling deploy: the migration is marked `online_compatible`. The previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also matches a partial unique index. Its conflict lookup has no state filter, so it can find a resolved row and throw. That fails closed and never creates a second sandbox. - A legacy runner-task operation without `creationProviderLeaseId` matches any lease it is attached to. Runner-task operations record this field before they attach, so this is only a theoretical gap. - An operation stays `attached` until the next creation with the same key checks it. This is the same as before when no recovery comes. - Stop-and-retain settlement is not tested directly. The shared receipt check covers it. - The fence is unchanged for every case without a stop receipt. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
afe86e9ded |
fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A plugin environment runs each Paperclip Runner turn as a provider task, and a follow-up run can continue the same Claude/ACPX conversation from the saved checkpoint > - Core authorizes that continuation only when the predecessor run's remote execution has stopped (`remoteExecutionHasStopped`), so two Runners never write one checkpoint > - The predecessor's own settlement already proves the stop: the provider reports `executionStopped`, and Core then completes the task > - On a per-turn (ephemeral) lease, Core records the stop receipt only when it releases the lease, and that release can be held for a long time (for example by a pending workspace export) > - In that window, an ordinary follow-up fails with `adapter_failed: Plugin Runner retained session is not authorized`, even though the stop is already proven > - This pull request records the stop receipt when the stop is proven, and the stop check accepts it before the release > - The benefit is that follow-ups continue as soon as the predecessor is proven stopped, and the release gates keep their meaning ## Linked Issues or Issue Description Fixes #21211 Refs #21214 and #21223. Those cover the creation fence for same-run recovery, which still requires a released lease. Production evidence for this change: in both observed failures, the predecessor was a per-turn plugin run. Its `task.settle` completed with all provider calls successful. Core then did not release its lease for 30 minutes or more, and no release call was made in that time. The follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint` → `resolvePluginTaskResume`) within seconds of admission. ## What Changed - `server/src/services/environment-task-runtime.ts` - After a recorded provider stop, `complete` writes the bound stop receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the current lease for every lease policy. Before, it did this only for `reuse_by_environment` leases. Reusable tasks still confirm every owner and still move retained owners to released. - New work on a lease (`submit`, `copilot_metadata`, first portable resume) removes the stop receipt in the same write that clears `runnerTaskExecutionStopped`. A replacement physical task does not copy the old receipt. - `server/src/services/remote-execution-termination.ts` - New `hasBoundRemoteStopReceipt` checks only the receipt binding (company, run, lease, provider and physical task). `hasRemoteTerminationReceipt` uses it and keeps its released-only semantics. Resource cleanup gates do not change. - `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a lease that is released with its receipt, or that is still `active` (not released) with a bound stop receipt. A lease whose release started and did not settle does not count. After release starts, the release owns the receipt. - New `remoteExecutionHasReleased` gives the release-based answer. - `server/src/services/environment-run-orchestrator.ts`: same-run plugin recovery uses `remoteExecutionHasReleased`. The same-run resume check after it requires released receipts, so its behavior does not change. - `resolvePluginTaskResume` keeps its checks and their order. Its refusal now names the first failing check: `Plugin Runner retained session is not authorized (reason: <code>)`, for example `predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The old message stays as the prefix, and no ids go into it. The stale-availability recheck in `environment-task-runtime.ts` names its refusal the same way. Production telemetry for #21211 could not tell which check fired. The next occurrence will show it. - `doc/execution-semantics.md` and `doc/testing-native-remote-continuity.md` describe the stop receipt and when it is recorded. Caller audit for the changed meaning of `remoteExecutionHasStopped`: | Caller | Needs | Decision | | --- | --- | --- | | `agent-instruction-working-copies.ts` | stop | Accepts. Directory cleanup still requires a released "destroyed" receipt. | | `environment-run-orchestrator.ts` | release | Moved to `remoteExecutionHasReleased`. | | `plugin-task-resume.ts` | stop | Accepts. This is the fix. | | `native-runtime/native-runner-ownership.ts` | stop | Accepts. `hasPendingRunCleanup` still counts the active lease as pending. | | `native-runtime/preserved-native-result-finalization.ts` | stop | Accepts. Workspace recovery has its own gate. | | `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts. `hasPendingRunCleanup` still holds active leases. | | `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop acknowledgement and quarantined ownership retirement. Holds and blockers still apply on the normal path. | | `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors still fail it. | ## Verification From `server/`, after `node scripts/agent-preflight.mjs`: - `npx vitest run src/__tests__/environment-task-authority.integration.test.ts`: 10/10 pass. The 2 new tests fail without the fix. - The plugin task and environment runtime suites (`environment-task-authority.integration`, `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `environment-task-runtime`, `plugin-task-source-recovery.integration`, `environment-runtime`): 6 files, 366 pass. - The audited caller suites (`agent-instruction-working-copies`, case handoff legacy and native, `environment-run-orchestrator`, `heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`, `native-remote-retry-admission`, `preserved-native-response-wait`): 9 files, 806 pass. - `pnpm --filter @paperclipai/server typecheck`: pass. The new integration tests cover: - `complete` writes a bound stop receipt on a still-active ephemeral lease. `remoteExecutionHasStopped` is true and `hasRemoteTerminationReceipt` is still false. - `resolvePluginTaskResume` (inspect and launch) authorizes a follow-up's new task, and the real host `read_state` checkpoint read succeeds, while the predecessor's release is held. - Refusals: no provider stop, a receipt for another provider task, an unstopped sibling lease, and a release in progress. - New work on the lease removes the receipt. Release overwrites the receipt with its own receipt. Warm owners keep their behavior. I did not add an executor-level test. `native-session-executor.test.ts` mocks `resolvePluginTaskResume`, so the integration test runs the real checkpoint read path instead. I did not run the full suite locally. CI runs it. ## Risks - `remoteExecutionHasStopped` now returns true earlier for a run whose leases are still held. Each caller is audited above. Callers that need released resources use the release-based predicates. - The stop receipt can now be on an `active` lease. The SQL readers of `remoteExecutionTermination` (cost accounting, execution blockers, idle-work proof, orphan settlement) also require a released status. `pinFailedObservedWorkspace` treats an active ephemeral lease with a receipt as not pinnable. That path applies only to the remote-workspace flow and falls back to `preserveNativeFailedWorkspace`. - Leases that settled before this change have no early receipt. They continue to wait for their release. - Nothing compares the refusal message exactly. Matchers use substrings, and run failure classification does not use the message. - A follow-up still fails closed if the predecessor's stop was never proven. This change does not add a recoverable wait. - The reason the release was held in production is not addressed here. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents, Honeycomb trace queries) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9b613562b7 |
feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Part 2 of this stack builds the Runner alone as the image `runner-artifacts`, in one layer of about 850 MB > - Most of those bytes are provider binaries and Node. They change only with a provider version or the Node pin, but one layer means that every Runner source change makes all 850 MB new > - About half of the image is the pack's transport archives. A consumer that copies the pack as files does not need them > - This pull request splits the image into layers by how often their content changes, and adds a slim variant without the transport archives > - The benefit is that a new image shares about 93% of its bytes with the one before it, and a consumer that needs no transport archives pulls half as much ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No issue exists for this change. **Problem or motivation** The image from part 2 is one layer. A registry and a builder store and move layers, not files. A change to one Runner source file therefore makes a new 850 MB layer, although 790 MB of its content is the same as before. The image also holds the provider pack two times: as files, and as transport archives of those files. Some consumers need only the files. **Proposed solution** Write the image as four layers: the pack's large dependency files, everything else, the transport archives of those large files, and the other transport content. Make the two large layers byte-identical from build to build. Add the target `runner-artifacts-slim`, which is the first two layers. The full image is the slim image plus the two transport layers, as the same layer blobs. Check all of this on the built images. **Alternatives considered** - Two unrelated images. Then the registry stores the shared content two times unless the layers happen to match, and nothing ties the two together. - Split by path only, with all files of 1 MiB or more in the first layer. Six of those files are the Runner's own bundles under `dist`. They change with the Runner's source, so the first layer would change with it too. - Leave the transport archives as they were written. Their bytes then depended on timing (see "What Changed"), and the archive layer never repeated. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 3 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant (this pull request) 4. #21230: publish both variants from `master` ## What Changed - `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY --link` layers. `runner-artifacts` starts from it and adds two more. The build stage sorts its output into the four layers. Every path is where it was in part 2: consumers copy by path and see no difference. | Layer | Content | Changes with | | --- | --- | --- | | 1 | The pack's dependency files of 1 MiB or more: provider binaries and Node | A provider version or the Node pin | | 2 | Everything else, with `runner-image.json` | Every Runner source change | | 3 (full only) | The transport archive of each file in layer 1 | The same as layer 1 | | 4 (full only) | The other transport archives, the overlay archive and the receipt | The same as layer 2 | - Stable bytes. The files of layers 1 and 3 get timestamp 0, and the large dependency files get it before their transport archives are written. A file under the pack's `dist` is the Runner's own output, so it and its archive are in layers 2 and 4. - `scripts/create-provider-pack-archive.mjs`, two separate commits. **This script also writes the application image's archives.** - The same file now gives the same archive every time. The script piped tar into Node's zstd compressor. When the pipe ended while the compressor still had its last chunk queued, Node finished the zstd stream two times, and the archive got an empty frame of nine bytes at its end. That depended on timing. In the pinned Linux base, 37 of 60 runs of the old code had the frame. The compressor is now ended only after it has taken every chunk. - Archive N holds file N of the receipt, and the receipt listed the files in path order. `dist` sorts before `node_modules`, so one more large file in the Runner's own output gave every dependency file another archive name, and layer 3 became a new blob with the same contents. The receipt now lists the files outside `dist` first. The readers (`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair file N with `N.tar.zst` and do not use the path order, so they need no change. For the application image, the archive numbers change one time, and the bytes of each archive do not change. - `runner-image.json` is the same file in both variants. Its new field `fullImageOnly` names the two paths that only the full image has. `paths` did not change. - `scripts/runner-image-layout.mjs check` takes `--variant slim|full`. It is required and has no default. A slim tree that holds a transport path fails. A full tree without them fails. - `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of the image, because the image build does not run it. - `unpack <oci archive> --out <tree>` writes the tree of an image from the layers of its archive, in order. It checks every blob against its digest. It refuses a whiteout, an absolute path, a path with `..`, a path that is written two times, an entry behind a link, and every entry that is not a file, a directory or a link. - `check-layers --slim <oci archive> --full <oci archive>` checks the table above on the built images. It reads the receipt to decide which archive belongs in which layer. It fails when the full image does not start with the slim image's layer blobs in the same order, when a path that is not a directory is in two layers, and when a shared layer holds transport content under any name or directory. - The script reads the archive and its layers itself. It does not use the system's `tar`, so every platform reads them the same way. It finds a member only under the name that buildx gives it. - `.github/workflows/runner-image.yml`: each leg builds each variant one time, as an OCI archive and nothing else. It unpacks both archives, runs `check` on each tree with its variant, runs `check-layers`, and runs the smoke matrix on the slim tree. The tree that is checked is therefore the bytes of the image, not a second export of the same build. Every step that builds has a time limit. The workflow still has no registry access. - Why one export: in the first runs of this pull request, the slim build stalled on all four legs. Each build command had two exports, a tree and an archive. The slim command, with every stage from the cache, wrote its archive and then made no progress on the tree for 50 minutes. A test now holds the rule of one export and a time limit for each build command. - The smoke matrix runs on the slim tree. The daemon is in a layer that both variants share, so one run covers both. - `server/src/__tests__/runner-image-identity.test.ts`: the test that holds the rules for the Runner stages now finds a stage wherever BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with another flag, and a `COPY` that continues on the next line. Nine samples show each case. - Two comments are corrected. The comment on `PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that no check reads the field. The `Dockerfile` comment on the publication argument named a workflow "that builds the image for publication". - `doc/runner-image-contract.md` (159 lines): the two variants, the section "Layers", how the checks get their tree, and the new checks. `doc/DOCKER.md` names the slim target. Script sizes: `runner-image-layout.mjs` 969 lines, `runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166, `create-provider-pack-archive.mjs` 114. ## Verification Image identity at this commit: `a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`. Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd, the worktree mounted read-only): - `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts` (100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts` (8), `provider-pack-archive.test.ts` (47, none skipped), `retained-provider-pack-source.test.ts` (19), `remote-provider-pack-verification.test.ts`, `docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293 tests pass in 9 files. - The same files pass on macOS. The tests of `check-layers` failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does. `linux/arm64`, on a local Docker host, with the new export path: - Each variant builds as an OCI archive only: the full target in 156 seconds without a cache, the slim target in 5 seconds from the cache. - `unpack` writes the full image (10,519 entries that are not directories) in 4 seconds, and the slim image in 4 seconds. - `check <tree> --variant <variant> --expect-inputs-digest <digest> --expect-source-revision <sha> --expect-architecture arm64` passes on each unpacked tree. `runner-image.json` is the same file in both. - `check-layers --slim <slim> --full <full>` passes. - The smoke matrix passes on all five base images, from the unpacked slim tree. | Layer | Digest | Bytes | In | | --- | --- | --- | --- | | 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full | | 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full | | 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full | | 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full | - The slim image is 423,516,545 bytes in two layers. The full image is 850,111,053 bytes in four. Stability, with three builds that share no cache for the build step: | Build | Source revision | Runner build output | Layer 1 | Layer 2 | Layer 3 | Layer 4 | | --- | --- | --- | --- | --- | --- | --- | | A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` | `3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` | | B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` | `4e6c492d…` | | D | `dddd…` | one new source file that adds two large files under `dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` | - Layers 1 and 3 are byte-identical in all three: 790,547,848 of 850,111,053 bytes, 93%. Build D has two more large files under `dist`, eight in place of six, and another image identity. Its dependency files keep the archive numbers 0 to 20. With the earlier numbering, those numbers moved by two, and layer 3 was a new blob with the same contents. - Layer 1 has the same digest as before this round. Layer 3 has a new digest one time, because the archives have new numbers. Other checks: - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and `docker buildx build --check` report nothing. GitHub, at this head: - The `Runner image` workflow passes on both architectures with the new export path: `Build and check (amd64)` in 14m41s and `Build and check (arm64)` in 8m41s. Each leg builds both variants, checks both layouts and the layer rule, and runs the smoke matrix. The first runs of this pull request stalled in the slim export before any check. Never run: - `native-session-executor.test.ts`. It does not load on this host, with or without this change, because `@paperclipai/plugin-sdk` is not built here. It imports the reader of the retained pack and no archive number. - The full test suite and the full typecheck. CI must supply them. ## Risks - The two changes to `create-provider-pack-archive.mjs` change the application image's transport archives: about half of them lose nine bytes, and all of them get new numbers one time. An archive's content does not change. A reader pairs file N of a receipt with archive N of the same build. - The large dependency files in the pack, and in their transport archives, now have the modification time 0 (1970). Before, they had the build time. Nothing in this repository reads those times. - `unpack` is a reader of tar and gzip that this repository owns. It reads the layers that BuildKit, GNU tar and bsdtar write (tested with each), and it fails for a header it does not know. - The check on the built images, not the recipe, is what proves the layer rule; it runs in each leg of the workflow. - `check` now needs `--variant`. A caller that does not pass it fails with a clear message. No caller exists outside this repository yet. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
3b4c3265ed |
fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run fails, the task thread shows a failure marker, and some markers offer Try again > - A `native_provider_model_rejected` failure blocks the task for the person and shows "Model unavailable", but it has no Try again control > - Recovery must just work: after the person fixes the model, they need one obvious control that continues the task, and that control must not start a run that is sure to fail > - This pull request shows Try again on the "Model unavailable" marker and makes the retry recheck the selected model before it starts a run > - The benefit is that the person fixes the model, presses Try again, and the task completes, and a too-early press explains which setting to change instead of failing again ## Linked Issue or Issue Description Refs #21210 **What happened?** Paid run 43321773136, `recovery.retry-exhausted` (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with `native_provider_model_rejected`. The task stayed blocked and the thread showed "Model unavailable", but no Try again control appeared (`task-chat-run-failed-try-again` not found in 30 s). Only the "Run failed" and "Usage limit reached" markers offer Try again. **Expected behavior** The "Model unavailable" marker offers Try again. Pressing it checks the selected model first. If the provider-rejected model is still selected, no run starts, and the person sees which setting to change with a link to it. After the person changes the model, Try again starts one retry of the original run. **Steps to reproduce** 1. Set an agent's model to an id the provider rejects. 2. Assign it a task. The run fails with `native_provider_model_rejected`. 3. Open the task. Before this change, the marker has no Try again. ## What Changed - `server/src/services/native-runtime/model-rejected-retry.ts`: `rejectedModelRetryRefusal` reads the model the failed run sent (`runnerProfileJson.nativeExecutionInput.provider.model`) and the model a new run would use (the task's model override for its assignee, else the agent's model). If they are the same, it returns a 409 refusal with code `native_provider_model_still_unavailable`, the setting to change (`agent_model` or `task_model_override`) and the rejected model. - `server/src/routes/agents.ts`: the exact failed-run retry (`POST /agents/:id/wakeup`, `reason: retry_failed_run`) applies this check before it queues anything. - `ui/src/components/task-chat/adjacent-errors.ts`, `TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now including "Model unavailable". - `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a refused retry shows "Model still unavailable" with the server's message and an "Open <agent> model settings" link to `/agents/<agent>/runtime`. A task override refusal names the override in the message. - Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`): the rejected model stays selected after the failure. The person presses Try again (409, toast names the model, its link opens the agent's model settings, no run starts in 20 s, task stays blocked), restores the working model, presses Try again (202), and the one retry of the original run uses the working model, saves the brief and completes the task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up chat phases are removed; the case now tests the model recheck. `recovery.yaml` context and expected outcome updated, recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its test, descriptors and inventory digest updated. ## Verification - RED on master: `TaskChatThread.test.tsx` "offers Try again on a model-unavailable failure and retries that run" fails (no control). `agent-live-run-routes.test.ts` "refuses a retry while the rejected model is still selected and starts no run" fails (the route queues the run). - GREEN: both pass. `agent-live-run-routes.test.ts` 101/101. - `server/src/services/native-runtime/model-rejected-retry.test.ts` (embedded PostgreSQL), 6/6: refuses while the agent still selects the rejected model (names the agent setting); names the task override; allows the retry when the agent or the override now selects another model, or the run failed for another reason; ignores another company's run. - `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links to `/agents/morgan/runtime`; no link for a task override; other errors fall through to the generic toast. - Server `tsc --noEmit` clean. UI `tsc` clean apart from the pre-existing generated storybook catalog. `node scripts/check-token-gates.mjs`: clean. - Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration rejects an accepted early retry, a run started while the model is rejected, a refusal without the model, setting link or blocked task, a retry that used the old model, and duplicate or missing retries. `pnpm test:e2e:runner:typecheck`: pass. Manifest test 10/10. `source-sdk-public-inputs --check`: `changed: false`. - Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch `chat-eval/qa-mr-paid`. Result to follow in a comment. ## Risks - The recheck compares the selected model with the model the provider rejected. It does not call the provider. A model the provider rejected stays refused until the person selects another one; a different but also unsupported model starts a run that fails with the same marker again. - The refusal message is a toast (15 s). The marker detail still says to choose a supported model. - No persisted shape changes; runs saved by older builds carry the same `runnerProfileJson` field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7154030ef6 |
fix(heartbeat): draining/yielding servers claim no queued run, and closed tasks get no system continuation (#21258)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server drains. It must not start new runs, because its own exit would cut them off. > - The drain gate (run admission closed at SIGTERM) and the hand-over gate (#21127, yield to a replacement that already dispatches) were checked only by the queued-run dispatch passes. > - Other start paths (executeRun of a queued run, delivered continuations, immediate wake starts) went straight to `claimQueuedRun`, so an outgoing server could still start runs after traffic moved. > - Also, a system continuation queued while its task was open started after another run had finished the task, and took an extra turn on a done task. > - This pull request makes `claimQueuedRun`, which every start path uses, honour the drain and hand-over gate, and cancel a system continuation whose task closed while it was queued. > - The benefit is fewer runs cut off by a swap, and no extra turns on finished tasks. ## Linked Issues or Issue Description Refs #21127, #21135 **What happened?** 1. An outgoing server kept starting runs minutes after traffic moved to its replacement. Each such run was cut off when the old server exited. The run start went through `executeRun` of a queued run, which called `claimQueuedRun` without the drain or hand-over check. 2. In release-swap waves, a continuation (execution reconciliation delivery, invariant or recovery continuation, blocker-stall wake) was queued while its task was open. Another run then finished the task. The continuation still started and took an extra turn on a done task. **Expected behavior** A draining or yielding server leaves a queued run queued for the serving server, on every start path. A system continuation does not start on a task that was closed while it was queued. A person's message on a done task still runs. **Steps to reproduce** 1. Close run admission (drain) or set the hand-over yield on a server. 2. Start a queued run through `executeRun` (for example a delivered continuation). 3. See the run claimed and started on the draining server. ## What Changed - `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`) refreshes and checks the run-admission gate first. When admission is closed (draining or yielding), it leaves the run queued and returns. The dispatch passes already did this check; now every start path does. - `claimQueuedRun` re-reads the task for a system continuation (`issue_recovery_action_restored`, `issue_continuation_needed`, `issue_assignment_recovery`, blocker-stall wake). If the task is `done` or `cancelled`, it cancels the run as `issue_closed_before_start`, skips its wake, and releases the task. Other wake reasons, including a person's message, are not affected. - That cancel never consumes a person's message. Admission can attach deferred comments to a queued system continuation without changing its wake reason. The cancel locks the run's wake and the run (the order admission's coalescing uses), reads the run's current comment ids, and if any is a person's comment that no run of this agent has received yet (read-only use of `deliveredQueuedCommentIds`, which includes the history-read rule), the run is not cancelled: it is claimed as that person's follow-up, which may reopen the task through the normal path. A comment attached concurrently is either seen under the lock or, after the cancel commits, saved behind the cancelled run by admission. - Admission is checked again inside each claim transaction (ordinary claim, queued-comment claim, native review claim), right after the write that moves the run to `running`. If admission closed during the awaited checks, the transaction rolls back: the run stays queued with no `startedAt`, its wake and the task's execution lock are untouched, and nothing is charged. - Every cancellation `claimQueuedRun` makes from the caller's copy of the run is now a compare-and-set from `queued` (`setRunStatusFromLive(..., ["queued"])` through a new `cancelRunIfStillQueued` helper): the closed-task cancel, the refused-interaction cancel, the missing-@mention cancel, the daily-cap cancel and the blocked-dependency cancel. A run another dispatcher already started or finished keeps its status and result, and no wake or release follows. The queue module no longer receives the unconditional `setRunStatus`. - The closed-task cancel takes its decision under the wake and run locks, then cancels through `setRunStatusFromLive` fenced on the row version (`xmin`) the decision read. That saves the same never-started proof (`executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`) as every other queued-run cancel, so task release promotes a person's message saved behind it. A message attached after the decision changes the run, the cancel matches nothing and the decision is taken again. `HeartbeatRecoveryDependencies.setRunStatusFromLive` (`recovery.ts`) now types its existing `cancellationCondition` parameter as `SQL`. - The dispatch ranking in `startNextQueuedRunForAgent` is not changed. ### What I checked in `claimQueuedRun` for writes from the caller's copy - Cancellations through `setRunStatus`: all five now conditional on `queued` (above). - The final claims (ordinary, queued-comment, native review) already lock and re-read the run and update `where status = 'queued'`. - The case-handoff cancel and the chat-control stop already lock the run and update `where status = 'queued'`. - The budget-stamp update already writes `where status = run.status` (`queued`). - `cancelRunInternal` (agent removed, agent not invokable, board-owned budget stop, pause hold, budget cancel) re-reads the run and only cancels a non-terminal status, so it cannot overwrite a finished result. It can still stop a run another server has just started; that is a policy stop (agent removed, budget, hold) that would apply to that run anyway, and it is unchanged here. - `cancelStaleQueuedRun` already takes `expectedStatus: "queued"`. ## Verification Red-to-green tests in `server/src/services/heartbeat/queue.test.ts`: - "claims no queued run once run admission is closed, whatever path asks (draining or yielding server)": red before the gate was added to `claimQueuedRun`, green after. - "does not start a %s run whose task another run already finished": `issue_continuation_needed` and `issue_recovery_action_restored` are cancelled as `issue_closed_before_start`; `issue_commented` (a person's message, carrying its `wakeCommentId`) is claimed and the stored run is `running`. - "answers a person's message attached to a system continuation on a closed task once": still undelivered, the continuation is claimed and is the only unfinished run (red when the cancel ignores comment ids, green after); already received by the closing run, the continuation is cancelled as `issue_closed_before_start` and no other run exists. - "keeps the result of a continuation another dispatcher already started and finished": the stale queued copy is passed after the run succeeded; the stored run stays `succeeded` with its result, and no wake or release follows. Red before the compare-and-set (the run was overwritten to `cancelled`), green after. - "returns the claim to the queue when run admission closes while the claim is being checked": admission closes inside an awaited check; the run stays `queued` with no `startedAt`, the task's execution lock and the wake are unchanged. Red before the in-transaction re-check, green after. Red-to-green test through the real release path, in `server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`: - "delivers a person's message saved behind a system continuation cancelled at claim because its task closed": a never-started legacy continuation on a done task, with a person's deferred message behind it. Red before the fix: the cancelled run had `resultJson: null`, with no never-started proof. Green after: the proof is present, the wake is skipped, and the message is promoted and runs once. Test expectations changed because they asserted the old behaviour: - `server/src/services/agent-interaction-delivery.test.ts` "cancels refused queued runs through the public queue claim without dispatch": now expects the cancel through `setRunStatusFromLive(runId, "cancelled", ["queued"], …)` instead of `setRunStatus`. - `server/src/services/heartbeat/queue.test.ts` (refused-assignment held case): asserts `setRunStatusFromLive` is not called, instead of `setRunStatus`. - The `issue_commented` case of "does not start a %s run…" now asserts the run is claimed and `running` (it previously asserted only that the error code was not `issue_closed_before_start`). Local command, on this branch based on current master (includes #21234; the dispatch ranking it changed is untouched here): - `cd server && pnpm exec vitest run src/services/heartbeat/queue.test.ts src/__tests__/recovery-wake-herd.test.ts`: 102 tests passed. After the proof fix, `queue.test.ts` again passed 97/97, and the new stale-queue test passed. (The file's `afterAll` database cleanup hook times out on this loaded machine on master too; it is not related to this change.) Why the local release-swap wave does not show the yield fix: - The "yield to the replacement" signal exists only when the run-start handover is created (`server/src/index.ts`, about line 1320). That happens only when a direct deployment listen URL exists, that is, with `PAPERCLIP_EPHEMERAL_SERVER=true` and `PAPERCLIP_DATABASE_LISTEN_URL` (the volume-free configuration). - The local short form of the wave leaves `--ephemeral-server` out, so the handover is disabled and `isRunAdmissionClosed()` becomes true only at SIGTERM. The wave therefore cannot exercise the yield. - The evidence for this fix is the unit test above and the reasoning that every start path now reaches the same gate in `claimQueuedRun`. Hosted confirmation must come from a volume-free instance. - The closed-task fix was found in release-swap waves (an extra turn on a done task in two tasks). ## Risks - `claimQueuedRun` can now await an admission refresh. The refresh returns at once when the cached state is fresh, so the normal path adds no wait before `turn.start`. - A system continuation for a closed task is cancelled. This matches admission, which already refuses such wakes for a closed task. - No migration. No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d169dd5a90 |
fix(presentation): offer Retry only when no automatic continuation or scheduled retry is pending (#21264, includes #21200)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a run fails with a transient error, Paperclip schedules the next attempt by itself and shows "retrying automatically" with a countdown. > - The task presentation also offered the Retry button next to that countdown. > - The system owns the next attempt while the run is `scheduled_retry`, so the button is misleading: it suggests that a person must act. > - This pull request stops offering Retry while a scheduled retry is pending. Retry returns when the scheduled retry is past due with nothing pending, or when the retries are exhausted. > - The benefit is one clear message: Paperclip retries by itself, and Retry appears only when it is the way forward. ## Linked Issues or Issue Description **What happened?** A legacy run in status `scheduled_retry` (for example reason `transient_failure`, which is not a shared-resource wait and has no system recovery) projected `state: "retry_scheduled"` with `canRetry: true`. The UI then showed the Retry button beside "retrying automatically". The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`: the `retryAt > now` branch set `canRetry: !preparationPending && !nativeResumeAt`, which ignores that the run itself is the scheduled attempt. **Expected behavior** While a run is `scheduled_retry` and its retry time is in the future, the task shows the countdown and offers no Retry. When the scheduled retry is past due with nothing pending, or the retries are exhausted, Retry is offered. **Steps to reproduce** 1. Let a task's run fail with a transient failure that schedules an automatic retry. 2. Open the task before the retry time. 3. The task shows "retrying automatically" and the Retry button. **What a Retry click does today in this state** `requestTaskRetry` (`server/src/services/task-retry.ts`) treats a latest run in `scheduled_retry` as busy. It accepts the click (202), records the request as `coalesced` onto the scheduled run and returns `scheduledRetryRunId`. It does not start a second attempt next to the scheduled one. The button is therefore misleading but not unsafe, and this pull request changes only what the task shows. ## What Changed - `server/src/services/task-interruption-presentation.ts`: in the `retry_scheduled` branch, `canRetry` is now also false when `run.status === "scheduled_retry"`. - `server/src/services/task-interruption-presentation.test.ts`: three tests for a scheduled transient-failure retry. - This branch is based on current `master` and includes #21200, which has not merged yet and changes the same file. The diff of this pull request against #21200 is one line and the tests. ## Verification - `cd server && npx vitest run src/services/task-interruption-presentation.test.ts -t "automatic scheduled retry"` - "shows a pending transient-failure retry as scheduled without a competing Retry": red on `master` (`canRetry: true`), green with this change. - "offers Retry once the scheduled retry is past due with nothing else pending": green before and after. - "offers Retry once the scheduled retries are exhausted": green before and after. - Locally: `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts` and `task-interruption-automatic-path.test.ts` pass (107 tests). Server typecheck: 0 errors. The wider suites that read the presentation run in CI; they were not repeated locally because the machine was under memory pressure. ## Risks - Low risk. The change affects presentation only. The Retry route still accepts a click and coalesces it onto the scheduled run, as before. - If a scheduled retry never dispatches, the task offers Retry again once the retry time passes, as before this change. - No schema, API or UI change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6cb0dd0c6b |
fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is cut off by a restart or a release swap, the fresh-continuation fallback continues the task in a new run. > - The continuation goes through normal admission, or through a person's saved follow-up when one is waiting. > - In release-swap waves, three gaps left tasks stopped with no run: admission refused an isolated continuation of a task with no project, a person's saved follow-up behind the replaced run was never promoted, and the fallback relied on a saved follow-up that the cut-off run had already read. > - This pull request closes those three gaps. > - The benefit is that a twice-interrupted task, and a task with a waiting follow-up, continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. A task with no project was interrupted twice. The second continuation used a fresh isolated workspace, which clears the task's workspace binding. The admission preflight still required that binding for an isolated task with no project, so it refused the continuation as `workspace_worktree_requires_project` about every 15 s. Each refusal re-blocked the task and added a comment (329 to 428 skipped wakes per wave). The runtime itself supports this case: it prepares a new workspace (`create_new`). 2. The fallback continued a task with delivery `saved_input` (a person's deferred follow-up is the next turn). The stranded-queue pass refused to promote that follow-up, because the replaced native run's finalization was `terminal_failure` (restart recovery ended it), not `committed`. The release path would also have re-blocked the task with a reconciliation incident. The follow-up never ran. 3. The fallback chose `saved_input` because a deferred user wake existed, but the cut-off run had already taken that message into its turn. Promotion settled the message as delivered, and no run followed. The task showed "working". **Expected behavior** An interrupted task continues by itself in each case, with no person action and no repeated comments. **Steps to reproduce** 1. Run a release-swap wave with projectless tasks in isolated mode and person follow-ups during runs. 2. Interrupt the same task twice, or interrupt a run that has a deferred follow-up behind it. 3. See the task stay blocked or "working" with no run. ## What Changed - Admission (`server/src/services/execution-workspace-policy.ts`, `server/src/services/heartbeat/queue.ts`, `server/src/services/fresh-continuation-fallback.ts`): `isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The admission preflight sets it from the new `isFreshIsolatedContinuationWake`, which returns at once (no query) unless the wake context carries `freshContinuationIsolation`. This does not touch the dispatch ranking. - Follow-up promotion (`server/src/modules/wake-queue/adapters/postgres.ts`): the stranded-queue pass and the release path skip the native finalization gate and the reconciliation incident for a run that the fallback superseded (`agentRunSupersededByFreshContinuation`). - Saved-input choice (`server/src/services/fresh-continuation-fallback.ts`): a deferred user wake counts as saved input only when it has no comment ids, or still has a message that the agent has not received (`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the fallback records its own continuation. - Review round 1: (a) whenever the old writer is not proven stopped (`fresh_isolated_workspace`), the fallback records its own isolated continuation first, also when the task had a binding; saved input follows it on the new workspace. (b) Release skips the manual-admission hold and the pending-cleanup check for a run the fallback replaced with an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`, via `freshContinuationIsolatesRun`); other replacements still wait. - Review round 2 (the isolated continuation must run before the waiting follow-up): while the fallback's `fresh_isolated_workspace` continuation is still pending delivery, release leaves the person's follow-up deferred. Otherwise the follow-up would become the task's next run without the isolation marker, and delivery would count it as the continuation and skip the isolated run. The continuation's own release promotes the follow-up behind it. `freshContinuationIsolatesRun` now also reports `continuationPending`. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/execution-workspace-policy.test.ts`: admission keeps a fresh isolated continuation of a projectless task runnable. - `server/src/modules/wake-queue/adapters/postgres.test.ts` "restart-ended native run": saved input behind a run the fallback replaced is promoted. - `server/src/services/fresh-continuation-fallback.test.ts` "relies on saved input only while it still holds an undelivered message" (taken: true gives `recorded`, taken: false gives `saved_input`). - `postgres.test.ts` "promotes saved input behind an exhausted restart hold ... (released, continuation pending)": the follow-up stays deferred while the isolated continuation is pending, then is promoted when the continuation's run is released. It was red before the fix (`expected 'promoted' to be 'released'`). - Local, round 2: `cd server && npx vitest run src/modules/wake-queue/adapters/postgres.test.ts src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests passed. Local command, on this branch based on current master (includes #21222 and #21234): - `cd server && npx vitest run src/services/fresh-continuation-fallback.test.ts src/__tests__/execution-workspace-policy.test.ts src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests passed (re-run after rebasing on #21234). Local release-swap cells: - L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered, including a second one on the same task; 0 `workspace_worktree_requires_project` rows (before: 329 to 428 skipped wakes per wave). - R2 (6 tasks, 3 swaps): a task interrupted three times had its second and third continuations delivered. The same cell found gap 3 (a follow-up already read by the cut-off run), which the last commit fixes. - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds these fixes and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lost-continuation and follow-up-promotion fixes were visibly working in it. Review-round tests: "moves the task off the old writer's workspace and runs the isolated continuation before saved input" (changed expectation: `recorded` instead of `saved_input`; red on the previous head); "promotes saved input behind an exhausted restart hold once a %s continuation replaced the run" (isolated: promoted, red before; `writer_ended`: held). 3 files, 224 tests passed. Conflict note: the last commit was resolved onto current master. It no longer imports `taskHistoryReadsForRun`, which belongs to a separate opening-note change. ## Risks - An isolated projectless continuation now passes admission. It runs in a new workspace that the run prepares, so it never shares a workspace with the cut-off run. - A superseded run's follow-up is promoted without its native finalization committed. The superseded run is fenced (its writes are refused), so there is still one writer. - No migration. No schema change. The normal admission path adds no query. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
db802d1230 |
fix(heartbeat): deliver a queued comment again unless the reading run cites it as handled (#21243)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place
## Linked Issue or Issue Description
Refs #21175
Refs #21203
Refs #21210
**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.
**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.
**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.
## What Changed
- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.
## Verification
- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
- read and cited: no continuation (`...cited: true`).
- never read: one continuation, as before.
- Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
|
||
|
|
53608bd1d6 |
fix: expose reconciliation wake reason in diagnostics (#18236)
## Thinking Path > - Paperclip projects wake records into safe task diagnostics. > - The projection keeps a small allowlist of stable, non-sensitive reasons. > - `execution_reconciliation_required` is a stable wake reason, but the projection currently converts it to `other`. > - This hides the actionable reason for a skipped wake. > - This pull request preserves that reason in diagnostics. > - The benefit is that operators can diagnose this wake state without exposing raw error or payload data. ## Linked Issues or Issue Description Refs #13532 Related: #16966 handles the separate future wake recovery path. This draft does not add recovery or retry behavior. ## What Changed - Add `execution_reconciliation_required` to the safe wake-diagnostic reason allowlist. - Add focused projection coverage for the known reason and unknown-reason redaction. - Extend the existing diagnostics-route regression matrix. ## Verification - Passed: `pnpm exec vitest run server/src/routes/issues-wake-diagnostics.test.ts --reporter=verbose` — 1 file, 2 tests. - Added endpoint-level embedded-Postgres coverage in the existing route suite. - The full embedded-Postgres route suite did not complete locally: its tests timed out in this isolated checkout. A direct embedded-Postgres start, query, and cleanup probe succeeded. This PR remains draft until hosted CI provides the broader result. ## Risks Low. The change exposes one existing stable reason. Unknown reasons still project to `other`. It does not expose payloads or raw error text, and it does not change wake admission, recovery, or scheduling. ## Model Used OpenAI Codex — `gpt-5.6-terra`; tool-assisted coding and reasoning. Context-window size was not disclosed by the provider runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR is a targeted bug fix, not a new roadmap feature - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked the existing public issue - [x] I have not referenced internal or instance-local issues or links - [x] My branch name describes the change and contains no instance-derived details - [x] I have run the focused local test and it passes - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: no public contract text changed) - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: DeadlySilent <3342508+DeadlySilent@users.noreply.github.com> |
||
|
|
1c33275ada |
fix(heartbeat): start the new owner after reassignment away from a queued creator run (#21256)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent often creates a task for itself and then reassigns it
to another agent
> - The creator is busy in its own chat run, so the new task's
assignment run stays queued and does not take the task's execution lock
> - #21030 releases the outgoing assignee's never-started run on
reassignment, but only when that run holds the execution lock
> - Here the run did not hold the lock, so it stayed queued. The new
owner's wake then adopted it as the active run and was deferred behind
it
> - When the queue later withheld the stale run as
`issue_assignee_changed`, nothing promoted the deferred wake, and the
stranded-wake sweep ignores assignment wakes. The new owner never ran
> - This pull request closes all three gaps
> - The benefit is that a reassigned task starts its new owner at once,
and recovery still starts it if a release is missed
## Linked Issues or Issue Description
Refs #21030
**What happened?**
In the paid chat evals `onboarding.first-work` and
`questions-and-approvals.duplicate-answer` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
|
||
|
|
57840ac6b2 |
fix(projects): say where a person restores an archived project (#21254)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can archive a project for a person, and the person may
later want it back
> - The project update route did not say how archiving works or where a
person restores a project
> - The agent's API search returned the status-card route, which says to
restore from the company Status page with Show archived
> - The agent then told the person to restore the project from the
Status page, which is wrong
> - This pull request describes project archive and restore on the
project update route, with the real page
> - The benefit is that agents send people to the project's own
Configuration page, where Unarchive is
## Linked Issues or Issue Description
**What happened?**
In the paid chat eval `projects.archive-project`, the agent archived the
project correctly. Its reply said the person could view or restore it
"from the company Status page using Show archived". That is the
status-card flow. Projects are unarchived on the project's Configuration
page.
**Steps to reproduce**
1. Ask an agent to archive a project and keep its history.
2. The agent searches the API with `archive project update project` and
sets `archivedAt`.
3. Read where the reply says to restore the project.
**Expected behavior**
The reply links `/<prefix>/projects/<projectId>/configuration` and says
to use Unarchive there.
## What Changed
- `server/src/routes/openapi.ts`: `PATCH /api/projects/{id}` now says
that archiving sets `archivedAt`, hides the project from the sidebar and
selectors, and keeps the project, its tasks and their history. Restoring
sets `archivedAt` to null. A person unarchives on
`/<prefix>/projects/<projectId>/configuration` with the Unarchive
button, not on the company Status page.
- `server/src/__tests__/project-archive-restore-guidance.test.ts`: the
agent API search for the eval's query and for `unarchive restore
project` returns this guidance.
## Verification
- `cd server && npx vitest run
src/__tests__/project-archive-restore-guidance.test.ts` passes (2
tests). The tests fail on master.
## Risks
- Low risk. Only an agent-facing route description changes.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
444dc18d98 |
fix(routines): keep webhook secrets out of chat and link the reveal page (#21253)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can add a webhook trigger to a routine, and the API
response includes the webhook URL and its secret
> - Paperclip redacts secrets from chat output, so a secret that an
agent writes in a reply reaches the person as `[REDACTED]`
> - An agent pasted the bearer token in chat, the person saw
`Authorization: Bearer [REDACTED]`, and had no way to get the key
> - The person can reveal a key once on the routine's Triggers page, but
the agent did not know that page
> - This pull request tells the agent never to write the secret in chat
and to link the Triggers page instead
> - The benefit is that secrets stay out of chat and the person still
gets a working key
## Linked Issues or Issue Description
**What happened?**
In the paid chat eval `routines.webhook-trigger`, the agent created a
bearer webhook trigger. Its reasoning said the token "is sensitive; but
user needs it. I'll provide it." Output redaction removed the token, so
the reply showed `Authorization: Bearer [REDACTED]` and gave no secure
way to get the key.
**Steps to reproduce**
1. Ask an agent to add a webhook to a routine so a test service can
trigger it.
2. The agent calls `POST /api/routines/{id}/triggers` and reads the
secret in the response.
3. Read the agent's reply.
**Expected behavior**
The agent shares the webhook URL and signing mode, does not write the
secret, and links the routine's Triggers page, where the person reveals
the key once.
## What Changed
- `server/src/routes/openapi.ts`: `POST /api/routines/{id}/triggers` and
`POST /api/routine-triggers/{id}/rotate-secret` now describe the secret
handling. For a signed or bearer trigger, the URL and signing mode can
be shared. With `signingMode: none` the URL alone starts the routine, so
it is treated as a credential too. The secret value, or a header that
contains it, is never written in a chat reply, comment, document or
task. The agent links `/<prefix>/routines/<routineId>/triggers`. For a
trigger still awaiting setup, the link is
`/<prefix>/routines/<routineId>/triggers?triggerSetup=<triggerId>` and
the person chooses Generate new key. Otherwise the person chooses Edit
webhook, then Replace key. These are the same controls the human webhook
setup uses.
- `server/src/__tests__/routine-webhook-secret-guidance.test.ts`: the
agent API search for both operations returns this guidance.
## Verification
- `cd server && npx vitest run
src/__tests__/routine-webhook-secret-guidance.test.ts` passes (2 tests).
The tests fail on master because the operations have no description. The
tests also check the unsigned-URL rule.
## Risks
- Low risk. Only agent-facing route descriptions change. Server-side
redaction is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
500bc694b5 |
fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap runs two servers for a short overlap. The old server drains, and the new server takes over the runs. > - Each server proves that it is alive with a controller-instance lease. Recovery uses that lease to decide whether a run's controller is gone. > - In local swap cells, a healthy old server exited during the overlap, because its lease renewal waited for a connection on the busy main pool. Every run it drove was interrupted. > - When a controller crashed, its runs waited for another server boot, or for a timer that restarted on every look. Tasks stood still for 21 to 28 minutes. > - This pull request renews the lease on its own connection, and lets the live server end a crashed controller's runs within a bounded time. > - The benefit is that a swap does not kill a healthy server, and a crashed server's runs end in about 2 minutes, so their tasks continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. During a two-server release swap (local cells, load 14 to 34), the outgoing server exited with "controller instance lease lost while holding work". Five lease renewals in a row timed out after 10 s. The 10 s timeout included the wait for a connection on the shared main pool, which the traffic switch and the replacement's startup pass kept busy. 2. When the old server was killed in the middle of the overlap, its runs stayed in progress for 21 to 28 minutes. The live server did not re-check a crashed controller's runs: only a new boot did. The unverified-controller wait started when a server first looked at the run, not when the controller left, and the successor's own wait record counted as the dead owner's sign of life. **Expected behavior** A healthy server keeps its lease during a swap. When a controller is gone (its lease lapsed and it is fenced or stopped), the live server ends that controller's runs within a bounded time, without another server boot, so restart recovery continues the tasks. **Steps to reproduce** 1. Run two servers against one database with a release swap and a 200 s overlap. 2. Kill the old server 40 s into the overlap (SIGKILL). 3. See that its runs stay in progress for more than 20 minutes, until another server starts. ## What Changed - Lease renewal on its own connection (`packages/db/src/client.ts`, `packages/db/src/index.ts`, `server/src/services/controller-instance-lease.ts`): - New `controllerInstanceLeaseDb(db)`: one private 1-connection pool (`application_name` `paperclip-controller-instance-lease`), made the same way as the existing `controllerLeaseDb` and closed by `closeLivenessDb`. - The renewal is one statement on that connection, as before. - The connection sends `statement_timeout` = 8 s (below the 10 s client timeout) as a startup parameter (new `DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The server cancels a slow renewal, so it never holds the single connection that the next renewal needs, and there is no extra round trip per renewal. (An earlier revision set it per transaction; that added three round trips to every renewal and broke the idle-server receipt test.) - If a connection pooler refuses the startup parameter ("unsupported startup parameter"), the lease logs a warning and renews on a replacement connection without it. - A crashed controller's runs (`server/src/services/recovery/ownership-registry.ts`, `server/src/index.ts`, `server/src/services/native-runtime/native-restart-recovery.ts`): - The ownership reconciler on the live server hands runs whose controller is proven gone to restart recovery. It does not wait for another server boot. - The unverified-controller wait is bounded from the moment the controller's lease lapsed, not from the first look. - When the controller boot is `stopped` or `fenced`, the run ends at once. - A successor's wait record no longer refreshes the dead owner's sign of life. Safety of ending a run when its controller boot is fenced or stopped: - Only `stopped` (written by the boot's own graceful stop) and `fenced` (a newer epoch fenced it after its lease expired plus 30 s grace) count. `live` and `grace` never count, so a controller that is alive and renewing is never treated as gone. `expired` alone does not count either. - A boot that cannot renew stops itself at its self-fence window (about its lease expiry), and a boot that finds itself fenced exits at once. So when another server can fence it, the old process has already stopped driving runs. The lease change above removes the main cause of healthy boots failing to renew. - If the old process is still alive (partition or stall) and its Runner continues: the old run is fenced. Its agent token is refused for writes (`agent_run_superseded`), its remote workspace epoch is abandoned or the continuation gets a new workspace, and the native coordinator records `replacementDenied`. There are never two writers in one workspace. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/controller-instance-lease.test.ts` "own connection": red (10004 ms timeout while the main pool is held) to green. Full file green (26). - `server/src/__tests__/controller-instance-lease.test.ts` "has the server cancel a renewal blocked on a row lock, and renews on the same connection next time": red without the connection-level timeout (the renewal runs to the 10 s client timeout and stays active), green with it (cancelled at about 8 s, no active lease statement left, the next renewal succeeds in under 2 s on the same backend). - `packages/db/src/client-options.test.ts`: the statement timeout is sent as a startup parameter. - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`: "lapsed longer ago" / "end of the bound" (wait counted from the lease lapse), the live-server re-check of a crashed controller, the fenced/stopped boot ending the run at once, and the successor's wait record. Full file green. Changed existing expectation (it asserted the old wait for a stopped or fenced controller): - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` "keeps foreign local Runner and provider ownership after its controller is stopped/fenced": the disposition is now `blocked` (`controller_process_space_unverified`) instead of `awaiting_evidence`, the run is `failed` with `native_restart_recovery_blocked`, and the finalization is `terminal_failure` with `executorStopEvidence: "unverified"`. The safety checks stay: no local PID probe, no claim, no stamp, no local stop proof, the run keeps its process identity and session checkpoint, and no result is written. Local commands, on this branch based on current master: - `cd server && npx vitest run src/__tests__/controller-instance-lease.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140 tests passed. - `npx vitest run src/__tests__/controller-instance-lease.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 225 passed, 14 skipped (real-process cases that need CI). - `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "releases an ordinary held task Retry"`: passed (idle receipt: one lease statement after the work ends). Local swap cells (two servers, embedded Postgres, real agents, old server SIGKILLed 40 s into a 200 s overlap): - Before: runs of the killed server stayed in progress 21 to 28 minutes. - N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the successor's wait write still moved the timer; fixed by the last commit). - R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4 hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run end 3 min 39 s. - N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min 01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the lease TTL (90 s) plus the fence grace (30 s). - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds this fix and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lease and crashed-controller fixes were visibly working in it. ## Risks - A pooled database URL whose pooler refuses `statement_timeout` as a startup parameter: the first renewal fails over to a connection without it (logged), so the lease keeps working without the server-side cancel. - One extra database connection per server, held while the server runs. It is idle except for one renewal every few seconds. - A run whose controller boot is fenced or stopped now ends at once instead of after the bound. See the safety notes above: only a boot that is not alive can be in these states. - No migration. No schema change. Nothing changes before `turn.start` on the normal path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9025597ce4 |
fix(feedback): keep run ids and dates intact when redacting phone numbers (#21251)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People can export a feedback bundle that holds run records for
review
> - The bundle is redacted before export, and one rule removes phone
numbers
> - The phone rule also matched two all-digit groups inside a run UUID
> - A run id such as `9224d1c8-1dbf-4314-9924-e16fcf3ad1fa` became
`9224d1c8-1dbf-[REDACTED_PHONE]-e16fcf3ad1fa`, so the bundle no longer
named its run
> - This pull request keeps UUIDs and ISO dates out of the phone rule
> - The benefit is that exported ids and dates stay correct and real
phone numbers stay redacted
## Linked Issues or Issue Description
**What happened?**
In the paid chat eval `cli-operations.feedback-bundle`, the exported
bundle's `envelope.targetId` was
`9224d1c8-1dbf-[REDACTED_PHONE]-e16fcf3ad1fa`. The phone rule
`(?<!\w)(?:\+?\d[\d ()-]{7,}\d)(?!\w)` matches two all-digit UUID groups
joined by `-`. About 2 to 3 percent of random UUIDs have that shape. The
same rule also matched a bare date such as `2026-11-12`.
**Steps to reproduce**
1. Call `sanitizeFeedbackText` with text that contains
`9224d1c8-1dbf-4314-9924-e16fcf3ad1fa`.
2. Read the output.
**Expected behavior**
The UUID stays unchanged. A real phone number such as `+1 (415)
555-1234` is still redacted.
## What Changed
- `server/src/services/feedback-redaction.ts`: the phone rule now runs
only on the text between public identifiers. UUIDs and ISO-8601 dates
and timestamps are copied through unchanged. Other rules are unchanged.
- `server/src/__tests__/feedback-redaction.test.ts`: a regression test
with the `-4314-9924-` UUID, an all-digit UUID, a date and a timestamp,
next to a real phone number that must stay redacted.
## Verification
- `cd server && npx vitest run src/__tests__/feedback-redaction.test.ts`
passes (13 tests). The new test fails on master: the UUID and the date
are replaced with `[REDACTED_PHONE]`.
## Risks
- Low risk. A phone number written in the exact shape of a UUID or an
ISO date is no longer redacted. Those shapes are not phone numbers.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ade7e0643f |
fix(recovery): pace recovery continuations and start person-initiated work first (#21234)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution recovery continues tasks that a restart or a lost run left unfinished > - Recent recovery changes restored many held tasks at the same time, and each restored task got a queued run on its agent > - The agent's queue starts runs oldest first, so a person's Retry or chat message waited behind every older recovery run > - On staging, one agent had 131 queued runs, and person Retries waited more than 50 minutes > - This pull request paces how many recovery continuations enter an agent's queue, and it starts person-initiated work before system recovery work > - The benefit is that a burst of recovery cannot starve people, and every recovery run still runs ## Linked Issues or Issue Description Refs #21222, #21139, #21135 **What happened?** Staging deployed #21139 and #21135 at 22:53Z. Within one hour, the company had 201 queued runs and 78 running runs. 122 queued runs had the recovery wake reason `issue_recovery_action_restored`. Each one was on a different task. They all came from the reconciled-execution delivery pass (`source: execution.reconciled`), at a rate of 5 to 15 per minute from 22:57Z to 23:51Z. One agent had 131 queued runs. The oldest queued run had waited since 22:53Z. A person's Retry at 23:14Z was still queued 4 minutes later. Six person Retries on that agent, queued from 23:08Z to 23:14Z, were still queued at 00:04Z, more than 50 minutes behind recovery runs. At 00:04Z, that agent's newest start was a recovery run created at 22:57Z. In the last 400 runs, 25 succeeded. **Expected behavior** A person's Retry, chat message, or assignment starts before system recovery work on the same agent. Recovery work still drains at a steady rate. All recovery runs run eventually, and none is dropped. A task whose continuation waits shows that the task continues automatically. It does not show that the task is idle, and it does not ask for Retry. **Steps to reproduce** 1. Give an agent a concurrency limit of 1. 2. Queue many recovery continuations for the agent (`issue_recovery_action_restored`), for example after a server restart with many held tasks. 3. Click Retry on another task of the same agent. 4. See that the Retry stays queued until all older recovery runs start. **Paperclip version or commit** Staging build 2026.1106.0+2187.git.57aeda190 **Deployment mode** Hosted cloud (staging) **Agent adapter(s) involved** paperclip_runner (native and legacy runtimes). The problem is in the shared run queue, so it applies to all adapters. ## What Changed - New module `server/src/services/heartbeat/queued-run-dispatch-order.ts` decides the order in which an agent's queued runs start. Person-initiated runs start first: `task_retry`, external chat messages, and any wake that a user requested. Other work comes next, together with recovery runs for interruptions that ended in the last 30 minutes. Old-backlog recovery runs come last. Inside each class, the existing order applies: readiness, then priority, then age. Runs whose dependencies are not ready stay last. - Recovery runs get a bounded share. When the agent started 3 other runs after the oldest queued recovery run arrived and none of them was recovery work, the next slot goes to a recovery run. The share applies across the whole batch when one dispatch fills many slots. Thus people cannot starve recovery. - `startNextQueuedRunForAgent` in `server/src/services/heartbeat/queue.ts` uses the new order. It changes only the ranking. It does not change `claimQueuedRun`, the admission preflight, or the promoter. - `deliverReconciledExecutions` in `server/src/services/execution-recovery-resolution.ts` paces delivery. It leaves a continuation pending when the agent already has a queued recovery run, or when the agent's queue is at or above its concurrency limit. One pass delivers at most 5 continuations per company and at most 1 per agent. The pass reads pending continuations in pages and leaves out agents and companies that it already handled, so continuations for busy agents cannot hide continuations for other agents. A continuation that has waited 5 minutes or more is admitted even when ordinary work keeps the queue full, and it goes ahead of fresher continuations, longest wait first. Otherwise the most recent interruption goes first. The existing 15-second continuation schedule delivers the remaining items. This schedule runs only while pending continuations exist, so an idle server does not poll. - The assigned-work sweep (`server/src/services/recovery/assigned-work-sweep.ts`) counts a pending continuation as a pending wake for its task. Thus the invariant does not see a held-back task as stranded, and it does not enqueue a second wake. ## Verification - `pnpm exec vitest run src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 17 tests pass. - Red/green: with the source changes reverted to master, all tests in `recovery-wake-herd.test.ts` fail. Each review fix also has a test that fails on the head before that fix. With this change, they pass. - `recovery-wake-herd.test.ts` covers these cases with embedded Postgres and the real heartbeat queue: - Six recovery wakes are queued, then a person's request. The first dispatch starts the person's run. All six recovery runs start later. - Four pending continuations for one agent: one pass delivers one. The others stay pending and are delivered on later passes. The continuation schedule stays armed while items are pending and returns null after all items are delivered. - A continuation stays pending while the agent's queue is at its concurrency limit. - The per-company cap applies, and the most recent interruption is delivered first. - A continuation that has waited past the bound is admitted while ordinary work keeps the queue full. - New interruptions arrive for the same agent on every 15-second pass, and the older continuation is still admitted within the bound. - Continuations for four busy agents (32 rows, more than one page) do not hide an idle agent's continuation. - A held-back continuation counts as a `queued_wake` pending intent for the assigned-work invariant, and the task shows `working` with `canRetry: false`. - `queued-run-dispatch-order.test.ts` simulates dispatch: a Retry starts ahead of 40 recovery wakes, and all 40 still run. With a person Retry arriving at every step, every window of 4 starts includes 1 recovery run. A fresh interruption's continuation is not put behind old-backlog recovery runs. When one dispatch fills 8 slots, the order is PPPRPPPR (P is a person run, R is a recovery run). - Related suites pass: fresh-continuation fallback, issue recovery actions, completed-turn cleanup hold, issue monitor scheduler, workspace recovery fallback, assigned-work invariant, execution control work, execution recovery resolution, and the heartbeat queue. - `node scripts/typecheck-layers.mjs` (in `server/`) passes. ## Risks - Behavior change: when a person's run waits behind recovery runs, it now starts first. Recovery runs start a little later, but they still get at least 1 of every 4 starts while people send work. - Delivery of continuations is slower during a burst: at most 1 per agent and 5 per company every 15 seconds. This is intentional. A continuation that waits shows as continuing automatically, and the invariant does not repair it twice. - Ranking adds up to three small indexed reads when an agent has more than one queued run. The reads run only when the queue has recovery runs or wake requests to classify. - No schema or migration changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, and read-only staging API requests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5d3fa712aa |
feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Today the application image build compiles the Runner. Other images that need the Runner build it again, or pull the whole application image to copy it out > - Each of those builds can produce a slightly different Runner, and each one costs time > - The Runner should be built one time for each change to its inputs, and every image should copy the same bytes > - This pull request adds the build target `runner-artifacts`, a contract for its layout, and a workflow that builds and checks it. It publishes nothing > - Nothing consumes the target in this pull request. The application image build does not change > - The benefit is one checked Runner build that later changes can publish and copy by digest ## Linked Issues or Issue Description Refs #21096 (the merge-queue image build) and #21167 (provider pack layers). Those are earlier work on the same build. No issue exists for this change. **Problem or motivation** The Runner is compiled inside the application image build. The daemon comes from the `runner-build` stage. The provider pack is built on top of the full server and UI build. Downstream sandbox and VM images need the same Runner. They rebuild it from source, or they pull the application image and copy paths out of it. There is no contract for those paths and no single identity for a Runner build. **Proposed solution** Build the Runner alone, from its own inputs, as a plain file tree. Give each build an identity: a digest of its inputs. Document the layout as a contract, and check every built tree against it. Keep every check that proves the Runner is good in this one build. **Alternatives considered** - Keep the current builds and only document the paths in the application image. This does not remove the duplicate builds, and consumers still need the full application image. - Build the Runner in `docker.yml`. That workflow has no pull-request trigger, so a change to the Runner build would have no check before it merges. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4. Each part is on top of the one before it: 1. `runner/pack-inputs-digest`: an optional field in the provider pack manifest 2. `runner/runner-image-build` (#21204): build and check the Runner image, with no publication (this pull request) 3. `runner/runner-image-layers`: layers by change rate, and the slim variant 4. `runner/runner-image-publish`: publish both variants from `master` This pull request had one larger form before. Publication, the image layers and the second variant are now parts 3 and 4. ## What Changed - `Dockerfile`: eight new stages that end in the target `runner-artifacts` (`FROM scratch`). They install with a filtered, frozen lockfile from a context that holds only the Runner's inputs. The base is a Node image on Debian bookworm, pinned by digest. No existing stage changed. `cloud` is still the default target. - The tree holds: the stripped daemon, the Runner package's complete `dist` with the build stamp that its own build records, the provider pack with its transport archives, the Grok provisioning inputs, the transport contract source and a bundled transport module, the lockfile, and `runner-image.json`. - `scripts/runner-image-identity.mjs`: computes the image identity. It hashes every file under the paths that the build copies from the build context (path, executable bit, content; symbolic links by target), the whole `Dockerfile`, and the two platforms. The build runs it on the files that the build context supplies after `.dockerignore`. - `scripts/runner-image-layout.mjs`: the layout contract. `write` assembles the tree in the build. `check` verifies a built tree: every documented path, the build stamp, every pack artifact and tree digest in `provider-pack.json`, and every transport archive. It also extracts the transport archives the way a sandbox does and compares the result with the pack. No checked path may resolve outside the tree. - Publication form. The build argument `PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it does the pack manifest carry `payload.runnerInputsDigest` (part 1), and `runner-image.json` records `publication: true`. Another image build that runs the stage inline builds the same Runner, but its pack names no image. `check` requires the field in a publication build and refuses it in any other. `--expect-inputs-digest` passes only for a publication build. - `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd --build-metadata` on each base image that a consumer uses. Each base is pinned by digest. - `.github/workflows/runner-image.yml`: two jobs, on pull requests that touch an input and on manual dispatch. `identity` resolves the lockfile and computes the digest in the build context. `build` builds each architecture on a native runner, checks the layout, and runs the smoke matrix. The workflow has `contents: read` only. It holds no credential, reads no registry, and publishes nothing. It builds the publication form, so the check can bind the tree to the digest that the identity job resolved. The tree stays on the runner. - `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`: the tree digest function moves here from `build-provider-pack.mjs` without a change in behavior, so the layout check can use it. A test pins its value. - `.github/workflows/pr-merge-review.yml`: lists the new workflow, as `doc/PR-MERGE-REVIEW-WEBHOOK.md` requires. - `doc/runner-image-contract.md` (127 lines): the contract for consumers. `doc/DOCKER.md` links to it. Changes from the earlier form of this pull request: - The identity hashes the whole `Dockerfile`. The script no longer parses stages, and it no longer refuses recipes. `scripts/runner-image-identity.mjs` went from 397 lines to 166. A short test keeps the three rules that make the hash enough: the Runner stages start only from pinned images, only `runner-artifacts-context` reads the build context, and it copies only the paths that the identity hashes. - The contract document went from 384 lines to 127. - The workflow tests keep only the security properties: job permissions, no credential and no registry access, triggers, actions pinned by commit, and no expression inside a shell script. They no longer pin step names, order, or wording. ## Verification Verified on `linux/arm64` only, on a local Docker host: - `docker buildx build --target runner-artifacts --platform linux/arm64 --build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .` builds in 166 seconds without a cache. The image is one layer of 850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files. - `node scripts/runner-image-layout.mjs check <tree> --expect-inputs-digest <digest> --expect-source-revision <sha> --expect-architecture arm64` passes on the exported tree. - `docker buildx build --file docker/runner-image/Dockerfile.smoke --build-context runner-artifacts=<tree> docker/runner-image` passes on all five base images. - The digest from `--target runner-artifacts-identity` equals the digest in the built `runner-image.json`: `d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`. - The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication argument builds. Its pack has `runnerSourceRevision` and no `runnerInputsDigest`. `check` passes on it, and `check --expect-inputs-digest` fails with "this is not a publication build". - An edit to `ui/package.json` and to `server/src` leaves the digest unchanged. A comment added to an application stage of the `Dockerfile` changes it. - The two points above ran one commit state earlier. The only difference to this commit is one type annotation in a comment of `scripts/runner-image-layout.mjs`. - A named build context replaces the `runner-artifacts` stage, and no Runner stage runs. A build with the application build's view of the context accepts the tree's `dist` and build stamp as current (`assessRunnerBuild`). - The daemon has the same bytes in the five cold builds of this stack that were compared. - Tests: `cd server && node node_modules/vitest/vitest.mjs run` on `runner-image-identity.test.ts`, `runner-image-layout.test.ts` and `runner-image-workflow.test.ts` (132 tests), with `remote-provider-pack-verification.test.ts` and the existing Dockerfile tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33 pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and `docker buildx build --check` report nothing. Never run: - The `linux/amd64` build. It is the leg that downloads the Cursor and Copilot assets for the pack. - The workflow. No job has run on GitHub. - The transport archive round trip with GNU tar outside the build. The build runs it with GNU tar. The local host ran it with bsdtar. - `actionlint` and `hadolint`. They are not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - Low risk to the application image. The change adds stages and files. No existing target reaches the new stages. - The first run of the workflow is the first test of the amd64 build. This pull request triggers it. - The `Dockerfile` is one file for all images. Any edit to it, also to an application stage, changes the Runner image identity. That is the price of not parsing it. - The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag, not a digest. A new frontend release under that tag does not change the identity. - The image does not run the drift checks of the Runner's `build` script. They need files outside the Runner's inputs, and repository CI runs them. A test fails when the `build` script gains a step. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
2d1c2db64a |
feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected apps through MCP. Paperclip already hosts MCP Apps views and speaks stateless MCP 2026-07-28. > - The MCP Skills extension (SEP-2640, Final) lets an app publish skills (instructions and files) next to its tools. Hugging Face already serves 25. > - Paperclip did not read them, so agents missed each app's own workflow guidance. > - Skills are a prompt-injection surface. The spec makes manifest binding, verification, origin tagging and approval revocation host requirements. > - This pull request makes Paperclip a Skills host: discover, review, verify, and deliver app skills to the agents that use the app. > - The benefit is that connected apps can teach agents their workflows, under board control. ## Linked Issues or Issue Description **Problem** Connected MCP servers can publish skills with the `io.modelcontextprotocol/skills` extension. Paperclip ignored them. A board could not see, review or approve them, and agents never received them. **Proposed solution** Implement the host side of SEP-2640. List skills during catalog refresh, record them per connection and company, bind board approval to the full file manifest, and load approved skills into agent runs with verified, lazily fetched content. **Why it matters** Apps ship workflow guidance with their tools. Without the extension, each team must copy that guidance into company skills by hand, and it goes stale. Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work this builds on). ## What Changed - **Client capability.** Every stateless request declares `io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The capabilities that `server/discover` advertises are remembered beside the protocol era. - **Discovery.** Catalog refresh calls `skills/list` (paged) only when the server declared the extension. A listing inside its `ttlMs` is reused. A skills failure never fails the tool refresh. Nothing runs while a run starts. - **Schema.** New `tool_mcp_skills` (company, connection, skill URI; frontmatter, sorted manifest, manifest digest, review state) and `tool_mcp_skill_files` (verified bytes keyed by connection, URI and digest). Both cascade with the connection. - **Governance.** New skills wait for review. Approval binds to the order-free digest of every `{uri, digest, size}`; a request for any other digest gets 409. A changed manifest or frontmatter revokes approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized and invalid skills are declined with a reason. Approve and decline use the company skill policy (`skills.install`, `external_package`, `mcp-connection:<id>`). - **Run time.** Approved skills of connections the agent can use load in parallel with other skill preparation, inside a 3 s budget. `SKILL.md` is read with the agent's grant and checked for size, SHA-256 and frontmatter. A failure calls `skills/get` and takes the skill out of use. Each skill becomes a per-run bundle on the connector skill path, named in its app's namespace, tagged with its origin. Paperclip writes the frontmatter: `allowed-tools` only if the board approved it. - **Supporting files.** `GET /api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed files to an agent in an active run, through the skill's own connection. The sandbox bridge allows the route. - **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…` (list, read instructions, approve, decline). The Skills page shows "Skills from connected apps" with app, state, size, requested tools and plain-language notes. It renders nothing when no app publishes skills. - **Docs.** `doc/connections/MCP-SKILLS.md`, linked from `doc/connections/README.md`. - **Shared frontmatter parser.** It now decodes YAML single-quoted scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and would fail the frontmatter check; now all 25 match. The generated runner copy is regenerated. File size notes: `server/src/services/tool-gateway.ts` grows by about 200 lines. The run-time reads must use its grant resolution and `callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting the gateway is a separate refactor. `CompanySkills.tsx` and `CompanySkills.production.tsx` gain 2–4 lines each; the section lives in its own component. ## Verification - `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake stateless skills server and embedded Postgres. It covers: discovery only after the server declares the extension; the capability in `_meta`; `ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md` at load; supporting files on read); a stale-digest approval refused; a reorder that keeps approval; a changed or added file that revokes it; dynamic skills declined; a same-size tampered `SKILL.md` rejected, refreshed with `skills/get`, and taken out of use; mismatched frontmatter rejected; reads outside the manifest refused; the same URI and name from two servers kept apart; company scoping; a slow app skipped within the budget and cached later; HTTP authorization (board only, agent needs an active run, run header must match) and activity logging. - Red/green: with the digest check disabled, the tamper test fails; with the manifest comparison disabled, the revocation test fails. Both pass with the code. - `server/src/__tests__/mcp-protocol-versions.test.ts`, `mcp-apps.test.ts`, `connection-instructions.test.ts`, `tool-gateway-discovery-http.test.ts`: 47 pass (two capability assertions updated for the new extension). - `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar case; the runner contract test passes (23 tests). - `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and `ui/src/pages/CompanySkills.test.tsx` (44) pass. `packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89) passes. - `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes except the known missing generated Storybook catalog files. - `pnpm verify` returned exit 2 (doc change is a global CI input); required CI supplies broad coverage. - Live check against `https://huggingface.co/mcp` without credentials, through Paperclip's own services and the real egress guard: 25 skills discovered as pending; three approved; cold load 135 ms, cached load 13 ms; a supporting file read through the manifest; the bundle tagged with its origin. - Storybook: `Agents and reusable capabilities/Skills and Skill Studio/Connected app skills` (needs review, all states, tool request, name collision, reading instructions). ## Risks - **Migration:** two new tables only. No changes to existing tables. Safe with the previous release running. - **Run start:** companies without an approved app skill pay one indexed read. With approved skills the wait is capped at 3 s; late or failing skills are left out with a `skills.mcp_unavailable` run warning and never fail the run. - **Agent harness enforcement:** Paperclip cannot gate each code-execution tool call an agent makes while following an app's skill. Approval is explicit and per skill, the approval note says agents may run commands the skill describes, and `allowed-tools` is never granted without a separate tick. - **Shared-home adapters** receive app skills like company skills. A revoked skill leaves on the next skill sync. - Not in this PR: per-connection opt-in for dynamic skills; serving Paperclip's own skills from its public MCP endpoint; `resources/directory/read`; app skills on an agent's skills tab. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs, live HTTP checks). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
826d41e646 |
feat(ui): redesign Settings → Secrets around what people manage (#21225)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents need API keys and tokens. People manage them in Settings →
Secrets.
> - The page showed every credential in one table. Tokens that Paperclip
saved for connections, chat channels and model sign-ins sat beside the
few secrets a person added, many with record IDs as names.
> - Each row repeated an always-true status, a version and a key. New
secret opened on two stacked mode switches and two storage pickers.
> - A person could not see which agents use a secret without opening it,
and had to open it again after creation to give an agent access.
> - This pull request redesigns the page, the New secret dialog, the
details panel, the Vaults tab, the import dialog, the Proposals tab and
My secrets. It keeps the existing data and API calls.
> - The benefit is a list that shows what a person manages, a create
flow with two fields, and a secret that an agent can use as soon as it
is saved.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The Settings → Secrets page, its New secret dialog, the secret details
panel, the Vaults tab, the import-from-vault dialog, the Proposals tab
and the My secrets tab.
**Subsystem affected**
UI: `ui/src/pages/Secrets.tsx` and `ui/src/pages/secrets/*`.
**Current behavior**
- One table lists all secrets. On a busy organization most rows are
credentials that Paperclip saved itself, and many have record IDs as
names.
- Each row shows Active, a version, a chip and the key. The row does not
show which agents use the secret.
- A three-line help banner is always visible.
- New secret starts with two mode switches (Organization / Each user,
Managed value / External reference), then a provider picker and a vault
picker. The Create button is disabled without a reason.
- The details panel has Details, Usage and Access events tabs.
- The Vaults tab has one section per provider, with empty placeholder
boxes and a status picker in the form.
**Proposed behavior**
- The list shows the secrets that the team added. Each row shows who
provides the value (Organization, or Each user with a count) and which
agents use it.
- Credentials that Paperclip saved collapse into "Managed by Paperclip".
They are grouped by origin and have readable names.
- One Filter menu covers provided-by, storage and status. Active filters
show as chips. Archived secrets stay hidden until a person asks for
them.
- New secret has a name, one explicit Organization / Each user choice, a
masked value, an optional list of agents, and "Stored in" only when more
than one storage place exists. Create stays enabled and says what is
missing. A `NAME=value` pasted into Name fills both fields.
- The details panel is one scrolling view: agents, other places the
secret is set, details, and recent activity.
- The Vaults tab is one flat list with health and secret counts. The
connect dialog has two required fields. Paperclip checks a new vault
immediately after it is saved.
**Reason and benefit**
A person can find their own secrets, see if a secret is used, and create
a secret that an agent can use in one step.
**Breaking changes**
None. The API gains one optional query parameter (`order` on the board
proposals list). The default order is unchanged. See Risks for the UI
behaviors that changed.
## What Changed
- **List.** Rewrote the Secrets tab in `Secrets.tsx`. Added
`secret-list-parts.tsx` for the Provided by and Used by cells, the help
popover and the managed groups. The Folders / Flat toggle is removed.
Folders are always on and search covers all folders.
- **Managed grouping.** Added `secret-origin.ts`. It infers the origin
of a secret from the name and key conventions of the server modules that
create secrets (`tool_app.*`, `tool_oauth.*`, `slack_manager.*`,
`chat.*`, `ai_*`, `routine-*`, `agent-observability.*`). It is used for
presentation only.
- **Agent usage.** Added `agent-secret-access.ts`. It holds the grant
and revoke logic that was inside the details panel, and builds one usage
index from the agents list for all rows.
- **New secret.** Rewrote the dialog. Added `buildCreateVaultOptions`
and `parseEnvAssignment`. Agents selected in the dialog get an
environment variable binding after the secret is saved. A failed grant
shows an error and keeps the secret.
- **Details.** Replaced the tabs with sections. "Also set in" lists the
bindings that are not agents. Activity shows agent names and the last
five entries, with "Show all activity".
- **Vaults.** Flat list with a built-in Paperclip row, a Default badge
that follows `getSelectableProviderConfig`, Check again, Enable, and an
Add vault menu. The dialog has Name and AWS region, with naming and tags
in a disclosure. The Status picker is removed.
- **Import, Proposals, My secrets.** Restyled to the same design.
Proposals adds a Recently decided list. The board list endpoint `GET
/api/companies/{companyId}/secret-proposals` takes a new optional
`order=resolved` to sort by decision time.
- **Copy.** "Provider vault" is "vault". "Local encrypted" is
"Paperclip" or "Paperclip-encrypted". Toasts and block reasons use plain
words.
- **Stories.** `secrets.stories.tsx` has new stories for the managed
section, a folder, New secret, the details panel and Vaults. The
Storybook usage fixture now returns the `target` field that the API
returns.
- **Tests.** Updated the page, import, proposals and My secrets tests.
Added `secret-origin.test.ts`. Updated the runner E2E vault-reference
journey and the Storybook visual render-state copy for the new labels.
- **Docs.** `Secrets → Provider vaults` is now `Secrets → Vaults` in
three documents.
`Secrets.tsx` was 4,855 lines and is 4,951 lines. The redesign edits the
page in place to keep its queries, mutations and deep links. Splitting
the file is a separate change.
## Verification
- `cd ui && node ../scripts/run-tsc.mjs -b`: no errors.
- `cd ui && pnpm vitest run src/pages/secrets src/pages/Secrets.test.ts
src/pages/Secrets.render.test.tsx
src/components/AgentSecretAccessEditor.test.tsx
src/components/AgentConfigForm src/components/IssueThreadInteractionCard
storybook/map-tooling.test.ts`: 14 files, 467 tests passed.
- `cd server && pnpm vitest run
src/__tests__/secret-proposals-routes.test.ts -t "decision time"`:
passed.
- `pnpm vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog-domains/external-vault-reference
chat-catalog-domains/external-personal-secret-menu
chat-catalog-domains/external-personal-secret-entry-dialog
chat-catalog-domains/external-personal-secret.test`: 32 passed, 2
skipped.
- `node scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`,
`node scripts/check-module-boundaries.mjs`, `node
scripts/storybook-catalog.mjs`: clean.
- `pnpm verify`: exit 2. It asks for broader coverage because a runner
E2E source file changed. Required CI supplies that coverage.
- All 22 Secrets stories render and their play functions pass in
headless Chromium against a local Storybook.
- End to end, on an isolated local instance (`paperclipai worktree init
--empty`, one company, two agents), with a headless Chromium script. 16
of 16 steps passed, with no page errors:
- Empty state. Create explains missing fields.
- Create an organization secret and give an agent access. The agent's
saved config has the `secret_ref` binding.
- Paste `NAME=value`. Create an each-user secret. Add a value in My
secrets. The list shows "1 of 1 added".
- Create a folder, create a secret in it, search across folders.
- In details: add an agent, replace the value, remove the agent. The
agent's saved config changes each time.
- Disable, archive, and the Archived filter.
- Managed secrets leave the team list and show under their groups.
- Add a Paperclip-encrypted vault and an AWS vault. New secret then
shows "Stored in".
- The import dialog explains a denied vault. The Proposals tab shows its
empty state. The list at 390 px width.
- Not run: the paid runner E2E journeys. I updated
`external-vault-reference.ts` for the new labels and could not execute
it. A live AWS import, and a proposal from a real agent run, were not
exercised.
## Risks
- **Origin grouping is a heuristic.** It reads naming conventions that
live in server modules. If a module changes its naming, its secrets show
in the team list again. Nothing is hidden or blocked by it. A recorded
origin on each secret is the durable fix.
- **Removed UI.** The Folders / Flat toggle, the "View references"
dialog, the vault Status picker, and creating draft GCP or HashiCorp
vault entries from the Add vault menu. Existing draft vaults still show
and can be edited. A disabled vault now has an Enable action.
- **Default status filter.** The list now shows active and disabled
secrets. Before, it showed active secrets only.
- **Built-in storage in "Stored in".** A provider's built-in storage is
not offered once that provider has a usable vault. The form already
replaced that choice with the vault before this change.
- **Agent access at creation.** Access is a second request for each
agent after the secret is saved. If one fails, the secret exists without
that access and an error says so.
- **Automation contracts.** Labels that agents and the runner E2E
journeys depend on are kept: the key in each My secrets row, "Set
value", "Set my value", "Update my value", "Save value", and the import
dialog test IDs. The import entry point moved into the "More secret
actions" menu and the journey is updated for it.
- **Value masking** uses `-webkit-text-security`. A browser without it
shows the value as typed.
## Model Used
- Claude Fable 5.2 (`claude-fable-5-2`) through Claude Code, with tool
use, code execution and browser automation. Three delegated agents on
the same model restyled the import dialog, the Proposals tab and My
secrets.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
2de5d97958 |
fix(startup): report ready without waiting for backlog first passes (#21247)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A hosted instance upgrades by starting a new server beside the old
server. The deployer sends traffic to the new server when `GET
/api/ready` returns 200.
> - `/api/ready` returns 200 only after the boot sequence in
`server/src/index.ts` sets the phase to "ready".
> - The boot sequence awaited the first pass of many delivery queues.
Each pass grows with saved backlog or waits on outside services.
> - On a busy instance, these passes took more than the deploy's
health-check window. The deployer killed two new servers in a row before
they became ready, including a rollback to an image that had started in
86 s one hour before.
> - This pull request moves those first passes off the readiness path.
Readiness now waits only on named, bounded preconditions, and each step
reports its duration.
> - The benefit: an upgrade or rollback becomes ready in boot time, not
in backlog time. A slow start also names its slow step in one log line.
## Linked Issues or Issue Description
**What happened?**
On the hosted staging instance, two replacement servers did not report
`/api/ready` and were killed after about 9 minutes. The first was a new
image and the second was a rollback to the old image. At that time the
instance had about 200 queued runs, about 80 running runs, a slow
database, and many continuation runs. The failed server logged "managed
environments ensured from managed config" (`index.ts` line 1284 on
master). It never logged "worktree run-execution cutoff state" (line
1726), which the server always logs when the heartbeat service exists.
Between those two lines, boot awaited only these steps: `tool_reviews`
(four tool-review queue first passes), `gateway_sessions`,
`delivery_workers` (budget enforcement, cost accounting, fast responses,
chat completions, connection continuations, tool action receipts and
question responses), the GitHub connection event and continuity workers,
and `terminalWorkspaceWorker.ready`. Each step is a delivery-coordinator
`register(...).ready`, and that promise resolves only after the queue's
whole first pass. No queue except `mcp-client-mirror` (also failing on
the healthy server) logged an error, so the cause was slow passes and
not failures. The log had no per-step markers, so it does not show which
of these awaits was still running at the kill.
**Expected behavior**
Readiness means "this server can serve requests and take over". It must
not wait on work that grows with backlog or that depends on an outside
service. Queues own that work and retry it on their own schedule after
readiness.
**Steps to reproduce**
1. Start the server while the first pass of any of the queues above does
not finish. The new test in `server-startup-feedback-export.test.ts`
does this for every queue.
2. On master, `startServer()` does not resolve and the phase stays
"recovering".
## What Changed
- `server/src/index.ts`: These first passes now start in the background
as tracked scheduler work and do not hold readiness: tool reviews,
gateway sessions, delivery workers, GitHub connection workers, the
terminal workspace sweep, environment lease cleanup (scheduler-disabled
path) and scheduled database backups. Shutdown and idle sleep still wait
for them through `trackHeartbeatSchedulerWork`, and each queue keeps its
retry schedule.
- `server/src/index.ts`: The restart backup after an idle wake (a
`pg_dump` whose time grows with the database) holds idle eligibility
(`markIdleStartupComplete`), as before. It no longer holds readiness.
- `server/src/index.ts`: Every remaining await on the boot path is a
named `startupTimeline.precondition(...)`: `listen`,
`runtime_control_operations`, `execution_policy`,
`managed_environments`, `worktree_run_execution`,
`scheduling_suppression`, `native_recovery`, `external_adapters` and
`cloud_configuration`. These preconditions do not change.
- `server/src/startup-timeline.ts`: Adds `precondition(step, work)` and
`background(step, work)`. A precondition that is still running after 30
s is named in the warning "startup step is still holding readiness",
which repeats every 30 s. A background step logs a warning once at 30 s
and logs its duration when it finishes late. A background step that
fails is logged with its step name. The line "startup timeline at
readiness" now includes `readinessStepsMs`, `backgroundStepsMs` and
`backgroundStepsOutstanding`. All timers are single unref'd timeouts and
the module adds no queries.
- `server/src/__tests__/startup-readiness-boundary.test.ts`: This guard
test parses `index.ts` with `parseAst` from `vite`. The server already
depends on `vite`, so this PR adds no dependency and no lockfile change.
The test fails when an await on the boot path (between the phases
"recovering" and "ready") is not a named precondition, or when the
precondition names do not match a reviewed list that gives the reason
for each name.
- `server/src/services/terminal-workspace-work.ts`: Updates a comment.
The first sweep no longer holds readiness, and it still skips
compaction.
## Verification
- Red/green regression test: `server-startup-feedback-export.test.ts`,
"reports ready while backlog queues' first passes never finish". This
test makes every first pass above never finish. With master `index.ts`
it fails with `expected 'still starting' to be 'ready'` for both
scheduler settings. With this change it passes.
- Guard test: with master `index.ts`,
`startup-readiness-boundary.test.ts` fails and lists 18 unnamed awaits,
including `timeStartupRecovery("delivery_workers")` and
`terminalWorkspaceWorker.ready`. With this change it passes.
- `cd server && npx vitest run
src/__tests__/server-startup-feedback-export.test.ts
src/startup-timeline.test.ts
src/__tests__/startup-readiness-boundary.test.ts
src/startup-recovery-timing.test.ts
src/__tests__/terminal-workspace-work.test.ts`: all tests pass.
- `cd server && pnpm typecheck`: passes.
- `pnpm verify` exited with code 2 on the first revision (a lockfile
change that this PR no longer makes). Required CI supplies the broad
coverage.
- After deploy, run `grep "startup timeline at readiness"` on a server
start. The line shows each precondition's duration and the background
passes that are still running.
## Risks
- The listed first passes now run at the same time as native run
recovery and the requests that the server serves, not before them.
Requests were already served during recovery. Commit-woken passes of the
same queues, and the fresh queued-run dispatch, already ran at the same
time as native recovery. The coordinator still serializes the passes of
each queue.
- `/api/ready` can now return 200 while a queue still has backlog. That
is the intended meaning, and it matches the queued-run and recovery
passes that already run after readiness (#21014).
- Native run recovery still holds readiness, which is correct because
adoption must happen before takeover. If that recovery is slow, the
server now logs its step name every 30 s.
- Low risk otherwise: no schema or API change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic) via Claude Code, model ID `claude-opus-5-5` (commit
trailer: Claude Fable 5.2). Tool use and code execution were enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
b0233b5b33 |
fix(native): count a parent run's read of a finished child as delivery of its result (#21227)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task gets a wake when a child task completes. If the parent is running, that wake waits behind the run. > - At finalization, a parent's done claim stays open (`native_child_completion_pending`) while such a wake is still owed, so the parent never finishes without the child result. > - The server had no record of what a running parent actually read. A parent that read its finished children mid-turn and then finished the task still kept the wake owed. > - The task stayed `in_progress`, the wake started a second parent turn, and the agent wrote a second completion summary. > - This pull request records a run's read of a finished child and counts that read as delivery of the child result. > - The benefit is one completion turn when the parent already read its children, while unread child results still get a fresh turn. ## Linked Issues or Issue Description Refs #21189 (child results saved before the run started). Refs #21175 (the same pattern for comments: `task_history.read`). Refs #15372 (the pending child-completion guard). **What happened?** Restart-chaos wave `w-20261112T223803-s7101`, on a master build that includes #21189, task t006 (OpenCode, native runner, staging-mix parent with two subtasks): - Parent run `bfe9941c` started at 22:55:56. - Its children CHA-11 and CHA-12 finished at 23:07:53 and 23:11:38. - The child-completion intent and its deferred dispatch row were saved at 23:11:38, after the run started. So the #21189 rule counted them as new input. - At 23:26:41 the run's reasoning says "Both subtask ledgers exist … Both subtasks are done". The run then saved the CHILD hashes and posted DONE at 23:33:39. - At finalization, the done claim was committed as `native_child_completion_pending`. The deferred wake then started run `40928116`, which wrote a second completion summary. Root cause: `hasPendingNativeChildCompletion()` (`server/src/services/native-runtime/native-child-completion-delivery.ts`) proved delivery only by wake binding and timestamps. No durable record showed that the finishing run had read the completed child. **Expected behavior** A finishing run's recorded read of a completed child counts as that child result being delivered. A read while the child was open, a read by another run, or a failed event write does not count. An unread child result still keeps the task open and gets one fresh parent turn. **Steps to reproduce** 1. Start a native parent run. 2. Complete a child while the run is active. The child-completion wake waits behind the run. 3. Let the run read the child (`get_task_context`, or `GET /api/issues/:childId`), then claim `done`. 4. Finalize and release the run. On `master`, the decision is `native_child_completion_pending` and the wake becomes a second run. The same wave's t001 grade ("duplicate completion") is not this bug. That run only said it *will* post the DONE marker later, and the grader counts any mention. Its `subtasks_once` failure comes from the restart-recovery continuation. This PR does not change either one. ## What Changed - `native-child-completion-delivery.ts`: - New `recordChildTaskRead()` writes a `child_task.read` run-log event. The payload is `{ parentIssueId, children: { id: { status, updatedAt } } }`, and it lists only children in `done`. - `hasPendingNativeChildCompletion()` no longer counts a child-completion wake as pending when the finishing run recorded a read of that wake's completed child in `done` at or after the child's completion decision (`status_decisions.created_at`, or the wake's `createdAt` when the decision row is missing). - The #21189 start-time rule is unchanged. - `paperclip-runner-tool-authority.ts`: `get_task_context` records the finished child tasks it returns. A failed write is logged and leaves the result owed. - `routes/issues.ts`: `GET /issues/:id`, `GET /issues/:id/documents` and `GET /issues/:id/documents/:key` record the read when the request has all of these: - an agent actor with a run token - a child issue in `done` This runs after the existing access and company checks. So a board user, another company's agent, or a read of an open task writes nothing. A qualifying read adds one statement: the existing `appendHeartbeatRunEvent` fast path, a single `update … returning` / `insert` CTE bound to the run's id, company and agent. Every other read adds no query. - `doc/run-log-events.md`: documents `child_task.read` and how delivery uses it. - Tests: a finalization-to-release regression in `heartbeat-process-recovery.test.ts` and a new `native-child-task-read-routes.test.ts`. ## Verification - RED on `master`: in `heartbeat-process-recovery.test.ts` "counts a child result as read by the finishing parent run (saved after the run started, read: after the child finished)", the run reads the child through the real `get_task_context`, then finalization and the wake-queue release run. - On master: `native_child_completion_pending`, task `in_progress`, and the wake is promoted into a second run, the same as t006. - With this change: `done`, the wake is cancelled, and there is one run. - The other cases in the same test behave the same before and after this change: - saved before the run started: done - not read: pending, fresh turn with the child results - read while the child was open: pending - `native-child-task-read-routes.test.ts` (5 tests): - An agent read of a done child through each of the three routes records the read and clears the pending result. - An open child records nothing. - A board read records nothing. - Another company's agent gets no 200 and records nothing. - Neighbouring suites after merging `origin/master`: `heartbeat-process-recovery`, `muse-control`, `heartbeat-held-comment-wake`, `native-status-arbiter-corpus`, `heartbeat-dependency-scheduling`, `paperclip-runner-tool-authority`, `agent-task-hierarchy-depth-routes`, and the two new test files: 8 files, 982/982 tests pass. - `pnpm --filter @paperclipai/server typecheck` passes. **Not verified** - No chaos wave has run on this change yet. - In t006, the parent's tool calls fell in the 4,220 run events that the evidence omitted. Its reasoning text proves the read, but the exact tool (`get_task_context`, `get_task`, `read_task_document` or `call_api`) is not visible. - A read through `search_tasks`, or through a shell command that calls the API without the run token, is not recorded. Such a result stays owed and gets one fresh turn, which is the old behavior. ## Risks - A recorded read of a done child now lets the parent finish without another turn. If the agent read the child but ignored it, no second turn reminds it. A read while the child was still open never counts. - Each qualifying agent read of a done child appends one run-log row. Board reads and reads of open tasks write nothing. - No migration. Wakes saved by older builds work unchanged. Without a recorded read, the old behavior applies. - This is a run-log change only (instance database). It adds no Telemetry or OpenTelemetry export. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8b4b6aec87 |
test(chat): keep the Retry route's background nudge out of the board-message retry test (#21174)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-channel integration suite runs in 3 "General tests (chat)"
CI shards on every PR
> - "retries the saved board-message turn in a connected task after
environment qualification failed before provider startup" failed on 4
unrelated PR branches in 48 hours
> - Every failure is a foreign-key error when the test deletes its saved
Retry: `heartbeat_runs_wakeup_request_id_agent_wakeup_requests_id_fk`
> - `POST /issues/:id/retry` answers 202 and then calls
`heartbeat.resumeTaskRetry` without awaiting it. When that nudge created
a run before the test's cleanup, the delete failed
> - The test drives dispatch itself (`resumeExecutionWaitComments`), so
the background nudge only adds a race
> - This pull request gives the test's route a heartbeat service whose
nudge does nothing
> - The benefit is that the chat shard stops failing at random on
unrelated PRs
## Linked Issues or Issue Description
**What happened?**
`server/src/__tests__/chat-channels.integration.test.ts` "retries the
saved board-message turn in a connected task after environment
qualification failed before provider startup" failed with `update or
delete on table "agent_wakeup_requests" violates foreign key constraint
"heartbeat_runs_wakeup_request_id_agent_wakeup_requests_id_fk"` on PRs
that did not touch chat or Retry code (for example
codex/github-pat-without-cloud, fix/21017-workspace-source-lock-wait,
fix/codex-internal-interrupt-recovery).
**Expected behavior**
The test passes every time. Its Retry dispatch happens only where the
test starts it.
**Steps to reproduce**
Add a 300 ms wait at the start of the `denySource` cleanup (before the
`agent_wakeup_requests` delete) and run the test. On master it fails 3
of 3 times with the CI error, because the route's background nudge
creates a run first.
**Paperclip version or commit**
master at
|
||
|
|
334b2ae066 |
fix(recovery): continue a task blocked behind its workspace's ended writer once the hold releases (#21221)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote task workspaces have one writer run at a time, and an ended
writer keeps the workspace until its files are saved or set aside
> - When a task's next runs fail setup on that hold ("The workspace
still has an unfinished writer or save operation."), stranded-work
recovery blocks the task with a board escalation
> - The retained-file recovery from #21136 later released such a hold on
staging (it continued from the last save), but nothing settled the board
escalation, so the task stayed Blocked until a person pressed Retry
> - Recovery must work without a person: the block belonged to the hold,
and the hold is gone
> - This pull request adds a pass to the active recovery-action sweep
that settles that escalation once the workspace's writer hold is free
> - The benefit is that such tasks continue by themselves with one plain
notice and one run, with no Retry
## Linked Issues or Issue Description
Refs #21136
Refs #21143
**What happened?**
On staging, an ended run held its task's own remote workspace after its
superseded workspace export failed. The task's queued runs failed setup
with the unfinished-writer error, and stranded-work recovery blocked the
task with a board-owned escalation. After #21136 was deployed, its
retained-file recovery recorded the hold 77 seconds after server start
and, 19 minutes later, continued the task from its last save and
released the workspace. The task still stayed Blocked behind the board
escalation, and no run was queued. A board user had to press Retry.
**What did you expect to happen?**
Once the hold that caused the setup failures is released, the task
returns to `todo` and its assignee continues it, with no Retry.
**Steps to reproduce**
1. Give a task a remote workspace whose writer is an ended run with
unsaved files.
2. Let the task's next run fail setup with the unfinished-writer error,
and let stranded-work recovery block the task.
3. Release the hold (save the files, or continue from the last save).
4. Run the active recovery-action sweep. Before this change the task
stays Blocked with an active board escalation.
**Why the merged fix did not unblock the task**
The retained-file fallback only releases a Blocked status that its own
hold set (`holdOwnsBlockedStatus`). Here stranded-work recovery set the
block with a different, board-owned escalation, and a late status
decision 1.4 seconds later kept that block. So the fallback correctly
did not take the status, and no code settled the board escalation after
the hold was gone.
## What Changed
- Add `continueTasksAfterWorkspaceWriterReleased`
(`server/src/services/recovery/released-writer-escalation.ts`). It
selects board-owned `stranded_assigned_issue` escalations whose latest
run failed setup with the unfinished-writer error before any provider
work, by the current assignee. In one transaction it checks that the
block belongs to the escalation, that no newer run exists for the task,
that dependencies allow work, and that the workspace's writer hold is
now free. Then it resolves the escalation with a named outcome, returns
the task to `todo`, records one plain activity notice, and records a
pending wake.
- A later status decision that only kept the Blocked status it found
(from `blocked` to `blocked`) does not own the block. A decision that
chose the block still keeps the task blocked.
- Run the pass in `reconcileActiveRecoveryActions`, next to the legacy
rehome pass, and include its next wake retry time in the pass deadline.
- Generalize the legacy rehome wake delivery to
`deliverPendingRecoveryWakes`, keyed by the evidence key, so both passes
share one crash-safe, backed-off wake delivery. The rehome wake keeps
its fresh session. The new wake resumes normally.
- Keep the company selected by the scheduled sweep while a
released-writer wake is owed (`recoveryActionSweepCondition`).
- Document the rule in `doc/execution-semantics.md`.
## Verification
- `cd server && npx vitest run
src/__tests__/released-writer-escalation.test.ts
src/__tests__/legacy-workspace-rehome.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts
src/__tests__/issue-blocker-stall-waiter-statement.test.ts
--maxWorkers=2`: all pass.
- Red/green: with the sweep hook removed, "continues the task once the
hold is released" fails (the task stays `blocked`). With the hook, it
passes.
- The new test seeds the staging row shape: a native writer run ended
with `native_finalization_superseded` still writing the task's own dirty
workspace, three legacy setup failures with the unfinished-writer error
and `{ kind: "bootstrap", providerWorkStarted: false }`, a board
escalation with `latestRunErrorCode: "setup_failed"`, the block 4.4
seconds after the last failure, and a later `blocked` to `blocked`
status decision.
- The tests also cover: no change while the writer still holds the
workspace, one wake only, a pending wake after a crash, a refused wake
that backs off, a later decision that chose the block, a dependency that
forbids work, a newer run, and another setup error.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.
## Risks
- Low risk. The pass only acts on board escalations whose evidence is
exactly the unfinished-writer setup failure, and only when workspace
reservation would now admit a new writer. It never changes the
workspace, never applies a result, and never releases a writer fence.
- The task gets one normal wake through the issue-state guard (`todo`,
same assignee), so it cannot start twice.
- The pass adds one bounded query per sweep of a company with active
recovery actions.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser API reads of staging) and extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
0904a65d82 |
fix(server): local board boundary identifies agent processes on Linux it cannot inspect (#21209)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In `local_trusted` mode, the operator's own local processes get implicit board access without credentials > - #21181 added a local board boundary. It refuses implicit board access to requests that come from an agent run's process tree > - On Linux CI hosts, the boundary sometimes did not decide a loopback request. It logged a warning and kept the earlier permissive behavior for that request > - The cause was not IPv6. IPv4 and IPv6 loopback lookups worked. The gap was a connection that belonged to a same-user process that the server cannot inspect > - This pull request makes the Linux boundary decide those connections from the process ancestry that the server can read, and refuse them when that ancestry is not clearly the operator's > - The benefit is that the local board boundary now holds on Linux for IPv4 and IPv6 loopback, and the operator's browser and CLI keep working ## Linked Issues or Issue Description Refs #21181 **What happened?** On Linux CI hosts, the local board boundary sometimes could not attribute a loopback request to a process. It then logged that it could not identify the local process and kept the earlier permissive behavior for that request. In one chat eval cohort, 37 of 191 server logs contained this warning. The warning is logged once per server process. **Expected behavior** The boundary decides every uncredentialed loopback request on Linux. A request from an agent run gets no implicit board access. Requests from the operator's browser and CLI keep implicit board access. **Actual behavior** Some requests on Linux were not decided by the boundary, and they got implicit board access. ## What Changed - The Linux inspector now records which same-user processes it cannot inspect. Before, it only recorded that one existed. - The Linux TCP table lookup now also returns the socket's owner UID. It covers `/proc/net/tcp` and `/proc/net/tcp6`, including IPv4-mapped addresses. - If no inspectable process owns the connection, the server reads the TCP tables again. If the connection is gone, has changed, or belongs to a different user, the request is refused. - If the connection is still open and same-user, the server checks every process it cannot inspect. Implicit board access stays only if each one has a readable parent chain that does not belong to an agent run. Otherwise the request is refused. - The "cannot identify" fallback now applies only when the platform has no inspector, or when Linux cannot read the TCP tables. The server keeps logging a warning in that case. - `doc/DEPLOYMENT-MODES.md` describes the new Linux behavior. **How the operator is recognized** - Browser and CLI processes are normal same-user processes. The server reads their open sockets and environment, so their requests are classified as before. - A browser helper process that the server cannot inspect is still the operator when its readable parent chain leads to the browser and has no agent-run markers. - The server's own requests and requests from the server's ancestors stay the operator, as before. - Connections owned by other users were refused before, and they are still refused. ## Verification - `pnpm --dir server exec vitest run src/__tests__/local-agent-process-boundary.test.ts` passes on macOS (15 passed, 10 Linux-only skipped). - The same file ran on Linux (Docker, `node:24-bookworm`, non-root user, IPv6 loopback on). Result: 25 passed. - Red check on Linux with the previous boundary and the new tests: the agent-run cases for processes the server cannot inspect failed for both `127.0.0.1` and `::1`. The operator cases passed. - New Linux-only tests run real processes over `127.0.0.1` and `::1`: - an operator process keeps board access - an agent-run process is refused - an operator process the server cannot inspect keeps board access - an agent-run process the server cannot inspect is refused - New platform-independent tests cover the classification rules and the `/proc/net/tcp6` fixture parsing. - A paid chat-feature run on Linux CI, with temporary diagnostic logging on a separate branch, checks the server logs for the warning and for refused operator requests. The results will be in a PR comment. ## Risks - A rare operator tool that the server cannot inspect, and whose parent chain the server cannot read, now loses implicit board access. Such a tool can use a board API key. - The extra TCP table read happens only when a connection has no inspectable owner. Normal requests do no extra work. - Platforms without process inspection keep the previous behavior. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Docker on Linux) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b51f1cb55f |
test(chat): size the long Board publication tests' timeout to their measured work (#21205)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-channel integration suite is one 86,000-line file. Its
cases share one database, and the chat services scan that whole database
> - The "Lossless long Board publications" cases publish
100,000-character messages in many durable parts. They use the file's
default 15 s test timeout
> - Run alone, a case takes about 0.6 s. Late in a full run of the file,
the same cases take 6–10 s locally and 1.4–15 s in CI, because earlier
cases leave rows that the services still scan
> - On #21041, "keeps an ambiguous slack text part blocked across
restart…" hit the 15 s limit twice (30 041 ms with its retry), and the
GitHub variant passed at 14 960 ms
> - This pull request gives this describe block a 45 s timeout, sized to
the measured work
> - The benefit is that the chat shard stops failing on unrelated PRs
when these cases run late in a shard
## Linked Issues or Issue Description
**What happened?**
`server/src/__tests__/chat-channels.integration.test.ts` > "Lossless
long Board publications" > "keeps an ambiguous slack text part blocked
across restart and retries only that part and its tail" failed with
`Test timed out in 15000ms` on PR #21041 (General tests chat 2/3), a PR
that did not change chat code. It passed on rerun.
**Expected behavior**
The case passes wherever the shard puts it.
**Steps to reproduce**
Run the whole file: `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts --reporter=json`. In my
local full run, "preserves all 100000 characters for slack new-comment
publication" took 9 505 ms and "keeps an ambiguous slack text part…"
took 5 856 ms. Run alone, they take 817 ms and 577 ms. In CI logs from
the last 48 hours, the ambiguous-part cases took between 1 408 ms and
over 15 000 ms.
**Paperclip version or commit**
master at
|
||
|
|
161ef8121e |
perf(recovery): prepare the blocker-stall waiter read once instead of writing its SQL for every stalled task (#21201)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a task stalls on a recovery action that needs the board, recovery wakes the agents of the tasks that wait on it: its parent and the tasks it blocks (`wakeWaitersOfStalledIssues` in `server/src/services/recovery/blocker-stall-wakeups.ts`). > - Each pass reads the waiters of every stalled task with one statement for each stall. The statement has 18 parameters and two correlated subqueries. Its SQL text is 0.8 KB, and it is the same for every stall of the same shape. > - The server wrote that text with the query builder for every stall on every pass. On a hosted instance the server main thread was near one full core and garbage collection used a quarter of it. A statistics sample showed this statement at about 7.8 executions per second. > - I measured it on a local PostgreSQL through the real client. The query builder allocates 103 KB to write and run this statement. The same statement from a prepared query allocates 15 KB. > - This pull request writes the SQL text of the waiter read one time for each client and shape. Each execution then only binds values. > - The benefit is 76% less allocation and 5 times fewer garbage collections for a pass over 20 stalled tasks. The statement, its parameters and its rows do not change. ## Linked Issues or Issue Description Related: #21195 (the same change for the native chat progress page read; it added the `preparedStatements` helper that this pull request uses), #21131 and #18965. No issue exists for this read. **What existing behavior does this improve?** The cost, in the server process, of the waiter read that the blocker-stall wake pass runs for each stalled task. **Subsystem affected** Recovery: blocker-stall wakes (`server/src/services/recovery/blocker-stall-wakeups.ts`). **Current behavior** `wakeWaitersOfStalledIssues` builds the waiter query with the Drizzle query builder inside its loop, and Drizzle then writes the SQL text from it, one time for each stall and page. Measured on the merge base, Node 26, local PostgreSQL 18, median of 5 passes of 300 calls. The fixture has 20 stalled tasks, each with a parent and one dependent task that are already woken, so a pass writes nothing: | Call | Statements | Waiter SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | --- | | Company pass over 20 stalls | 21 | 20 | 2,298.3 KB | 3,855 µs | 100 (163.7 ms) | | Pass for one stalled task | 4 | 1 | 355.0 KB | 547 µs | 13.3 (14.3 ms) | At the sampled rate of 7.8 executions per second the read allocates about 0.8 MB per second. This rate is one sample of one instance, taken soon after a long outage, when recovery had a backlog. It is not a steady-state figure. **Proposed behavior** The query moves into one builder function, `stallWaiterPageQuery`. The module prepares it one time for each client and shape, with placeholders for the values. The shapes are: the stalled task has a parent or not; all waiters or one waiter; first page or later page. The loop executes the prepared statement. **Reason and benefit** Same benchmark, this branch: | Call | Statements | Waiter SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | --- | | Company pass over 20 stalls | 21 | 0 | 547.3 KB | 1,474 µs | 20 (24.8 ms) | | Pass for one stalled task | 4 | 0 | 266.9 KB | 496 µs | 10 (9.9 ms) | One waiter read costs about 88 KB less. At the sampled rate this removes about 0.7 MB per second of allocation, which is about 0.9% of the 77.7 MB per second that the instance allocated. It is a small part of the total. The CPU numbers come from a machine with other load and are less stable than the allocation numbers. **Breaking changes** None. ## What Changed - `server/src/services/recovery/blocker-stall-wakeups.ts`: the waiter query moves out of the loop into `stallWaiterPageQuery(db, binding)`. `stallWaiterPageStatement` returns the prepared statement for the shape and the values to bind. The loop executes it. The lines inside the two SQL fragments keep their indentation, so the SQL text is byte-identical. The pull request adds no comment. The one existing comment of this read stays at the call site, where the code still makes that choice. - `server/src/__tests__/issue-blocker-stall-waiter-statement.test.ts` (new): statement equivalence, row equivalence on PostgreSQL, and the red/green mechanism test. - `server/src/__tests__/helpers/blocker-stall-fixture.ts` (new): the fixture that the test and the benchmark share. - `server/src/__tests__/prepared-statements-driver.test.ts` (new): proves at the driver that a prepared read and the query builder ask postgres.js for the same thing, with the driver's prepared-statement setting unset, false and true. It covers this read and the native chat progress page read of #21195. - `server/scripts/benchmark-blocker-stall-waiters.ts` (new, no comments): the benchmark above. It loads the module at `--baseline-ref` next to the current module in one process. `sweep` is one company pass over `STALLS` stalled tasks whose waiters are already woken: one stall read, then one waiter read for each stall. `task` is the pass for one stalled task. Before it measures, the script also runs a company with a paused agent and 101 waiters on a stall with a parent and on a stall without one, and one pass for a single waiter of each. This covers six shapes: the first page, the later page and one waiter, each for a stall with a parent and for a stall without one. The script fails unless both versions send the same statements, byte for byte, with the same parameters and return the same result. Passes alternate between the versions with a full collection before each pass. CPU is the whole process. Allocated bytes are V8's `total_allocated_bytes`. Optional environment variables: `STALLS` (default 20), `ITERATIONS` (default 300 calls for each pass), `PASSES` (default 5 for each version). Not changed: which statements run, their order, the other reads of the pass (they are small and vary by waiter), the wake itself, the driver's prepared-statement setting, and the schema. The read takes no lock and runs on the root client outside a transaction. Its two lists (waiter statuses and wake statuses) are module constants, so the statement has no variable-length list parameter. ## Verification - Statement equivalence (`sends the SQL text and the parameters that the query builder writes, for every shape of the read`): for all eight shapes the prepared SQL text equals the text that the builder writes with the values, and the bound parameters are equal. The test also proves that a stall without a parent binds `null` for the parent. - Against the merge base at the driver: the benchmark compares every statement that the merge base module and this branch send for six of the eight waiter-read shapes. The texts are byte-identical and the parameters are equal. The two shapes that it does not reach (one waiter on a later page) cannot occur: a one-waiter read returns at most one row, so it has no later page. - Row equivalence on PostgreSQL (`returns the rows of the query builder: same values, types and order, and no other company's rows`): two companies. The fixture has a stall with a parent and six dependents, and a stall without a parent. One dependent is already woken, one is done, one is hidden and one has no assignee. One has a failed wake only, and one has a wake row with the same key in the other company; both stay waiters. The prepared statement and the builder return strictly equal rows for all waiters, one waiter, a later page, and a stall read under the other company's id (no rows). Null columns stay null. - Mechanism, red then green (`writes the SQL text of the waiter read one time and then only binds values`): a pass over three stalls writes no waiter SQL text and sends one identical text three times. On the merge base module the same test fails: the pass writes three texts. - Driver behavior (`asks the driver for the same thing as the query builder with DATABASE_PREPARED_STATEMENTS=...`, three cases): Drizzle's `.prepare(name)` on postgres.js does not send the name to the database. `PostgresJsSession.prepareQuery` drops it (`drizzle-orm/postgres-js/session.js`, lines 87 to 100), and `PostgresJsPreparedQuery.execute` calls `client.unsafe(text, params)` (lines 33 and 43). An ordinary query reaches the same call through `_prepare().execute()` (`pg-core/query-builders/select.js`, lines 795 to 799). The test runs both reads through the builder and through the prepared statement on one connection for each setting. The arguments and the options of the driver call are strictly equal. With the setting unset or false, the server has no named statement after either path, so a transaction-mode pooler sees no change. With the setting true, the server has one named statement. Its name is the one that postgres.js generates for the connection, and both paths use that same statement. - Existing tests: `issue-blocker-stall-wakeups.test.ts` and `legacy-workspace-rehome.test.ts` passed (30 tests) with the new file (3 tests). `prepared-statements-driver.test.ts` passed (3 tests). - `pnpm --filter @paperclipai/server typecheck` passed. `pnpm check:module-boundaries` passed. - `pnpm verify` returns exit 2 (broader verification required, because the change adds a file under `server/scripts`). I did not run the full local suite. CI supplies that coverage. Reproduce: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-blocker-stall-waiter-statement.test.ts \ src/__tests__/issue-blocker-stall-wakeups.test.ts src/__tests__/legacy-workspace-rehome.test.ts \ src/__tests__/prepared-statements-driver.test.ts TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-blocker-stall-waiters.ts \ --baseline-ref=549816e6a0 ``` Not verified: the effect on a hosted instance. The rate above is one sample. The local numbers are from Node 26; the hosted instance runs Node 24. ## Risks - Low. The statement text and parameters are equal by test, the read takes no lock, and it runs on the root client outside a transaction. - A prepared statement keeps the text of the first build. The lists inside the statement are module constants, so the text cannot go stale in a running process. - A test that mocks `db.select` to observe this read no longer sees it. I found none for this read. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude Code with tool use and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
7f1cdb842d |
fix(evals): admit the owner-routed routine follow-up run and brief only the requested change (#21198)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat eval catalog checks that agents in chat do the right thing on saved product state. > - In `routines.manage-other-owner`, the CEO cannot change Morgan's routine. The valid outcome is to route the change to Morgan in one linked task. > - That valid path uses three runs: the CEO's turn, Morgan's run on the task, and the CEO's automatic completion follow-up on the chat. The case allowed only two runs, so the harness stopped valid attempts with "exceeded its declared run bound". > - In one of those attempts, the CEO also told Morgan to use UTC. Nobody asked for UTC. The routine was on America/Chicago. The oracle correctly failed that change. > - This pull request admits the third run only in its declared role, and it tells agents to brief only the change the person asked for. > - The benefit is that the eval accepts the valid delegation path and still fails extra work and unrequested setting changes. ## Linked Issues or Issue Description **What happened?** Runner Full-Stack E2E runs 43298807862 and 43300040122 stopped `routines.manage-other-owner` with "Scenario exceeded its declared run bound". Both had three runs: the CEO turn, Morgan's run on the routed task, and the CEO's `chat_task_completed` follow-up. In 43298807862 the CEO's brief also told Morgan to switch the routine to UTC, and `same-owned-definition-and-triggers` failed. **Expected behavior** The delegation path with exactly one routed owner task passes with three runs. The refusal path stays at two runs. An agent that relays a change keeps every setting the person did not ask to change. **Steps to reproduce** Run `chat-features.runner-opencode.local.routines.manage-other-owner` and let the CEO route the change to Morgan. **Paperclip version or commit** master at |
||
|
|
549816e6a0 |
perf(chat): prepare the native progress page read once instead of writing its SQL on every wake (#21195)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run that came from a chat reports safe progress to that chat. Each progress event wakes a queue, and each wake reads one page of candidates (`nativeProgressPages` in `server/src/services/chat-native-progress.ts`). > - The page read joins six tables and has 44 parameters. Its SQL text is 3.6 KB. The server wrote that text with the query builder on every wake, and the text is the same for every wake of the same filter. > - On a hosted instance the server main thread was near one full core and garbage collection used a quarter of it. An allocation sample put 46% of all allocation under the query builder's SQL writing. The sample does not name statements. > - I measured single statements on a local PostgreSQL through the real client. The query builder allocates about 20 KB for the smallest statement, plus about 2 KB for each selected column and about 3 KB for each predicate. This page read costs 296 KB and 0.34 ms of CPU to write. The same statement from a prepared query costs 21 KB. > - This pull request writes the SQL text of the page read one time for each client and filter shape. Each execution then only binds values. > - The benefit is 91% less allocation and 14 times fewer garbage collections for one page read. The statement, its parameters and its rows do not change. ## Linked Issues or Issue Description Related: #21131 (fixed SQL text for the native run event append) and #18965 (an earlier prepared read, in the GitHub review wait). No issue exists for this read. **What existing behavior does this improve?** The cost, in the server process, of the candidate page read that every native chat progress wake runs. **Subsystem affected** Chat channels: native run progress publication (`server/src/services/chat-native-progress.ts`). **Current behavior** `nativeProgressPages` builds the complete query with the Drizzle query builder for each page, and Drizzle then writes the SQL text from it. Measured on the merge base, Node 26, local PostgreSQL 18, median of 5 passes of 2,000 calls: | Call | SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | | One page read | 1 | 297.3 KB | 851 µs | 14.5 (30.7 ms) | | One wake for a task (page read + one transaction) | 7 | 681.3 KB | 2,238 µs | 34 (76.6 ms) | A statistics sample of one hosted instance on one morning showed this statement at about 3.9 executions per second. At that rate the read allocates about 1.1 MB per second for SQL text. This rate is one sample, taken soon after a long outage. It is not a steady-state figure. **Proposed behavior** The query stays in one builder function, `nativeProgressPageQuery`. The module prepares it one time for each client and each of the six filter shapes (all tasks, one company, one task; first page or later page), with placeholders for the values. Each page read executes the prepared statement. **Reason and benefit** Same benchmark, this branch: | Call | SQL texts written | Allocated | Process CPU | Collections per 1,000 calls (pause time) | | --- | --- | --- | --- | --- | | One page read | 0 | 26.8 KB | 219 µs | 1 (1.4 ms) | | One wake for a task | 6 | 412.1 KB | 1,699 µs | 21 (55.5 ms) | At the sampled rate this removes about 1.05 MB per second of allocation, which is about 1.4% of the 77.7 MB per second that the instance allocated. It is a small part of the total. The remaining six statements of a wake run inside a transaction, where a prepared query gains nothing, and this pull request does not change them. **Breaking changes** None. ## What Changed - `server/src/services/chat-native-progress.ts`: the page query moves into `nativeProgressPageQuery(db, binding)` without a change to its text. `nativeProgressPageStatement` returns the prepared statement for the filter shape and the values to bind. `nativeProgressPages` executes it. - `server/src/services/prepared-statements.ts` (new, 17 lines, no comments): - `preparedStatements(prepare)` keeps one prepared statement for each client and shape. A prepared statement runs on the client that prepared it, so the cache key is the client object. A transaction client gets its own statement and gains nothing. - `columnPlaceholder(name, column)` makes a placeholder whose value the column encodes. A bare `sql.placeholder` skips the column's encoding, so a `Date` would reach the driver unencoded. With this helper it reaches the driver as the same ISO string as a value that the query builder binds. - `server/src/__tests__/chat-native-progress-statement.test.ts` (new): statement equivalence, row equivalence on PostgreSQL, and the red/green mechanism test. - `server/src/__tests__/prepared-statements.test.ts` (new): the helper. - `server/src/__tests__/chat-native-progress-work.test.ts` and `chat-channels.integration.test.ts`: three tests watched or failed the page read through `db.select`. A prepared statement does not call `db.select`, so these tests now watch the driver (`db.$client.unsafe`). The idle assertions are stricter: they now fail on any statement. - `server/scripts/benchmark-native-progress-page.ts` (new, no comments): the benchmark above. It loads the module at `--baseline-ref` next to the current module in one process. `page` is `loadNativeChatProgressScopes` for one task, which sends the page read and nothing else. `wake` is `reconcileNativeChatProgress` for one task whose progress is already published, so it writes nothing and every pass does the same work. The script fails unless both versions send the same statements with the same parameters and return the same result. Passes alternate between the versions with a full collection before each pass. CPU is the whole process (`process.cpuUsage`). Allocated bytes are V8's `total_allocated_bytes`. A collection counts when it starts inside a pass. Optional environment variables: `ITERATIONS` (default 2000 calls for each pass) and `PASSES` (default 5 for each version). `TSX_DISABLE_CACHE=1` keeps the tsx cache scan out of the first passes. Not changed: which statements run, their order, the transaction of a wake, its `FOR UPDATE SKIP LOCKED` reads, the driver's prepared-statement setting, and the schema. The read has no variable-length list parameter. ## Verification - Statement equivalence (`sends the SQL text and the parameters that the query builder writes, for every shape of the read`): for all six shapes, and for an empty task id, the prepared SQL text equals the text the builder writes with the values, and the bound parameters are equal. The test also proves that the time filter is bound as the ISO string. - One-time check against the merge base: I derived the builder of the merge base from its source and compared 24 inputs (4 scopes, with and without a cursor, 3 page sizes). The prepared SQL text is byte-identical, and the parameters are strictly equal. - Row equivalence on PostgreSQL (`returns the rows of the query builder: same values, types and order, and no other company's rows`): two companies, four candidate runs, and three runs that are not candidates. The fixture has a null payload, a nested JSON payload, a run bound through its task column only, and microsecond timestamps. The prepared statement and the builder return strictly equal rows for every filter, for pages of one row walked with the cursor, and for a task of another company (no rows). - Mechanism, red then green (`writes the SQL text of a read one time and then only binds values`): three reads for different tasks write no SQL text and send one identical text three times. On the merge base module the same test fails with "expected sqlToQuery to not be called at all, but actually been called 3 times". - The benchmark asserts that both versions send the same statements with the same parameters and return the same result, for the page read and for a full wake. - Existing tests: `chat-native-progress-work.test.ts` (19 passed), `chat-channels.integration.test.ts` (1,954 of 1,963 passed in the complete local run, including the 49 tests that match `native progress|saved progress|safe native`. Two tests fail on this machine because the worktree is inside another checkout: the bundled Slack skill resolves to that checkout's skills directory, which has no `slack/SKILL.md` (`ENOENT` in `connector-runtime.ts`). Seven later tests failed in the same run from the state that those two left. The seven pass when I run them by name. CI is the evidence for these nine), `app-private-hostname-gate.test.ts` (passed). - `pnpm --filter @paperclipai/server typecheck` passed. `pnpm check:module-boundaries` passed. - `pnpm verify` returned exit 2 (broader verification required, because the change adds a file under `server/scripts`). I did not run the full local suite. CI supplies that coverage. Reproduce: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-native-progress-statement.test.ts \ src/__tests__/prepared-statements.test.ts src/__tests__/chat-native-progress-work.test.ts TSX_DISABLE_CACHE=1 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-progress-page.ts \ --baseline-ref=267983356c ``` Not verified: the effect on a hosted instance. The rate above is one sample. The local numbers are from Node 26; the hosted instance runs Node 24. ## Risks - Low. The statement text and parameters are equal by test, and the read runs on the root client outside a transaction. - A prepared statement keeps the text of the first build. The lists inside the statement (safe event types, conversation states, publication states) are module constants, so the text cannot go stale in a running process. - The helper keys statements by client object. A caller that passes a transaction client gets a statement for that transaction only. This is correct, and it costs the same as before. - Tests that mock `db.select` to observe this read no longer see it. I found three and changed them to watch the driver. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model `claude-fable-5-2` (Fable 5.2), in Claude Code with tool use and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
7747487ee7 |
fix(budgets): retry contended accounting writes with jittered backoff (#21185)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget policies and usage records are written in serialized accounting transactions for each company > - When several writes for one company run at the same time, PostgreSQL rejects all but one and Paperclip retries the others > - The retries were few and short (4 attempts, 10 to 30 ms, the same delay for every writer), so concurrent writers collided again and the request failed with 503 > - The 503 message also said "this run's budget" for a policy write that has no run > - This pull request retries contended accounting writes with jittered exponential backoff and gives the error a correct message > - The benefit is that normal concurrent budget changes succeed, and a real failure tells the user what happened ## Linked Issues or Issue Description **What happened?** Several budget policy writes for one company at the same time returned 503 after 4 serialization retries. The same retry exhaustion appeared often in server logs of the chat feature evals. The message said "Paperclip could not save this run's budget or usage records … Retry the run" for a policy write. **Expected behavior** Concurrent budget policy writes succeed. If contention continues, the error describes budget or usage changes, not a run. **Steps to reproduce** Send 9 budget policy upserts for one company at the same time. Before this change, about half of them fail with 503. **Paperclip version or commit** master ## What Changed - `withOperationAccountingTransaction` retries up to `ACCOUNTING_TRANSACTION_ATTEMPTS` (8) times. The delay before each retry is jittered exponential backoff (`accountingRetryDelayMs`): 10 ms × 2^(attempt+1), capped at 400 ms, randomized between half and the full value. The worst case total wait is about 1.4 s. - `AccountingContentionError` uses a new generic message: "Paperclip could not save these budget or usage changes because other updates to the same company kept conflicting with them. Try again." - A failed run keeps its run-specific `ACCOUNTING_CONTENTION_MESSAGE` (run completion still uses it). - The existing exhausted-retry test uses the new attempt constant. ## Verification - `cd server && npx vitest run src/__tests__/budget-policy-write-contention.test.ts` - Red before the change: 5 of 9 concurrent policy writes failed with `AccountingContentionError`, and the message mentioned a run. - Green after: all writes succeed (5 of 5 repeated runs; also 3 of 3 runs with 17 concurrent writers). - `run-accounting-contention`, `run-completion`, `budget-control-publication`, `budget-evaluation-work` and `heartbeat-native-predispatch-retry` tests. - Paid runner E2E evidence for `goals.align-existing-work` is linked in a comment. ## Risks - Low. A write that keeps losing now waits up to about 1.4 s before it fails, instead of about 60 ms. Only transactions that PostgreSQL rolled back (40001 or 40P01) are retried, as before. - Moving policy writes out of the accounting transaction was considered. It is not done here, because the policy fence keeps budget evaluation consistent with policy changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
de675d924a |
fix(heartbeat): stop a comment the closing run already read from reopening its finished task (#21175)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can add a comment to a task while the agent's run on that task is still working > - The comment must reach the agent once: in the running turn, or in one later turn > - In a restart-chaos wave, a comment that the running turn already read reopened the finished task, and the agent did the completion turn a second time 15 minutes later > - This pull request makes a read by the running turn count as delivery, and holds the comment behind the running run so the run's own release settles it > - The benefit is no reopened done task, no repeated completion turn, and an unread comment that is delivered when its run finishes, not after a slot wait and a stale cancellation ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261112T181948-s7101`, task t005 (OpenCode, native Runner), during a two-server overlap deploy (K15). - 18:26:56 run A started on the old server. 18:26:55 traffic switched to the new server. - 18:27:03.953 a board user posted a follow-up comment. The new server handled it. Run A was not in that process's `activeRunExecutions`, so `filterZombieCoalesceTarget` treated it as a zombie. Admission returned `proceed`, and the comment got its own queued run B (18:27:04.252). The queued run is not part of the release drain. - 18:33:32 run A called `get_task_history`, received the comment, applied it, and finished the task (`done`, 18:34:20). - Run B waited for an agent slot until 18:41:49 and was cancelled as `continuation_task_ownership_changed`. - `deferUndeliveredCommentsOfStaleRun` then saved the comment again. Its "unread" test is only "a run started before the comment and finished after it". `deliveredQueuedCommentIds` had no record of run A's read. The release drain reopened the task (done to todo, 18:41:51), and run C (18:49:22) posted a second "Task complete" summary. The harness grade `done_comment_once` failed. **Expected behavior** - A comment that the running turn received is consumed. No queued work outlives that run, and the task stays done. - A comment that the running turn did not receive is delivered once, through the normal comment-on-done-task path, when that run finishes. - A stale continuation is never re-issued for a comment that a completed run already read. **Steps to reproduce** 1. Start a native run on a task. 2. Post a person's comment while another server instance executes the run (or seed a queued comment run behind it). 3. Let the run read `get_task_history`, then finish the task. 4. Before this change, the queued comment run is cancelled as stale, the task reopens, and the agent runs a second completion turn. **Paperclip version or commit** `master` at |
||
|
|
f92f91f0fb |
fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task gets a wake when one of its child tasks completes. The wake carries the child results. > - When the parent already has a run, the child result wake waits (`deferred_issue_execution`) behind that run. > - At finalization, a parent completion claim stays open (`native_child_completion_pending`) while such a wake is still queued, claimed or deferred. This guard protects child results that arrive while the parent works. > - The guard did not check when the child result arrived. A child result saved before the parent run started was already visible to that run, but it still blocked the claim. > - So the task stayed `in_progress`, the deferred wake started a second run, and the agent repeated its completion summary. > - This pull request counts a child result as undelivered only when it was saved after the finishing run started. > - The benefit is that a parent that already read its child results finishes in one turn, and child results that arrive during the run still get a fresh turn. ## Linked Issues or Issue Description Refs #15372 (added the pending child-completion guard). Refs #20935 (same family: a self-wake after a settled run). **What happened?** Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode, native runner, staging-mix parent with two subtasks). Both subtasks were done before parent run `8006a205` started at 18:45:29. The run read them, posted its final comment and claimed `done` at 18:46:15.860. A child-completion wake for the parent had been deferred since about 18:42 behind that run. The finalizer found that wake and committed `native_child_completion_pending`, so the task stayed `in_progress`. 63 ms later the release promoted the deferred wake into run `9282de34` (`invocationSource: automation`, `wakeCommentId: null`). Its `paperclip.native-continuation.v1` turn carried the subtask results that the first run had already used, plus the agent's own two comments (resume delta since the base run). That run posted a second "complete" summary and only then moved the task to `done`. Root cause: `hasPendingNativeChildCompletion()` (`server/src/services/native-runtime/native-child-completion-delivery.ts`) treated every queued, claimed or deferred child-completion wake for the issue as unread, except a wake bound to the finishing run itself. It did not compare the wake's save time with the run's start. **Expected behavior** A child result that was saved before the finishing run started is not new input for that run. The completion claim completes the task. The deferred wake is cancelled at release as a wake on a terminal task. A child result saved during the run still keeps the task open and gets a fresh parent turn. **Steps to reproduce** 1. Create a parent task with a done child. Save a deferred `issue_children_completed` wake for the parent. 2. Start a native parent run after that wake was saved. Let it claim `done`. 3. Finalize the run. On `master` the decision is `native_child_completion_pending` and the task stays `in_progress`; the deferred wake then starts a second run. ## What Changed - `native-child-completion-delivery.ts`: `hasPendingNativeChildCompletion()` counts a child-completion wake only when its `createdAt` is at or after the finishing run's `startedAt`. A run without a recorded start keeps the old behavior: every such wake stays pending. The finalizer, the status-commit recheck and status reassessment all use this function. - `heartbeat-process-recovery.test.ts`: a regression test finalizes a real native done claim with a deferred child-completion wake saved before (task becomes `done`) or after (task stays `in_progress` as `native_child_completion_pending`) the run started. It then releases the issue execution through the wake queue. In the `before` case the deferred wake is cancelled and there is one run total. In the `after` case the wake is promoted into a fresh run that carries the child results. - `doc/architecture/native-status-arbitration.md`: describes when a child result counts as undelivered. ## Verification - RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts` "treats a child result saved before the parent run started…" fails with decision `reasonCode: "native_child_completion_pending"`, `toStatus: "in_progress"`. This is the same decision as the wave. The "after" case passes on `master`. - GREEN: both cases pass with the fix. - Neighbouring suites pass: `native-status-arbiter-corpus` (includes the queued / claimed / deferred / consumed child-delivery cases from #15372), `heartbeat-dependency-scheduling` (revised child completion delivered after a running parent), `heartbeat-process-recovery`, `heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`. - `pnpm --filter @paperclipai/server typecheck` passes. - The regression test runs finalization and release end to end. The cancel-at-release step itself is existing behavior, also covered by `modules/wake-queue/application/use-cases.test.ts` ("cancels stale assignee continuations before claiming promotion on %s tasks", `done` case). ## Risks - A child result saved before the run started no longer blocks the claim. The child status was already `done` when the run started, so the run could read it. If the clocks of two server instances differ, a child result saved during the run could look older than the run start; the window is the clock skew. - A re-dispatched native status wake gets a new row with a later `createdAt`. Such a row stays pending, which is the old behavior. - No migration. Saved wakes from older builds work unchanged, because the check reads only existing columns. - Scope: this change does not alter the comment-wake promotion path or the resume-delta builder. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |